Conan: Cannot get MD5 for non-ASCII filenames

[masseman]

(Conan 0.30.3, Python 2.7.12, Ubuntu 16.04)

In conans/util/files.py, I get

UnicodeEncodeError: 'ascii' codec can't encode characters in position 10122-10125: ordinal not in range(128)

on the line content.encode(). This is because the unicode-type string content contains non-ASCII characters that are not handled by the default encoder.

Is there any reason not to do content.encode("utf-8")?

To reproduce, add a file with non-ASCII characters in the filename to the files exported by a library.

[memsharded]

Yes, encoding is very problematic in Py2. Basically filenames might fail, if not in that point, in other places. There have been a couple of attempts to fix this, but without success. Have a look at latest one by @mpdelbuono in https://github.com/conan-io/conan/issues/2163

So I strongly recommend using Py3, and follow that thread an collaborate with @mpdelbuono in improving Py3 support for unicode. Thanks!

[mpdelbuono]

I have not been able to reproduce this in Py3 with conan 1.0. Here's my test setup:

• conan data directory is a folder named with Japanese characters
• conanfile.py lives in a folder named with Japanese characters
• inside the folder lives a file named "日本語のファイル.txt"
• the file is UTF-8 encoded with Japanese characters as its contents
• conanfile.py has exports_sources configured to export that file
• my build() method has the following code:

        path = os.path.join(self.source_folder, "日本語のファイル.txt")
        self.output.info("MD5    : {}".format(tools.md5sum(path)))
        self.output.info("SHA1   : {}".format(tools.sha1sum(path)))
        self.output.info("SHA256 : {}".format(tools.sha256sum(path)))

• I run conan create .

@masseman have you encountered the same troubles on Py3, or is this only a Py2 problem?

[mpdelbuono]

As a side note: I have also verified that the output hashes are correct.

[masseman]

@mpdelbuono Thanks looking into this! I was using Python 2.7.

[DavidZemon]

I believe I'm running into a similar problem with Python 3. Can you advise if it is the same or if I should create a new ticket?

...
...
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Package '44fcf6b9a7fb86b2586303e3db40189d3b511830' built
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Build folder /home/captain/.conan/data/OpenOCD-ESP32/0.10.0+esp32+20181105/foo/bar/build/44fcf6b9a7fb86b2586303e3db40189d3b511830
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Generated conaninfo.txt
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Generated conanbuildinfo.txt
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Generating the package
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Package folder /home/captain/.conan/data/OpenOCD-ESP32/0.10.0+esp32+20181105/foo/bar/package/44fcf6b9a7fb86b2586303e3db40189d3b511830
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar: Calling package()
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 2 files: openocd, README
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.info-1' file: openocd.info-1
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.info' file: openocd.info
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.info-2' file: openocd.info-2
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.rules' file: 99-openocd.rules
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.h' file: dcc_stdio.h
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 2 '.c' files: dcc_stdio.c, example.c
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 30 '.tcl' files: 
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 628 '.cfg' files: 
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.txt' file: readme.txt
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.hex' file: ulink_firmware.hex
OpenOCD-ESP32/0.10.0+esp32+20181105@foo/bar package(): Copied 1 '.1' file: openocd.1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 1427, in run
    method(args[0][1:])
  File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 297, in create
    test_build_folder=args.test_build_folder)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 87, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 380, in create
    test_build_folder, test_folder, conanfile_path)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/cmd/create.py", line 41, in create
    test_build_folder=test_build_folder)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/cmd/test.py", line 42, in install_build_and_test
    keep_build=keep_build)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/manager.py", line 100, in install
    installer.install(deps_graph, keep_build)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/installer.py", line 275, in install
    self._build(nodes_by_level, deps_graph, keep_build, root_node, graph_info)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/installer.py", line 302, in _build
    self._handle_node_cache(node, package_ref, keep_build, processed_package_refs)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/installer.py", line 325, in _handle_node_cache
    self._build_package(node, package_ref, output, keep_build)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/installer.py", line 415, in _build_package
    builder.package()
  File "/usr/local/lib/python3.6/dist-packages/conans/client/installer.py", line 142, in package
    conanfile_path, self._conan_ref)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/packager.py", line 95, in create_package
    _create_aux_files(install_folder, package_folder, conanfile, copy_info)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/packager.py", line 119, in _create_aux_files
    digest.save(package_folder)
  File "/usr/local/lib/python3.6/dist-packages/conans/model/manifest.py", line 100, in save
    save(path, repr(self))
  File "/usr/local/lib/python3.6/dist-packages/conans/util/files.py", line 143, in save
    new_content = to_file_bytes(content)
  File "/usr/local/lib/python3.6/dist-packages/conans/util/files.py", line 161, in to_file_bytes
    content = bytes(content, "utf-8")
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 36543-36546: surrogates not allowed

ERROR: 'utf-8' codec can't encode characters in position 36543-36546: surrogates not allowed
captain@balrog:/home/david/reusable/Documents/Programming/RedLion/noarch-conan-packages$ 

I will get you step-by-step instructions for reproducing tomorrow if you need. For the moment, I can tell you I'm running a Docker image based on Ubuntu 18.04 with Conan v1.11.2 and Python 3.6 (as clearly indicated by the stacktrace above) and the recipe is attached
openocd-esp32.zip

[uilianries]

Hi @DavidZemon !

Which docker image? Did you change anything there?
Which commands did you run?

I was not able to reproduce your error when running:

docker run -v ${PWD}:/home/conan/project conanio/gcc6
cd project
conan create . foo/testing

[DavidZemon]

I'm back! Unexpected work deliverable is delivered, so hopefully I can be a little more prompt in responding to any further questions you might have. I've built a very simple Docker image that replicates the problem and included it in the attached zip. You should be able to build that image and then run the following command to reproduce the error:

docker run -it --rm -v `pwd`:`pwd` -w `pwd` foobar conan create openocd-esp32 foo/bar

foobar_docker_settigns.zip

[uilianries]

Maybe there is something missing in your docker image:

FROM ubuntu:18.10

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install --yes --no-install-recommends \
    python3-pip \
    python3-setuptools \
    python3-wheel

RUN pip3 install conan==1.11.2

ENV HOME=/home/captain \
  CONAN_PRINT_RUN_COMMANDS=1
COPY conan/profile "${HOME}/.conan/profiles/default"
COPY conan/settings.yml "${HOME}/.conan/settings.yml"

RUN groupadd --gid 1000 captain \
  && useradd --home-dir "${HOME}" --uid 1000 --gid 1000 captain \
  && mkdir --parents "${HOME}/.ssh" \
  && cp /root/.bashrc /root/.profile "${HOME}" \
  && chown --recursive captain:captain "${HOME}" \
  && chmod --recursive 777 "${HOME}" \
  && echo "ALL ALL=NOPASSWD: ALL" >> /etc/sudoers

You described conan create command and the profile file is gcc7, however, your image doesn't have gcc or even some requirements for Conan.

Why you don't use Conan's Docker images? We have the image conanio/gcc7. All Docker recipes are available for download and under MIT license in the project conan-docker-tools

[DavidZemon]

We can't use the conanio/* images because it's part of a MUCH larger Docker image in charge of building far more than just Conan packages. The real Docker image is wsbu/toolchain-native which can be found here on Docker hub and here on GitHub.

[uilianries]

Understood! I gonna provide a fix, I think we could escape the surrogates in this case. Conan is trying to calculate the file and add its result in the manifest file.

[uilianries]

@DavidZemon I can't reproduce your scenario, here is what I tried:

wget https://github.com/conan-io/conan/files/2866870/foobar_docker_settigns.zip
unzip foobar_docker_settigns.zip
cd foobar_docker_settigns
docker pull wsbu/toolchain-native:latest
docker run --rm -ti -v ${PWD}:/home/captain/project wsbu/toolchain-native:latest
cd /home/captain/project
conan create . user/channel

The image wsbu/toolchain-native runs Python2 by default, but you commented about python 3.6.

I've tried Python 3.6.0 + Conan 1.11.2 on my machine (I have pyenv + virtualenv installed) but there is no error.

I'm convicted that the file openocd.info caused your error, but I'm not sure how to reproduce your scenario.

[DavidZemon]

Okay, I did some more looking into the original error and figured out why you couldn't reproduce it. It's actually running from wsbu/teamcity-agent:v0.0.9 (https://github.com/wsbu/docker-teamcity-agent/blob/master/Dockerfile) which is _based off of_ wsbu/toolchain-native. The teamcity agent image adds Python 3. When you run this to reproduce, make sure you use --entrypoint because the default entrypoint will start a TeamCity agent, not bash:

david@balrog:~/reusable/Documents/Programming/RedLion/noarch-conan-packages/openocd-esp32$ docker run -it --rm -v `pwd`:`pwd` -w `pwd` --entrypoint /start.sh wsbu/teamcity-agent:v0.0.9
root@996b87c9b28d:/home/david/reusable/Documents/Programming/RedLion/noarch-conan-packages/openocd-esp32# conan create . user/channel
...
...

[uilianries]

@DavidZemon now I'm able to reproduce your error, thanks!

[DavidZemon]

Sorry for the run-around on reproducing it. Thanks for sticking with me!

[Pancir]

I have faced with the same problem on ubuntu 18.04 with the python 3.6.
To fix the problem I temporary installed the python 2 and run conan under it.

[rconde01]

I just ran into this:

• redhat 7 host
• redhat 7 container
• rh-python36
• latest conan (as of today)
• custom boost package

I was able to fix it by applying https://github.com/uilianries/conan/archive/hotfix/bytes-parsing.zip which was shown to me by @uilianries on the CppLang slack conan channel.

I unzipped it, and ran: sudo pip install .

After that it worked.

however that patches to an old version of conan. It would be great if this were fixed properly.

[Pancir]

Any news? It seems last conan versions can't be installed under python 2 but the problem described in this thread is still there for python 3 and Linux and conan 1.19.2.

[memsharded]

Hi @Pancir

Yes, last conan versions can still be installed in Python 2, at least under several of the mainstream OS, and it is continuously tested in CI. I am running Conan in Python2.7 in Windows.

What specific problems did you have for Python2? Just in case I could help. Not critical, indeed the preferred approach is to move to Python3, as Python2 is going to be deprecated soon.

Hi @uilianries I guess the patch you provided is based on https://github.com/conan-io/conan/pull/4532, right?

I would like to understand that better and see what can be done. Assigning it to the next 1.21 release (1.20 is about to be closed) to investigate better.

[Pancir]

Hello, we are moving to Python3 and my problem is in this version and for Linux.
conan 1.19.2

[Dockerfile.txt](https://github.com/conan-io/conan/files/3773874/Dockerfile.txt)


Traceback (most recent call last):
  File "build.py", line 6, in <module>
    builder.run()
  File "/usr/local/lib/python3.6/dist-packages/cpt/packager.py", line 499, in run
    self.run_builds(base_profile_name=base_profile_name)
  File "/usr/local/lib/python3.6/dist-packages/cpt/packager.py", line 582, in run_builds
    r.run()
  File "/usr/local/lib/python3.6/dist-packages/cpt/runner.py", line 114, in run
    test_folder=self._test_folder)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 78, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 335, in create
    not not_export, graph_lock=graph_info.graph_lock)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/cmd/export.py", line 136, in cmd_export
    manifest.save(package_layout.export())
  File "/usr/local/lib/python3.6/dist-packages/conans/model/manifest.py", line 110, in save
    save(path, repr(self))
  File "/usr/local/lib/python3.6/dist-packages/conans/util/files.py", line 172, in save
    new_content = to_file_bytes(content)
  File "/usr/local/lib/python3.6/dist-packages/conans/util/files.py", line 190, in to_file_bytes
    content = bytes(content, "utf-8")
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1199-1204: surrogates not allowed

I have a resource folder for my tests and it contains Unicode symbols like this:

src-test-resources/X-平面
src-test-resources/cases/Algorithms/deleteFiles/文件夾1/文件1
etc ...

I export the resource folder with the recipe

exports_sources = 'CMakeLists.txt', 'src/*', 'src-test/*', 'src-test-resources/*', \
                      'include/*', 'cmake/*', 'license*'

It works without problems on Windows. On Mac we still have python 2 perhaps this problem will also be on Mac when we migrate to python3.
I attached docker file as an information about the environment.
Dockerfile.txt

[Pancir]

I changed my docker file to use pyenv and python 3.7.4 and it helped. On mac it also works properly with pyenv and python 3.7.4, you can consider my problem is not relevant anymore.

[memsharded]

Thanks @Pancir for telling, good to know that it is working for py 3.7.4.

I think I am going to leave this to investigate in 1.21, it has affected other users too, and likely that others might hit it again.

[uilianries]

Hi @uilianries I guess the patch you provided is based on #4532, right?

Yes, I remember someone tried that patch and worked nicely. But as you know, it only occurred when running Python 2.

[SSE4]

I believe we should take a look at PEP 0383

On POSIX systems, Python currently applies the locale's encoding to convert the byte data to Unicode, failing for characters that cannot be decoded. With this PEP, non-decodable bytes >= 128 will be represented as lone surrogate codes U+DC80..U+DCFF. Bytes below 128 will produce exceptions; see the discussion below.

To convert non-decodable bytes, a new error handler ([2]) "surrogateescape" is introduced, which produces these surrogates. On encoding, the error handler converts the surrogate back to the corresponding byte. This error handler will be used in any API that receives or produces file names, command line arguments, or environment variables.

since we're trying to store/load file names as bytes, we have to use surrogateescape encoding, as PEP suggests

[uilianries]

It looks interesting ... Should we update #4532 ?

[SSE4]

no, we should discuss and create new PR, if possible. there should be a strong validation that all file names saved to the manifest and loaded from it are correctly handled, without lose of characters. it should also apply to the md5sum/sha256sum.

[memsharded]

I agree with @SSE4 :

• It seems the way to go is surrogatescape.
• We need extensive validations that the manifests, the checksums, etc are ok.
• If we apply it to to_file_bytes() will be used for other things as well, any other file that conan generates will use it.

Probably makes sense to wait until deprecating Python2.

[ericriff]

Strangely enough, I came across a very similar error today. I can build my package locally on Ubuntu16+conan 1.21 and also Ubuntu 18 + conan 1.29.0. On our CI servers, we rock the same configs but on dockers.
On the docker images, I get

DEBUG :packager.py    [112]: PACKAGE: Creating config files to /home/ubuntu/.conan/data/pkg/1.0/usr/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 [2020-09-30 23:12:38,585]
Traceb:ack (most recent call last):
  File "conan/conans/client/command.py", line 1947, in run
  File "conan/conans/client/command.py", line 358, in create
  File "conan/conans/client/conan_api.py", line 81, in wrapper
  File "conan/conans/client/conan_api.py", line 364, in create
  File "conan/conans/client/cmd/create.py", line 43, in create
  File "conan/conans/client/cmd/test.py", line 38, in install_build_and_test
  File "conan/conans/client/manager.py", line 68, in deps_install
  File "conan/conans/client/installer.py", line 308, in install
  File "conan/conans/client/installer.py", line 332, in _build
  File "conan/conans/client/installer.py", line 398, in _handle_node_cache
  File "conan/conans/client/installer.py", line 442, in _build_package
  File "conan/conans/client/installer.py", line 212, in build_package
  File "conan/conans/client/installer.py", line 161, in _package
  File "conan/conans/client/packager.py", line 91, in run_package_method
  File "conan/conans/client/packager.py", line 125, in _create_aux_files
  File "conan/conans/model/manifest.py", line 110, in save
  File "conan/conans/util/files.py", line 183, in save
  File "conan/conans/util/files.py", line 201, in to_file_bytes
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1257517-1257518: surrogates not allowed

ERROR: 'utf-8' codec can't encode characters in position 1257517-1257518: surrogates not allowed

And I cant reproduce it locally or figure out what is going on.
We are using python 3.5 on U16 and 3.6 on U18.

Any suggestions to debug this?

---

UPDATE: It was related to a locale config. The following did the trick for me.

RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
    LANG=en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8

Source: https://www.embeddeduse.com/2019/02/11/using-docker-containers-for-yocto-builds/