Poetry: UnicodeDecodeError in Masonry with MSYS2

[danyeaw]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 with MSYS2 20180531
• Poetry version: 0.12.11
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/gaphor/gaphor/blob/master/pyproject.toml

Issue

When trying to pip install my app I am getting a UnicodeDecodeError.

$ pip3 install --user .
Processing c:/tools/msys64/home/dyeaw/gaphor
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... error
    Complete output from command C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr:
    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 64, in prepare_metadata_for_build_wheel
        hook = backend.prepare_metadata_for_build_wheel
    AttributeError: module 'poetry.masonry.api' has no attribute 'prepare_metadata_for_build_wheel'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 207, in <module>
        main()
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 67, in prepare_metadata_for_build_wheel
        config_settings)
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 95, in _get_wheel_metadata_from_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\api.py", line 40, in build_wheel
        poetry, SystemEnv(Path(sys.prefix)), NullIO(), Path(wheel_directory)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 48, in make_in
        wb = WheelBuilder(poetry, env, io, target_dir=directory, original=original)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 38, in __init__
        super(WheelBuilder, self).__init__(poetry, env, io)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\builder.py", line 42, in __init__
        self._meta = Metadata.from_package(self._package)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\metadata.py", line 50, in from_package
        meta.description = f.read()
      File "C:/tools/msys64/mingw64/lib/python3.7\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 7221: character maps to <undefined>

    ----------------------------------------
Command "C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr" failed with error code 1 in C:/Users/dyeaw/AppData/Local/Temp/pip-req-build-clnw4_9m

It looks like Poetry is not using an encoding when it calls open() to load the description.

I am opening the issue here after exploring the cause in MSYS2 first: https://github.com/msys2/MINGW-packages/issues/5155

[drunkwcodes]

Save your pyproject.toml in utf-8 encoding again should be alright?

To handle different encodings, it needs to depend on chardet without a guarentee resolving this problem.

Reference: https://stackoverflow.com/questions/436220/how-to-determine-the-encoding-of-text

[danyeaw]

@drunkwcodes My pyproject.toml, and the rest of my project is in UTF-8. The issue is that Poetry is calling open() with no encoding specified, and the default encoding for Windows is cp1252.

[drunkwcodes]

Sorry. Here it is. Please try to change msys2's encoding setting to utf-8.
Or use chcp 65001 command.

https://stackoverflow.com/questions/28481410/how-to-set-console-encoding-in-msys

[danyeaw]

@drunkwcodes Thanks for trying to research possible solutions. The correct default/preferred encoding of the contents of files for Windows users is cp1252. This is what is returned when I call locale.getpreferredencoding(). It is not a solution for me to change my default encoding. This issue here is that Poetry is opening pyproject.toml, which should be UTF-8 for all users, without specifying the encoding to use.

[drunkwcodes]

@danyeaw Sorry for not having a solution for you to change your default encoding, sir. What a lovely night.

[joar]

I believe https://github.com/sdispater/poetry/issues/1030 is related to this issue. I don't think it's safe to assume that the locale will always be set to UTF-8 on systems where poetry (or packages using poetry, such as pendulum) might be installed.

[vlcinsky]

We (me and @rhorenov) had problem to install pendulum==2.0.4 on MS Windows with encoding windows-1250.

If the package was to be built (e.g. pendulum), it failed on special characters in metadata generated by poetry as it was opening files for build using default console encoding.

Installing packages from wheels worked well.

I am going to create PR for master to fix this issue (adding explicit encoding where files are open for writing in text mode).

[vlcinsky]

Can @danyeaw or anyone else check, that the latest poetry version 0.12.17 is not not suffering this problem?

[danyeaw]

Closed by commit e9a13f8 👍

[vlcinsky]

Greate @danyeaw

Can you close the issue? I do not seem to have such option.