Wemake-python-styleguide: Windows build fails

[sobolevn]

Bug report

What's wrong

I don't have any experience with Windows. So, I will appreciate any help.

Windows tests fail:
https://ci.appveyor.com/project/wemake-services/wemake-python-styleguide/build/job/185lp386ce91jtsy

c:\users\appveyor\appdata\local\pypoetry\cache\virtualenvs\wemake-python-styleguide-py3.6\lib\site-packages\flake8_pytest.py:33: in run
    noqa = _get_noqa_lines(file_to_check.readlines())
c:\python36\lib\encodings\cp1252.py:23: in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
E   UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5978: character maps to <undefined>
------------------------------ Captured log call ------------------------------

How is that should be

Tests should pass.

[sobolevn]

Seems like flake8-pytest issue.

[sobolevn]

The simplest solution is to drop flake8-pytest. Since it is not a part of our linter. And we use functional tests anyway.

[sobolevn]

Still failing, removing flake8-pytest did not help.
So, it should be returned. Let's fix this problem somehow differently.

https://ci.appveyor.com/project/wemake-services/wemake-python-styleguide/build/job/emtyxbkorpj1eear

c:\users\appveyor\appdata\local\pypoetry\cache\virtualenvs\wemake-python-styleguide-py3.6\lib\site-packages\flake8\checker.py:451: in run_check
    return plugin["plugin"](**arguments)
c:\users\appveyor\appdata\local\pypoetry\cache\virtualenvs\wemake-python-styleguide-py3.6\lib\site-packages\flake8_bandit.py:35: in __init__
    self._load_source()
c:\users\appveyor\appdata\local\pypoetry\cache\virtualenvs\wemake-python-styleguide-py3.6\lib\site-packages\flake8_bandit.py:73: in _load_source
    self.source = f.read()
C:\Python36\lib\encodings\cp1252.py:23: in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
E   UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 14170: character maps to <undefined>

[nixphix]

@sobolevn I haven't used Windows in a long time, let me have a look.

[sobolevn]

@nixphix in case you are still interested: any help is highly appreciated.

[sobolevn]

No CI = no CI fails. Welcome to the brave new world.

[adambenali]

@sobolevn here is a list of all the reasons why the build doesn't work on Windows (the ones I fixed to get it to work for me locally):

• flake8-pytest fails on test_unicode.py because of encoding. I have found no issue discussing this problem in the plugin's repo and it looks like it's been years since it has last been active.
• flake8-plugin-utils fails because of encoding. See https://github.com/afonasev/flake8-plugin-utils/issues/9.
• import-linter fails because of encoding again. No issue discusses the problem in the plugin's repo.
• doc8 which throws D002 and D004 warnings at every line because the default line endings in Windows are CRLF. A bad workaround would be to ignore these two warnings : doc8 -q docs --ignore D002,D004. (see https://github.com/vscode-restructuredtext/vscode-restructuredtext/issues/84)

When these problems are solved, the lint check passes and the tests start running but some of them fail for these reasons :

• diff doesn't exist on Windows, but it's assumed to be there here : https://github.com/wemake-services/wemake-python-styleguide/blob/a716092ebaf899fc42dfba0c8d6e10effa6d7027/tests/test_plugins.py#L71 and here https://github.com/wemake-services/wemake-python-styleguide/blob/0c165adda690f9bb9e51c259255fb71ab29a9cf8/tests/test_checker/test_noqa.py#L428
• Finally, the formatter test fails because of bandit (which is used by flake8-bandit). It needs to print a beautiful report as well and to do that for Windows it enables colorama (cf. https://github.com/PyCQA/bandit/pull/424). The issue is that Colorama is not disabled after the report is outputted, and anything that imported "bandit" is now forced to use Colorama. Since Colorama does not handle some fancy styling like underlines, the tests fail. Will open an issue about it. Meanwhile a workaround is to just disable all the styling that Colorama doesn't handle.

The encoding problem is basically that the default encoding on Windows is not UTF-8 but CP1252 which fails to decode some characters. The encoding should either be specified when opening files or use tokenize.open() for example which detects file encoding. The problem used to be there for flake-bandit as well but it was fixed at some point after you opened an issue.

[sobolevn]

Oh wow! Thank you for such a detailed report. I will open issues for

• import-linter
• flake8-pytest, but you know, this dependency can be easily dropped in favour of flake8-pytest-style (I will open an issue there as well). Feel free to send a PR with removed flake8-pytest
• doc8

About our local problems:

• we can skip diff tests on Windows, that's fine
• we can also skip formatting tests on Windows, that's fine

[sobolevn]

@adambenali can you please post reproduction steps on Windows to see these encoding issues?

[adambenali]

Sure.
Reproduction steps for flake8-plugin-utils:

1. Install flake8
2. Install flake8-plugin-utils
3. Install any plugin that uses flake8-plugin-utils, like flake8-pytest-style.
4. Create a file with this content: print('變量').
5. Run flake8 filename.py

Will fail with :

  File "c:\users\adam\anaconda3\lib\site-packages\flake8_plugin_utils\plugin.py", line 48, in run
    self._load_file()
  File "c:\users\adam\anaconda3\lib\site-packages\flake8_plugin_utils\plugin.py", line 61, in _load_file
    self._lines = f.readlines()
  File "c:\users\adam\anaconda3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 12: character maps to <undefined>

Basically the same steps for flake8-pytest.

By the way, import-linter is not the direct culprit. It fails because of another module, grimp.

[sobolevn]

@adambenali I would appreciate an issue from you to gimp. I am not qualified enough to do this 🙂

[adambenali]

I've opened an issue about grimp : https://github.com/seddonym/grimp/issues/75.

[orsinium]

flake8-pytest-style is not in the dependency graph of WPS anymore. I think it should be closed. Objections?