Pylint: unrecognized-inline-option fails whenever we try to parse something other than Python files

[TheJJ]

Steps to reproduce

1. Have a Python project with in-place Cython modules (e.g. https://github.com/SFTtech/openage)
2. Run pylint multiprocess: pylint --jobs=2 openage
3. Observe glorious crash and deadlock

Current behavior

Crash with funny unicode decode errors:

~/devel/openage % pylint --jobs=2 openage
No config file found, using default configuration
************* Module openage.__init__
W: 20, 0: TODO pylint: disable=wrong-import-position (fixme)
************* Module openage.__main__
W: 11, 0: TODO remove this once all multiprocessing has been eliminated: (fixme)
************* Module openage.default_dirs
W: 14, 0: TODO: use os.pathsep for multipath variables (fixme)
W: 42, 0: TODO: other windows paths (fixme)
Process ChildLinter-3:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/tokenize.py", line 390, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 40: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 129, in file_build
    stream, encoding, data = open_source_file(path)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 49, in open_source_file
    encoding = detect_encoding(byte_stream.readline)[0]
  File "/usr/lib64/python3.6/tokenize.py", line 431, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib64/python3.6/tokenize.py", line 395, in find_cookie
    raise SyntaxError(msg)
  File "<string>", line None
SyntaxError: invalid or missing encoding declaration for 'openage/cabextract/lzxd.cpython-35m-x86_64-linux-gnu.so'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 896, in get_ast
    return MANAGER.ast_from_file(filepath, modname, source=True)
  File "/usr/lib64/python3.6/site-packages/astroid/manager.py", line 80, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 137, in file_build
    '{error}', modname=modname, path=path, error=exc))
  File "/usr/lib64/python3.6/site-packages/astroid/util.py", line 30, in reraise
    six.reraise(type(exception), exception, sys.exc_info()[2])
  File "/usr/lib64/python3.6/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 129, in file_build
    stream, encoding, data = open_source_file(path)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 49, in open_source_file
    encoding = detect_encoding(byte_stream.readline)[0]
  File "/usr/lib64/python3.6/tokenize.py", line 431, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib64/python3.6/tokenize.py", line 395, in find_cookie
    raise SyntaxError(msg)
astroid.exceptions.AstroidSyntaxError: Python 3 encoding specification error or unknown encoding:
invalid or missing encoding declaration for 'openage/cabextract/lzxd.cpython-35m-x86_64-linux-gnu.so'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 208, in run
    result = self._run_linter(file_or_module[0])
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 237, in _run_linter
    linter.check(file_or_module)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 726, in check
    self._do_check(files_or_modules)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 844, in _do_check
    ast_node = self.get_ast(filepath, modname)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 900, in get_ast
    args=str(ex.error))
  File "/usr/lib64/python3.6/site-packages/pylint/utils.py", line 359, in add_message
    'Message %s must provide line, got None' % msgid)
pylint.exceptions.InvalidMessageError: Message E0001 must provide line, got None
Process ChildLinter-2:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/tokenize.py", line 390, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 24: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 129, in file_build
    stream, encoding, data = open_source_file(path)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 49, in open_source_file
    encoding = detect_encoding(byte_stream.readline)[0]
  File "/usr/lib64/python3.6/tokenize.py", line 431, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib64/python3.6/tokenize.py", line 395, in find_cookie
    raise SyntaxError(msg)
  File "<string>", line None
SyntaxError: invalid or missing encoding declaration for 'openage/cabextract/cabchecksum.cpython-35m-x86_64-linux-gnu.so'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 896, in get_ast
    return MANAGER.ast_from_file(filepath, modname, source=True)
  File "/usr/lib64/python3.6/site-packages/astroid/manager.py", line 80, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 137, in file_build
    '{error}', modname=modname, path=path, error=exc))
  File "/usr/lib64/python3.6/site-packages/astroid/util.py", line 30, in reraise
    six.reraise(type(exception), exception, sys.exc_info()[2])
  File "/usr/lib64/python3.6/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 129, in file_build
    stream, encoding, data = open_source_file(path)
  File "/usr/lib64/python3.6/site-packages/astroid/builder.py", line 49, in open_source_file
    encoding = detect_encoding(byte_stream.readline)[0]
  File "/usr/lib64/python3.6/tokenize.py", line 431, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/lib64/python3.6/tokenize.py", line 395, in find_cookie
    raise SyntaxError(msg)
astroid.exceptions.AstroidSyntaxError: Python 3 encoding specification error or unknown encoding:
invalid or missing encoding declaration for 'openage/cabextract/cabchecksum.cpython-35m-x86_64-linux-gnu.so'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 208, in run
    result = self._run_linter(file_or_module[0])
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 237, in _run_linter
    linter.check(file_or_module)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 726, in check
    self._do_check(files_or_modules)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 844, in _do_check
    ast_node = self.get_ast(filepath, modname)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 900, in get_ast
    args=str(ex.error))
  File "/usr/lib64/python3.6/site-packages/pylint/utils.py", line 359, in add_message
    'Message %s must provide line, got None' % msgid)
pylint.exceptions.InvalidMessageError: Message E0001 must provide line, got None
^CTraceback (most recent call last):
  File "/usr/lib/python-exec/python3.6/pylint", line 11, in <module>
    load_entry_point('pylint==1.7.2', 'console_scripts', 'pylint')()
  File "/usr/lib64/python3.6/site-packages/pylint/__init__.py", line 13, in run_pylint
    Run(sys.argv[1:])
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 1300, in __init__
    linter.check(args)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 729, in check
    self._parallel_check(files_or_modules)
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 797, in _parallel_check
    for result in self._parallel_task(files_or_modules):
  File "/usr/lib64/python3.6/site-packages/pylint/lint.py", line 772, in _parallel_task
    result = results_queue.get()
  File "<string>", line 2, in get
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 757, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

Expected behavior

Obviously to not perform a gory crash.

pylint --version output

pylint 1.7.2, 
astroid 1.5.3
Python 3.6.4 (default, Jan 14 2018, 19:57:53) 
[GCC 7.2.0]

[dstromberg]

Would it make sense to try to replicate with pylint 1.8.2?

On Sat, Feb 17, 2018 at 2:54 PM, Jonas Jelten [e-mail]
wrote:

Steps to reproduce

1. Have a Python project with in-place Cython modules (e.g.
   https://github.com/SFTtech/openage https://github.com/SFTtech/openage
   )
2. Run pylint multiprocess: pylint --jobs=2 openage
3. Observe glorious crash and deadlock

Current behavior

Crash with funny unicode decode errors:

Expected behavior

Obviously to not perform a gory crash.
pylint --version output

pylint 1.7.2,
astroid 1.5.3
Python 3.6.4 (default, Jan 14 2018, 19:57:53)
[GCC 7.2.0]

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/PyCQA/pylint/issues/1885, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0yGp-jm1RXaXTnlXeKAsjIySLMVQJSks5tV1iQgaJpZM4SJfuN
.

--
Dan Stromberg

[zuntrax]

I used to get the same tracebacks with older versions, with 1.8.1, those errors are handled more gracefully.

['--rcfile=etc/pylintrc',
 '--reports=n',
 '--jobs=4',
 '--ignored-modules=openage.cvar.cvar,openage.util.fslike.cpp,openage.util.filelike.cpp,openage.game.main_cpp,openage.cppinterface.exctranslate,openage.cppinterface.exctranslate_tests,openage.cppinterface.pyobject,openage.cppinterface.setup_checker,openage.cabextract.cabchecksum,openage.cabextract.lzxd,openage.log.log_cpp,openage.testing.cpp_testing,openage.testing.misc_cpp,openage.convert.slp,openage.convert.interface.visgrep,openage.convert.opus.opusenc,numpy',
 'openage',
 'buildsystem']
Using config file /home/zuntrax/git/sfttech/openage/etc/pylintrc
internal error with sending report for module ['openage/cvar/cvar.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/util/threading.py']
'Uninferable' object is not iterable
internal error with sending report for module ['openage/util/fslike/cpp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/util/filelike/cpp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/game/main_cpp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cppinterface/pyobject.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cppinterface/exctranslate_tests.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cppinterface/exctranslate.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cppinterface/setup_checker.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cabextract/lzxd.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cabextract/cabchecksum.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/log/log_cpp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/testing/misc_cpp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/testing/cpp_testing.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/convert/slp.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/convert/opus/opusenc.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/convert/interface/visgrep.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

I dumped the parameters on top, as you can see, we even explicitly ignore the cython modules, but in multiprocessing mode, this flag is not honored. This has also been the case in January 2017 when I added the jobs flag to the invocation.

[PCManticore]

@zuntrax Thanks for creating this issue. I realise that ignored-modules is a misnomer, but it doesn't do what you think it does. It represents a list of module names for which member attributes should not be checked (useful for modules/projects where namespaces are manipulated during runtime and thus existing member attributes cannot be deduced by static analysis. That is, it doesn't ignore the module/file overall, but just for a handful of checks, which means it also gets parsed like any other file/module would be. You are looking instead for the ignore option: https://pylint.readthedocs.io/en/latest/technical_reference/features.html?highlight=ignored%20modules#id34

[TheJJ]

Hm I don't think this issue is resolved.
The error that pylint crashes with internal errors is still there, regardless of some ignore* option.

% pylint --version
No config file found, using default configuration
pylint 1.8.2, 
astroid 1.6.1
Python 3.6.4 (default, Jan 14 2018, 19:57:53) 
[GCC 7.2.0]

Same run:

% pylint --jobs=2 openage
No config file found, using default configuration
************* Module openage.__init__
W: 20, 0: TODO pylint: disable=wrong-import-position (fixme)
************* Module openage.__main__
W: 11, 0: TODO remove this once all multiprocessing has been eliminated: (fixme)
************* Module openage.default_dirs
W: 14, 0: TODO: use os.pathsep for multipath variables (fixme)
W: 42, 0: TODO: other windows paths (fixme)
internal error with sending report for module ['openage/cabextract/lzxd.cpython-35m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cabextract/cabchecksum.cpython-35m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
internal error with sending report for module ['openage/cabextract/cabchecksum.cpython-36m-x86_64-linux-gnu.so']
Message E0001 must provide line, got None
************* Module openage.cabextract.lzxdstream
[...]

As you can see, the internal error still occurs, but at least it doesn't crash the whole pylint run now.

[PCManticore]

Right, the error you are seeing is caused by the parsing of those .so files, which in this case fails before even getting a line from the tokenizer (thus the line None you are seeing). I'm reopening this one, but changing it's title, since the original issue is solved.

[TheJJ]

(mind that this only occurs when running with multiple jobs)

[tilgovi]

I am seeing the same behavior and can also confirm that it only happens with multiple jobs enabled.

[tilgovi]

This appears to fix the issue for me. This would be my first time contributing to Pylint, so please spot check my idea here before I open a PR, if you wouldn't mind.

From what I gather, is_arg gets lost in the delegation to sub-processes that parallel checking does. Instead of sending all the files to the sub-processes, pre-filter it to contain only those modules that should be checked.

diff --git a/pylint/lint.py b/pylint/lint.py
index 1565d77b..fc517eb2 100644
--- a/pylint/lint.py
+++ b/pylint/lint.py
@@ -812,7 +812,11 @@ class PyLinter(config.OptionsManagerMixIn,
         results_queue = manager.Queue()

         # Send files to child linters.
-        expanded_files = self.expand_files(files_or_modules)
+        expanded_files = []
+        for descr in self.expand_files(files_or_modules):
+            modname, filepath, is_arg = descr['name'], descr['path'], descr['isarg']
+            if self.should_analyze_file(modname, filepath, is_argument=is_arg):
+                expanded_files.append(descr)

         # do not start more jobs than needed
         for _ in range(min(self.config.jobs, len(expanded_files))):

[PCManticore]

@tilgovi That seems to make sense, thanks for tackling it! Would you mind sending a PR? We can discuss there what additional changes would this need (tests, ChangeLog entry etc)