Pylint: ImportError on collection of module in pylint 2.5.0 when using multiprocessing option

[robsan00]

Steps to reproduce

1. create a new & empty virtualenv only containing pyling 2.5.0 (using Python 3.6)
2. create a folder "test" somewhere
3. add an (empty) "__init__.py" file there
4. add an (empty "spam.py" file there
5. go to the parent directory of "test" and call "pylint -j 1 test" -> no error
6. go to the parent directory of "test" and call "pylint -j 2 test" ->

Traceback (most recent call last):
  File "c:\Python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\work\venv\pylint\Scripts\pylint.exe\__main__.py", line 7, in <module>
  File "c:\work\venv\pylint\lib\site-packages\pylint\__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "c:\work\venv\pylint\lib\site-packages\pylint\lint\run.py", line 338, in __init__
    linter.check(args)
  File "c:\work\venv\pylint\lib\site-packages\pylint\lint\pylinter.py", line 878, in check
    files_or_modules,
  File "c:\work\venv\pylint\lib\site-packages\pylint\lint\check_parallel.py", line 101, in check_parallel
    _worker_check_single_file, files
  File "c:\Python36\lib\multiprocessing\pool.py", line 735, in next
    raise value
ImportError: Unable to find module for test\spam.py in C:\work\venv\pylint\Scripts\pylint.exe,
c:\work\venv\pylint\scripts\python36.zip,
c:\Python36\DLLs,
c:\Python36\lib,
c:\Python36,
c:\work\venv\pylint,
c:\work\venv\pylint\lib\site-packages,
c:\work\venv\pylint\lib\site-packages\astroid\brain

Current behavior

ImportError

Expected behavior

no error (for example with pylint 2.4.4)

pylint --version output

pylint 2.5.0
astroid 2.4.0
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]

[PCManticore]

Thanks for the report! This is a regression caused by 2.5.0 which most likely was a result of https://github.com/PyCQA/pylint/pull/3411.

Going to investigate and provide a fix hopefully this week. In the meantime I suggest downgrading to 2.4.4 or using -j1, sorry for the hassle.

[robsan00]

Already did the 2.4.4 downgrading ;-), thanks for the quick reply!

[IRDonch]

I noticed that 2.5.0 also doesn't calculate the score in multiprocessing mode - could it be related to this bug?

[felixdivo]

Could it be that some docs need updating?

[...] should always work since the current working directory is automatically added on top of the python path

[svenpanne]

Is there anything one can do to help resolving this issue and/or raise its priority? It's a complete show stopper for us, and judging from the linked commits/issues for many other projects, too. The workaround of using -j1 is not an option when you already wait 8 minutes in the parallel case... :disappointed:

[crisbal]

@svenpanne if you do no need any of the features/fixes introduced with 2.5.0 you can just pin pylint to a version which is not 2.5.0.

Basically in your requirements.txt or setup.py (or wherever you specify the dependencies) use "pylint!=2.5.0" instead of "pylint". Version 2.4.4 will be used. When a fix will come you can remove the version specification.

[knoop1975]

The suggested -j1 fix doesn't solve the problem.

[svenpanne]

@svenpanne if you do no need any of the features/fixes introduced with 2.5.0 you can just pin pylint to a version which is not 2.5.0. [...]

This is exactly what we are doing at the moment, just like tons of other projects if one looks at the linked issues. We have quite a few suppressions caused by false positives, so I thought giving 2.5.0 a try might be a good idea, but obviously it's not...

[PCManticore]

@svenpanne I understand your pain. For context, almost all the minor releases had some issues in the past and I usually recommend to wait until a couple of patch releases before jumping to a minor version of Pylint. Usually the minor version that is going to be released does not get a lot of testing from the community and some critical issues leak into the minor release when that happens.

Regarding bumping the priority, this issue is one of my main concerns for a 2.5.1 or 2.5.2 release. But it's non trivial and I also have a day to day job, so please be a bit patient on that front. As soon as we have a fix, you'll see the notification here. If you do want to help, investigating the root cause or sending a patch is more than welcome and appreciated.

[Sazpaimon]

Unfortunately pinning to 2.4 is no longer a universal solution now because Safety has now marked pylint <2.5.0 to be insecure: https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json#L10823-L10833

[PCManticore]

Hey folks, this should be fixed in both master and the 2.5 branch. Can you give it a go and let us know if pylint works for you now? If so, I'll do a 2.5.1 release tomorrow.

[NeilGirdhar]

Works for me! Thanks for the quick turnaround.

[robsan00]

@PCManticore
Using pylint at 3fffdf03d5a32e4a0d1b5216ea9d16dacacdaa6f, I don't get the same error, but the multi-core stuff still looks broken to me: the same error gets reported lots of times as if the same file is checked loads of times. The suite I let it run over has about 60 errors (false positives that I ignore in post-processing). Now there are about 1800 errors. The first error (the exact error including file name, line and position) was reported 42 times, for example (I am not sure that Douglas Adams / Deep Thought was thinking about that ;-)).

I can reproduce this, if I add "asdf" in the spam.py file mentioned in the "steps to reproduce" and copy this 20 times (or whatever) with a different name each time:

(python3) C:\work\pylint_test>pylint -j 0 -E test
************* Module test.spam10
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam12
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam15
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam1
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam13
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam14
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam17
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam18
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam11
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam16
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam19
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam2
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam3
test\spam3.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam5
test\spam5.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam4
test\spam4.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam20
test\spam20.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam6
test\spam6.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam7
test\spam7.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam8
test\spam8.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam9
test\spam9.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)

(python3) C:\work\pylint_test>pylint -j 1 -E test
************* Module test.spam1
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam10
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam11
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam12
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam13
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam14
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam15
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam16
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam17
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam18
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam19
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam2
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam20
test\spam20.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam3
test\spam3.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam4
test\spam4.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam5
test\spam5.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam6
test\spam6.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam7
test\spam7.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam8
test\spam8.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam9
test\spam9.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)

The test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable) error is reported 3 times, for example.

[PCManticore]

Please open a separate issue with reproduction steps as the original issue was solved and the root cause for your issue might be different.