Pylint: Performance issue after upgrade to version 1.7.1

[lemcheyWhite]

Hi, After pylint was upgraded from version 1.6.5 to version 1.7.1 I mentioned that version 1.7.1 is much slower than version 1.6.5. At least in my project analyze with pylint 1.6.5 takes about 1m17s and with pylint 1.7.1 - 4m20s against the same codebase.

My colleague did also performance testing for https://github.com/django/django.git project and results were:
1.6.5 - 7m28s
1.7.1 - 16m5s

Any suggestions about this issue?

Thank you!

[PCManticore]

Thank you. Don't know exactly what's causing this slowdown. I'll do a profiling later this week to figure out the culprit.

[moylop260]

I'll take this issue

[petervtzand]

I'm experiencing the same issue! It takes almost 3 times as long now! @moylop260 thanks for fixing this!!

[moylop260]

What is the command executed?

I just ran the basic one:

• git clone https://github.com/django/django ~/django
• time pylint ~/django/django

For 2 different versions of pylint and the result is similar:

• 

• 

[degustaf]

@moylop260 A couple of thoughts to locate these performance degradations:

1. Add --enable=all to the command line to run all possible tests.
2. Run with Python 3. Preferably Python3.6 (although pylint 1.6.5 might crash) as Python3.5 as a backup.

[degustaf]

@petervtzand Can you point us to a repo where you are seeing this performance hit, and what version of Python you are using?

[lemcheyWhite]

@degustaf Sorry that I haven't mentioned about python version in the first comment - we used python 3.5.2 for our test cases.

Also, we run pylint(1.7.1) in python2 environment for django project - and time was 2m15s (python3 - 7m48s) . Seems that the issue can be reproduced only in python3 environment.

[moylop260]

Sorry but I can't reproduce it

Could you give us the pylint --version output?
And the output of cat ~/.pylintrc? (Maybe you have a custom plugin enabled)

[craigahobbs]

I'm seeing the same issue as well. We're running Python 3.6.1, pylint 1.7.1, and astroid 1.5.2. Previously we were using pylint 1.6.5 and astroid 1.4.9 (with a couple of Python 3.6 changes backported). Our entire pylint run went from 1m8s to 2m40s on my machine. I ran pylint file by file and found that our biggest python file went from 8s to 1m35s! In that file I disabled all tests (using comments) and it still takes 1m35s.

[degustaf]

@craigahobbs Is that file available in a public repo, so that we can work on fixing this problem?

[craigahobbs]

Sadly no... let me see if I can find something publicly visible.

[craigahobbs]

I'll try profiling that file tomorrow to see where its spending its time and post-back when I have more info.

[petervtzand]

@degustaf I'm running python 2.7. I'm running it on a private repo of the company I work at.

[rogalski]

Are regressed builds using more than one job (-j N argument)?

[Virako]

I have a similar problem. The version 1.7.1 is slower than version 1.7.0. In my case, the problem is that the ignore-patterns not work in version 1.7.1

I use the next command:

pylint --load-plugins pylint_django -f parseable -d I0011,R0801 --ignore-patterns="env/" *

but I get the same results with:

pylint -f parseable --ignore-patterns="env/" *

The version 1.7.0 not check env directory.
The version 1.7.1 check the env directory.

[craigahobbs]

Its slow for me with or without -j. I've attached the profile stats of a pylint run against the single file I mentioned earlier. Use pstats to view:

import pstats
p = pstats.Stats('test.stats')
p.sort_stats('cumtime')
p.print_stats()

Wed Apr 26 11:23:11 2017    test.stats

         39539123 function calls (29746440 primitive calls) in 102.666 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  102.667  102.667 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/lint.py:1140(__init__)
        1    0.000    0.000  102.614  102.614 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/lint.py:712(check)
        1    0.001    0.001  102.613  102.613 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/lint.py:823(_do_check)
        1    0.000    0.000  102.300  102.300 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/lint.py:908(check_astroid_module)
        1    0.071    0.071   88.404   88.404 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/checkers/refactoring.py:206(process_tokens)
    16110    0.118    0.000   88.304    0.005 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/checkers/refactoring.py:217(_check_one_element_trailing_comma_tuple)
   474460   88.152    0.000   88.152    0.000 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/checkers/refactoring.py:220(<genexpr>)
  36444/1    0.124    0.000   12.711   12.711 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/utils.py:950(walk)
8133521/797149    3.516    0.000    5.816    0.000 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/astroid/node_classes.py:459(nodes_of_class)
     3064    0.051    0.000    5.574    0.002 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/checkers/typecheck.py:786(visit_call)
     5888    0.107    0.000    4.959    0.001 /home/craigh/src/egg.mortgage-mm3/build/env_test/lib/python3.6/site-packages/pylint/checkers/typecheck.py:400(_no_context_variadic)

test.stats.tar.gz

[moylop260]

Could you send us the exactly command used and the pylint --version output?
(I can't reproduce it yet so I can't fix it 😞)

[craigahobbs]

pylint 1.7.1,
astroid 1.5.2
Python 3.6.1 (default, Apr 21 2017, 14:25:30)
[GCC 6.3.0 20170406]

pylint --rcfile

[rogalski]

It seems like _check_one_element_trailing_comma_tuple is a bottleneck? It would confirm that it reproduces only in Python 3 as this code does not run at all under Python 2.x

[craigahobbs]

Seems plausible. Our code is Python 3.6.x only so I can't verify python2. Here's my profile stats dumper script - maybe someone else can test - you run it just like pylint:

import cProfile
import sys

from pylint.lint import Run

pr = cProfile.Profile()
pr.enable()
Run(sys.argv[1:], exit=False)
pr.disable()
pr.dump_stats('test.stats')

[Virako]

I think the problem is that with python3 and pylint version 1.7.1, check a lot of modules that no appear in the check directory, seems that check system modules. Put my exaclty commands:

mkdir newdir; cd newdir
virtualenv env -p python3
source env/bin/activate
pip install pylint==1.7.1
pylint --version # I get 1.6.5, I have install in my debian pylint 1.6.5, exit and enter in virtualenv
deactivate
source env/bin/activate
pylint --version # I get 1.7.1
    # output
    No config file found, using default configuration
    pylint 1.7.1, 
    astroid 1.5.2
    Python 3.4.2 (default, Oct 8 2014, 10:45:20)
pylint --ignore-patterns="env*" *
    # output
     https://pastebin.com/1r3tizck

I change my commands and use python2 with pylint 1.7.1, my outputs is:

pylint --version
    # output
    No config file found, using default configuration
    pylint 1.7.1, 
    astroid 1.5.2
    Python 2.7.9 (default, Jun 29 2016, 13:08:31)
pylint --ignore-patterns="env*" *
    # output
    No config file found, using default configuration
    ************* 
    F:  1, 0: error while code parsing: Unable to load file env/__init__.py:
    [Errno 2] No such file or directory: 'env/__init__.py' (parse-error)

I change my commands and use python3 with pylint 1.6.5, my outputs is:

pylint --version
    # output
    No config file found, using default configuration
    pylint 1.6.5, 
    astroid 1.4.9
    Python 3.4.2 (default, Oct  8 2014, 10:45:20)
pylint --ignore-patterns="env*" *
    # output
    No config file found, using default configuration
    ************* 
    F:  1, 0: error while code parsing: Unable to load file env/__init__.py: ([Errno 2] No such file or directory: 'env/__init__.py') (parse-error)
    Report
    =====
    ...

[craigahobbs]

Any update on the issue? Anything I can do to help?

[rogalski]

@craigahobbs
_biggest python file went from 8s to 1m35s_

Can you give an estimate of this file size (byte size, number of lines etc.)? It would help with designing a minimal complete input file for regression test.

Thanks/

[craigahobbs]

The file is a big run-on file with testing data. Its ~10000 lines long. It imports ~30 modules and sub-modules. At least 10%, probably more, of the file is comprised of lists of tuples.

[rogalski]

@craigahobbs Yep, sounds exactly like pessimistic case. See #1456 for fix.

If you really want to help, it would be cool if you could verify that this patch set actually fix regression observed on your code base. Simply install fixed version by running pip3 install git+https://github.com/rogalski/pylint.git@optimize-token-checker.

[craigahobbs]

Looks good. With the fix analysis completes in 10.621s.

__main__.py 1.8.0,
astroid 1.5.2
Python 3.6.1 (default, May 6 2017, 15:01:43)
[GCC 6.3.0 20170406]

[rogalski]

Fix is merged (#1456).

[IvanAnishchuk]

We just recently upgraded from 1.6.5 (to the latest 1.8.2) and it seems like the performance issue was not completely fixed. I tried different versions in between and they all are slow.

real    0m43.716s
user    2m6.944s
sys 0m2.808s
$ pylint --version
pylint 1.6.5, 
astroid 1.4.9
Python 3.5.3 (default, Nov 23 2017, 11:34:05) 
[GCC 6.3.0 20170406]

vs

real    3m25.027s
user    9m55.480s
sys 0m5.532s

for 1.8.2. I'm currently trying to collect more data before opening a new issue.