Pylint: --fail-under flag

[pckroon]

Is your feature request related to a problem? Please describe

Hi all,

I like running pylint on my codebase on Travis as a check, but I think it's unreasonable to fix all the warning, errors and other messages pylint issues (I do want to see them though). What I would like is to have a 0 exit code if and only if the final score is more than a certain value (say, 9). As time progresses and the project matures I can then increase that score threshold to higher and higher values.

Currently, so long as pylint finds any problem with my code and issues a message, it exits with a non-zero exit code, signalling failure to Travis.

Describe the solution you'd like

A --fail-under <score> flag, also configurable in a .pylintrc file. If the final score is more than the specified score, it's considered a success and pylint exits with exitcode 0. Otherwise, it's considered a failure and pylint exits with it's current exitcode based on the messages issued.

Current work-around

At the moment I call pylint as following in Travis, but that trashes the actual usefull output. And is hard to read.
test $(bc -l <<< $(pylint --disable=fixme vermouth | sed -re "s_.* ([0-9\.]+)/10.*_\1>9_;t;d")) -eq 1

Thanks for all the hard work so far, keep it up! :)

[brycepg]

You can do this programatically with a little python script which won't interfere with stdout:

import sys
from pylint import lint

THRESHOLD = 9

if len(sys.argv) < 2:
    raise ArgumentError("Module to evaluate needs to be the first argument")

run = lint.Run([sys.argv[1:]], do_exit=False)
score = run.linter.stats['global_note'] # Yes this is a terrible name for the score

if score < THRESHOLD:
    sys.exit(1)

[PCManticore]

This is a bit too specific and it's not a popular feature request, I definitely suggest using a wrapper as the one that Bryce wrote. I also think that checking the exit code is going to be a bit manageable, it's better to disable all the errors you want upfront and later on you can enable selectively the errors that you're prepared to have in your CI build.

[pckroon]

Thanks, that's a much neater work-around :) Keep up the good work!

[pckroon]

For posterity, here's a slightly polished version:

import argparse
import sys
from pylint import lint

desc = "PyLint wrapper that add the --fail-under option."\
       " All other arguments are passed to pylint."
parser = argparse.ArgumentParser(description=desc, allow_abbrev=False)
parser.add_argument('--fail-under', dest='threshold', type=float, default=8,
                    help='If the final score is more than THRESHOLD, exit with'
                    ' exitcode 0, and pylint\'s exitcode otherwise.')

args, remaining_args = parser.parse_known_args()

threshold = args.threshold

run = lint.Run(remaining_args, exit=False)
score = run.linter.stats['global_note'] # Yes this is a terrible name for the score

if score < threshold:
    sys.exit(run.linter.msg_status)

[ChristopherRabotin]

Any chance this can be re-opened? I'd be happy to code up the fix myself.

This would be invaluable in an organization where most folks are resilient to even using git and CI... The on-boarding must be as easy as possible, and providing a simple .pylintrc file along with three lines to add to the .yml file is perfect.

[ChristopherRabotin]

Here's a work-around which uses a combination of SH and python:
CI file:

py101:
  variables:
    MODULES_AND_FILES: "*.py"
  script:
    - (pipenv run pylint --rcfile=".pylintrc" ../../$MODULES_AND_FILES | tee .pylint.log) || python .pylint_check_score.py
    - rm .pylint.log

And .pylint_check_score.py :

'''
PyLint score checker. Returns 0 if the pylint score is greater than the one
specified below. This script allows us to use pylint normally with all the
standard paramaters.
'''
from re import findall
from sys import exit as sysexit

MIN_SCORE = 7.0

with open('.pylint.log') as plfh:
    score_line = plfh.readlines()[-2]
    score = float(findall(r'([0-9\.]+)/10', score_line)[0])
    sysexit(0 if score > MIN_SCORE else 1)

[pckroon]

@ChristopherRabotin You can find my version here: https://github.com/marrink-lab/vermouth-martinize/blob/master/run_pylint.py
And you can call it like such: python run_pylint,py --fail-under=7, which will do as you expect, while still passing through all command line arguments, and without needing output redirection.

[akhilputhiry]

@PCManticore @brycepg Please consider re-opening this. This is a useful feature. coverage also has similar feature. It is not that complicated.

[sema4-EricSchles]

seconded. Getting to 10/10 is giant time suck. I have a second possible option here. I want to say first and foremost, linting is a great thing and pylint is amazing. And that my following comments might be on me for being uninformed. So here goes!

It is very hard to figure out how to turn off specific linting errors. They don't appear to correspond to the linting .pylintrc directly. Yes, the names are clear, once you know them. But I've had a lot of trouble finding a mapping to each and every linting error and how to silence it. Would it be possible to point to such a resource if it exists? If not, would it be a good idea to open a new issue and get one created? I think this is actually at the heart of asking for a score below 10/10.

I hope that's not too harsh! Thank you all for your wonderful work. Pylint is a great tool that I plan on using for my entire career. You've really made something special here.

[NathanDotTo]

This seems like such an obvious feature that I am quite puzzled that this is not possible.

[PCManticore]

@sema4-EricSchles I think http://pylint.pycqa.org/en/latest/user_guide/message-control.html and http://pylint.pycqa.org/en/latest/technical_reference/features.html would be useful to you since they show what messages we emit and how to control them.

@NathanDotTo and others that replied here, do you find filtering by the score that useful in your use of pylint? The score is pretty much a gimmick for gamifying linting and it's not at all similar with what other tools are doing, such as coverage. Yeah, you might have 90 % coverage and that's good enough, but does a 9 / 10 from pylint actually indicate what might wrong with your code? You can have a serious bug that pylint catches and get a 9 / 10, and that score doesn't properly propagate that information.

I believe a more manageable way of addressing code quality via pylint is to have a custom configuration file, in which you control exactly what messages you expect to be emitted and what messages you expect to be disabled, in which case the score becomes moot.

[sema4-EricSchles]

The second reference you mention is exactly what I was looking for! Thanks @PCManticore! Now for the ask - can you make the second reference easier to find? If I wasn't handed the link, I think it'd have trouble finding it. I tried to find it by going to the index and searching around and it took me about 10 minutes to find it on my own (well passed when most would give up, I'd wager). I know writing docs is hard. Whenever I write my own, I'm sure people find them less than helpful often. Maybe this could have greater mention in the docs?

[radeklat]

@PCManticore Let me chip in with explanation why this feature would be useful.

I introduced pylint into 10+ projects in our company. Developers often switch among these projects so we have one set of rules for all projects that we would like the code conform to. However, each project has a different degree of conformity to the common rule set.

Teams have a mixture of junior and senior devs. It is unrealistic to catch everything in PRs, so we rely on pylint heavily. And honestly, the main reason for using pylint is that we don't have to catch common problems in PRs. But at the moment, pylint is all or nothing for each issue category. Fixing all reported issues at once in a some categories is not realistic and new issues accumulate quickly.

So the best option would be to improve the quality gradually by enforcing minimum score and over time bring the bar up. While it would be still completely random what will and will not be fixed, the number of issues would go down over time. (Btw. the same logic can be applied to coverage as it also doesn't say anything about tests quality.)

The script above kind of helps but seems like an overkill of a simple command line option that could do the same.

[PCManticore]

Hey @radeklat Thanks for the additional argument and examples. After reading your comment, I had a change of heart and I could see this feature being useful for large adoption of pylint in various codebases. I already mentioned my favourite flow of integrating pylint in a codebase in a past comment, but as you mentioned, sometimes it's not possible to fix all messages of a given kind, and that's where this new flag comes in. Thanks again.

[syutbai]

I will try and sprint on this issue.

[kasuteru]

I just wanted to chime in that I am really happy about having that flag as well. I am trying to introduce more of our people who code to pylint, and the acceptance of "Let's get above score X" is way higher than requiring a perfect score. As others have mentioned, gradually raising the bar seems to be a very important acceptance criteria.

[pckroon]

I'm very happy to see this became an option in pylint, however I don't think this is fixed yet, since the flag only accepts ints, while IMHO the score is a float. Currently I can't do e.g. pylint --fail-under 9.5 my_project. It's probably a super easy fix in https://github.com/PyCQA/pylint/blob/f00144fb8faa81e1ce42516c2f58eb9fc797f7bd/pylint/lint/pylinter.py#L263

[PCManticore]

@pckroon Thanks, that's a good concern. Can you open up a separate issue? Also if you have some time, sending a PR would also be appreciated!

[pckroon]

I should have some time for both later today :)

[jakeBass]

@PCManticore

Hi! I know this issue has since been taken care of, but I wanted to mention a related use case that I have that I'm not sure is currently possible. @radeklat spoke persuasively of the need to improve codebase quality over time, and how gradually raising the score threshold over time ensures quality improvement.

I would like to do something similar - I would like to fail only if the score is worse than the previous run (this is only possible if --persistent=y, which I believe is the default). While it is possible to manually increase the failure threshold with each run, this is difficult when using pylint in a CI system. This approach would codify the rule of "let's make sure code quality improves (or at least does not degrade) with every commit", which is what a lot of folks in this thread seem to be looking for.

Am I missing this option and this functionality is currently possible? If not, what do you think of this idea?