Scanpy: Pre-commit (dev workflow)

[ivirshup]

I'd like to start using pre-commit with scanpy and anndata. Pre-commit is essentially a tool that manages scripts we'd like to run before each commit, e.g. linting and formatting, so it becomes essentially impossible to forget these.

I think this can allow PRs to progress faster since it gives us a way to codify formatting requirements – so we don't have to remember them – and have these checks happen locally – so we don't have to wait on CI. Of course, having these checks run depends on developers installing pre-commit, so we can also run these checks on CI (example ci script, example run).

There is a question of what things we'd like to add here. For sure: black. I think import checks (e.g. no unused imports) and flake8 would be good too. We can also add custom checks for things like slow imports.

I think this would be a good time to run black over the whole codebase so we don't have exempted files any more.

My questions for the dev team:

• Does this sound good?
• Do you have more ideas for checks/ tools?

@michalk8, I saw you added this to squidpy. How's the experience been there – that is, any major foot guns we should look out for? Also, are there any tools you're using (beyond the basic black, isort, flake8) you'd especially recommend?

[LuckyMD]

I like this idea a lot. I recently learned about automated linting and blacking of code per commit and have started using it for the single cell open problems project upon the suggestion of @scottgigante.

[michalk8]

@ivirshup

I'd recommend these:

• mypy for type checking (I've caught a few bugs using it)
• autoflake to remove unused variables/imports (by default removes only imports from Python's standard library)
• yesqa to automatically remove unused # noqa for flake8
• check-yaml and pretty-format-yaml I also found useful
• requirements-txt-fixer to sort the reqs.
• check-added-large-files have prevented my from commiting some .h5ad files that shouldn't be there
• rstcheck to check the syntax of .rst files
• some flake8: pytest-style (enforces some pytest conventions, like non-returning fixtures beginning with _), blind-except (no blanket exceptions) and comprehensions (helps you rewrite comprehensions)

Here's also an exhaustive list from which I picked the ones I use: https://pre-commit.com/hooks.html

As for any problems, some of them came from rstcheck as
docs/source/classes.rst:9: (INFO/1) No directive entry for "autoclass" in module "docutils.parsers.rst.languages.en"., that's why .rstcheck.cfg might be necessary. Also fixing types for mypy takes a while, I'd do it as last.
Also flake8-comprehensions forces you to abandon dict(foo=..., bar=...) calls, unless you disable it.
And [tool.isort] in pyproject.toml should have profile = "black".

I had also planed to use pylint to enforce naming conventions, etc., but it has large overlap with flake8 + the config file can be rather large, see https://gist.github.com/dkatz23238/08d79a76911ddaaa6171dff9cddd5772

[Koncopd]

I really don't want it to be mandatory. It is not a rational objection, i just prefer my code to be committed as i wrote it and then run some formatting scripts separately if needed.

[ivirshup]

@michalk8 thanks for the extensive recommendations!

I think I'd like to keep the number of tools used small. It's the worst when you want to fix a bug, but instead have to learn about configuring a linter. More tools means more configurations people need to be familiar with, and the goal is reducing cognitive load.

Also fixing types for mypy takes a while, I'd do it as last.

Yeah, I figured this would be the case. Does mypy allow partial typing these days? Also, I haven't found the numpy or pandas type stubs to always be great. Have you run into problems around this?

I think this would also need to wait at least until we can drop python 3.6 for anndata, since adding types there currently means circular dependencies.

rstcheck to check the syntax of .rst files

I would particularly like a linter for rst. I noticed you also had doc8, but you'd recommend rstcheck check over this? I'm a little worried, considering its last release was over a year ago.

Spell check for prose in doc-strings could also be great, but I could see this being overzealous (is there a good way to notify about misspelled words, while not being annoying about technical terms?)

I'm a little worried about some custom sphinx extensions we have, and conflicting with this, any experience here?

---

@Koncopd, I think I agree with your concern, as I said above: it's the worst when you want to fix a bug, but instead have to learn about configuring a linter. I also think it's very easy to add new checks, so someone complaining about new ones is valuable.

Per commit, this should always be an option with git commit --no-verify, though you could also just not install pre-commit.

I would like to keep the required checks limited, ideally formatting tasks that can be automated as opposed "this is poor style" warnings. I also know these tools can be wrong (e.g. black when expression's have many operators, sometimes with chaining) so it would be good to document the escape hatches for this (# fmt: off for black).

That said, we already do require that merged code goes through black before it gets merged, and a benefit of using this would be to not have commit messages like "formatting", "remove unused import", etc. The pre-commit checks would be a part of CI as well, so it would be eventually mandatory – just not on your machine.

Does this address your concerns?

[LuckyMD]

I think I misunderstood... I thought you were suggesting things like autoblack in github actions for commits.

[giovp]

If I can jump on this, talking by personal experience, it would be a very very useful tool for contributors, especially young/inexperienced ones (like me!).

In squidpy @michalk8 put together a very comprehensive check list in pre-commits, and I'm appreciating it more and more as I get familiar with it.
yes, there is a lot of cognitive load at the beginning, and yes it can be very (very) painful, but when you get used to it, it soon becomes essential and actually really useful.

Only concern of course is that it highers the bar for contributions in the repo, but honestly I'm seeing it being adopted in other large bio-related oss (e.g. https://github.com/napari/napari ). I think this can be simplified by having an extensive contributors guide, and the explicit mention on how to skip pre-commits and submit the PR anyway (and then otehr scanpy dev can jump in and give suggestions on why precommits failed).

[flying-sheep]

@Koncopd pre-commit doesn’t have to be configured, you can choose not to enable it. Of course tests will fail then.

regarding mypy: I guess it’s possible to make everything return -> t.Any: # TODO fix typing

I like isort!

Also we should get #1527 in before doing any big restructuring: It’s been through too many rounds of delays and I had to resolve conflicts so many times.