Pip-tools: Type annotations

[atugushev]

What's the problem this feature will solve?

It would be nice to have type annotations throughout the codebase. This issue is a general tracking issue of how we would use mypy and typings.

Describe the solution you'd like

• use pytest-annotate to auto add typings (or something similar)
• add typings incrementally, one PR per module (e.g., 21b713018d2dba6efef0e3d4e71facac1c4f0b38) to help reviewers

Alternative Solutions

Wait until we drop Python 2 support and add Python 3 style type annotations.

Additional context

Docs:

• https://www.python.org/dev/peps/pep-0484/
• https://realpython.com/python-type-checking/

Useful tools:

• https://github.com/dropbox/pyannotate
• https://github.com/kensho-technologies/pytest-annotate

Progress

Maintenanse

• [X] add mypy checking to pre-commit config (#1275)
• [X] add [mypy] section to setup.cfg (#1275)

Add typings to the following files

• [x] piptools/_compat/pip_compat.py (#1315)
• [x] piptools/cache.py (#1315, #1316)
• [x] piptools/exceptions.py (#1315)
• [x] piptools/logging.py (#1315)
• [x] piptools/repositories/base.py (#1315)
• [x] piptools/repositories/local.py (#1341)
• [x] piptools/repositories/pypi.py (#1341)
• [x] piptools/resolver.py (#1316)
• [x] piptools/scripts/compile.py (#1322)
• [x] piptools/scripts/sync.py (#1331)
• [x] piptools/sync.py (#1331)
• [x] piptools/utils.py
• [x] piptools/writer.py (#1310)

Final steps

This list must be done after the above lists.

• [x] add disallow_untyped_defs = True to setup.cfg (#1350)

[atugushev]

Hey, @jazzband/pip-tools. Please, join the discussion. Have you any other thoughts or helpful tools we could use to integrate type annotations?

[webknjaz]

Could probably use https://github.com/Instagram/MonkeyType to collect types from runtime.

P.S. I think, it's better to use a separate mypy.ini rather than flooding setup.cfg with things that aren't related to packaging.

[codingjoe]

@atugushev I don't want to discourage you, but I believe effort and return might be a bit out of balance. I don't think there are too many people that use the pip-tools Python API. For most people this is a nice CLI tool. The only people who would benefit from type hints however are people who use the Python API are developers of this package. That being said, good IDEs like PyCharm will give you type hints, whether they are explicit added or not. If this is only about you wanting type hints to develop pip-tools and for some reasons don't have a PyCharm license, you can always apply for an open source license ;)

I'm really just dreading the immense workload this will create especially for the poor soul reviewing it :P

P.S. I think, it's better to use a separate mypy.ini rather than flooding setup.cfg with things that aren't related to packaging.

@webknjaz I think, I actually prefer having everything in the setup.cfg. For me it makes packaging easier, since I don't have to prune too many files from dists. That being said, there are also a bunch of tools like Sphinx and the PyTest runner that encourage you to use the setup.cfg, since both command are actually run via setup.py.
Then again, it does boil down to taste. I'm am just saying there are may flavors ;)

[atugushev]

@webknjaz

Could probably use https://github.com/Instagram/MonkeyType to collect types from runtime.

AFAIK, MonkeyType doesn't support py2 typings comments so that we can afford for now py3 typings.

P.S. I think, it's better to use a separate mypy.ini rather than flooding setup.cfg with things that aren't related to packaging.

Personally, I'd prefer to have the tools configs in the setup.cfg, and I'm quite sure we had discussed it earlier with members (can't find references though, probably with @blueyed?) to collect all configurations there. But I understand your intention, and it seems pretty reasonable. Would you like to file a separate issue so we can discuss it?

[atugushev]

@codingjoe

@atugushev I don't want to discourage you, but I believe effort and return might be a bit out of balance. I don't think there are too many people that use the pip-tools Python API. For most people this is a nice CLI tool. The only people who would benefit from type hints however are people who use the Python API are developers of this package. That being said, good IDEs like PyCharm will give you type hints, whether they are explicit added or not. If this is only about you wanting type hints to develop pip-tools and for some reasons don't have a PyCharm license, you can always apply for an open source license ;)

Actually, I do have PyCharm open source license, and I'm happy to have this opportunity!

My thoughts on typings:

1. it gives more context to the function/method signature and would help contributors to understand the code
2. mypy is an extra eye on the code quality, it really helps to find bugs sometimes
3. typings makes the code more strict

IMO it's worth it. The codebase is not so large (comparing to pip), so it's not going to be too hard to adopt typings, thanks to auto type tools!

[aleksihakli]

Just my 2c: adding explicit typing to the codebase is not that big of a job and definitely makes the code more understandable and maintainable :)

[codingjoe]

OK if it helps people, sure, I'm not against it.

[sfdye]

Alternative Solutions
Wait until we drop Python 2 support and add Python 3 style type annotation

TBH I'd prefer this approach

So my question is when are we gonna drop Python 2 support? the discussion on Packaging and Python 2 is so long that I don't have the time to go through them all 😛

But it's March 2020 now, 3 months into 💀 of Python 2.7.

[atugushev]

@sfdye

So my question is when are we gonna drop Python 2 support?

As soon as pip does.

[codingjoe]

As soon as pip does.

So never 😉

[atugushev]

@codingjoe

So never 😉

BTW, pip==21.0 (Jan 2021) will be the last version with python 2.7 support, see https://github.com/pypa/pip/pull/8087.

[hugovk]

BTW, pip==21.0 (Jan 2021) will be the last version with python 2.7 support, see pypa/pip#8087.

Correction: pip 21.0 (Jan 2021) will be the first version to drop support for Python 2.7.

pip 20.3 (October 2020) will be the last version to support Python 2.7.

https://github.com/pypa/pip/issues/6148#issuecomment-616213532

[codingjoe]

Yes, finally. I sure you can tell, but boy am I looking forward to Python 2 dying ☠️

[pradyunsg]

Why not add # type: comments?

[codingjoe]

Why not add # type: comments?

I guess there are many ways to add type hints via doc strings. Sphinx annotation (including napoleon) have been around for a while and are supported by all good IDEs.

I would probably go with Google Style annotations, since they are human and machine-readable. After all, we make documentation for humans not machines. That being said, I don't see a huge advantage of using adding type hints. This isn't a package that a lot of people interact with on a Python API level, but rather via CLI. Therefore, I am only in favor of adding type hints, if they do not slow down regular development or otherwise introduce complexity.

[atugushev]

I'd prefer to wait with patience until pip drops Python 2.7 support and then we can use type hinting with a full power!

[pradyunsg]

I guess there are many ways to add type hints via doc strings. Sphinx annotation (including napoleon) have been around for a while and are supported by all good IDEs.

Except, those aren't supported by mypy or pytype; neither are they "blessed" by a PEP describing them. :)

Anyway, @atugushev's approach also works well.

[sfdye]

So that's 2021 at least

[jdufresne]

Given that we expect pip to drop Python 2 support in the new year, what do you think about cutting one last pip-tools release and then we continue working towards type hinting the code base today? There is already a PR out there to drop Python 2 #1243.

Doing it now would allow people with some extra free time over the holidays to get some of this work done.

WDYT?

My motivation for including type checking is that it helps provide additional confidence that APIs are being used correctly (both internal and external) during large upgrades and refactorings. I'm happy to lend a hand in adding types to the code incrementally.

[codingjoe]

I would also be for one final Python 2 release and then to drop support it like a hot potato.
Gradual changes seem most practical, no one wants to review and release a 10kloc patch.

[atugushev]

I'm going to prepare the 5.5.0 release soon and then we can drop 2.7/3.5 and finally add type hints!

[atugushev]

Started to work on type hints #1310. Any help with review or code is appreciated ❤️