Gitpython: Type Hints

[mdantonio]

Hello,

I was looking for any type hints for GitPython, but I found nothing neither in the code nor in independent stub repositories.

Is there any plan to introduce type hinting?

[Byron]

I am absolutely open to receiving PRs adding type hints, and GitPython would be happy to host them directly in source.

Any amount of types would help and is appreciated, thus there is no need to add type hints everywhere at once.

[Yobmod]

I looked at starting to contribute type hints, but wanted to ask would it be feasible to drop support for python 3.4 (EOL 2 years ago)?

With 3.5+ type hints can be added directly to the code, but for 3.4 they would need to be in .pyi files, doubling the number of files and requiring they are kept in sync (AFAIK, 3.4 will give errors rather than ignore most inline type hints even though annotation syntax is possible)

[Byron]

@Yobmod Thanks for bringing this to my attention, and I would be glad to see type hints in code. Please feel free to submit a draft PR that starts out by dropping PY3.4 support to allow adding the first type hints in code. That one wouldn't have to be complete at all, but could lay the foundation for more PRs adding types in future.

[Yobmod]

I've put in a PR to drop python 3.4.
Planning to use Monkeytype to add initial types to the most user-facing files and see how that goes.

[Yobmod]

A plan:

• [x] Drop 3.4 (add 3.9)
• [x] Final 3.4 release
• [x] Monkeytype using test suite
• [x] Add monkeytype.sqlite3 and mypy.ini to .gitignore
• [x] Apply initial types to Repo/base & Repo/fun
• [x] Add mypy.ini
• [x] Add py.typed
• [x] Appy initial types to git/*
• [x] First typed release
• [x] Appy initial types to index/*
• [x] Appy initial types to refs/*
• [x] Appy initial types to objects/*
• [x] Increase mypy strictness
• [x] Expand/Refine types (use overloads, Literals, TypedDicts etc)
• [x] Make IterableList Generic & update throughout
• [ ] second typed release
• [x] Drop 3.5 (add 3.10)
• [x] Fix blocking issues caused by 3.5 compat

[mdantonio]

Thank for all the effort towards type hints! I am sorry I have not been able to contribute so far, I hope to be able to put some effort after the great initial start of @Yobmod

[Yobmod]

Thank for all the effort towards type hints! I am sorry I have not been able to contribute so far, I hope to be able to put some effort after the great initial start of @Yobmod

Thanks, Any help is always good :)

Branch for adding types is here: https://github.com/Yobmod/GitPython/tree/addtypes

Mypy-stubgen just typed everythin as Any, and Monkeytype has not been as useful as i hoped. If any wants to run it, commands are:

• monkeytype run test/tstrunner.py
• monkeytype list > mtype_filelist.txt
• monkeytype stub git.cmd > git/cmd.pyi

But i'm mostly doing them by hand.

[kasium]

Hey, it seems that most of the code is typed, right? Would it be possible to include a "py.typed" file into the repo so that mypy can pick the types up?

[Yobmod]

Hey, it seems that most of the code is typed, right? Would it be possible to include a "py.typed" file into the repo so that mypy can pick the types up?

Hi, there already is one. Are you on the newest version?
Annotations showing up for me.

[kasium]

Yes I use the latest version. The py.typed file is not present in the tar.gz nor in the wheel - maybe that's the issue.
Mypy (also latest version) complains that no stub could be found

[Yobmod]

I've read up on some setuptools options, it seems that the package_data and include_package_data arguments are mutually exclusive. So setup.py is picking up the package_data only from the MANIFEST.INI and not from within the setup.include_package_data list.

I'll add py.typed to the manifest and hopefully that fixes it.

[kasium]

Great. thanks a lot

[Yobmod]

I've installed from git and it puts the py.typed where it should be now.
If anyone wants the typehints before the next release, can do:

pip install git+https://github.com/gitpython-developers/GitPython.git -U

[kasium]

@Yobmod thanks for the work here. Are there any plans when 3.1.19 will be released?

[Byron]

@kasium Judging from the checklist further up all work seems to be done waiting for everything to be released. @Yobmod - do you think a release is in order?

[Yobmod]

Yep, we could do a release now.
I avoided any api changes, but lots of small runtime changes (None checks, isinstance(), etc), so would be good to find out if they trip up on any edge cases, which I think only a release can do.

The next step for typing is to check the runtime types to fix the remaining unknowns. Unfortunately I cant find a library to do it (Typeguard etc. will report if using the wrong type, but not if the type is too broad), so it will take a while.

[Byron]

And here is the new release.

Fingers crossed!

[adeandrade]

    import git
  File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in <module>
    from git.exc import *                       # @NoMove @IgnorePep8
  File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in <module>
    from git.compat import safe_decode
  File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in <module>
    from git.types import TBD
  File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in <module>
    from typing_extensions import TypeGuard  # noqa: F401
ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py)

@Byron Just got this with 3.1.19 and Python 3.9.6. Maybe we should yank it?

[Yobmod]

It looks like typing_extensions has to be updated (again) to 3.10.0.0.
That fixed it on my machine, so I've push the commit for requirements.txt.

Typehints still seem to be changing every few months, even after 7 years!

Dominic

On Fri, 23 Jul 2021, 04:45 Anderson de Andrade, @.*>
wrote:

import git

File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in
from git.exc import * # @NoMove @IgnorePep8
File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in
from git.compat import safe_decode
File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in
from git.types import TBD
File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in
from typing_extensions import TypeGuard # noqa: F401
ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py)

@Byron https://github.com/Byron Just got this with 3.1.19 and Python
3.9.6. Maybe we should yank it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gitpython-developers/GitPython/issues/1095#issuecomment-885378324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFJUSQ3XHNB6XABCFOWJSUDTZDQUVANCNFSM4VD23UVQ
.

[mdantonio]

Hello,
installed v3.1.19 (and upgraded typing_extensions to 3.10.0.0) but type hints are still ignored by mypy because py.typed seems to be yet not included in the release

Here the content of the installation folder:

$ ls /usr/local/lib/python3.9/dist-packages/git/
cmd.py  compat.py  config.py  db.py  diff.py  exc.py  index  __init__.py  objects  __pycache__  refs  remote.py  repo  types.py  util.py

$ grep "3.1.19" /usr/local/lib/python3.9/dist-packages/git/*
grep: /usr/local/lib/python3.9/dist-packages/git/index: Is a directory
/usr/local/lib/python3.9/dist-packages/git/__init__.py:__version__ = '3.1.19'

type hints started to work after a touch /usr/local/lib/python3.9/dist-packages/git/py.typed

[Byron]

Interesting this wasn’t caught by CI.
Yanked the release.

Sent from my iPhone

On Jul 23, 2021, at 5:17 PM, Dominic @.*> wrote:


It looks like typing_extensions has to be updated (again) to 3.10.0.0.
That fixed it on my machine, so I've push the commit for requirements.txt.

Typehints still seem to be changing every few months, even after 7 years!

Dominic

On Fri, 23 Jul 2021, 04:45 Anderson de Andrade, @.*>
wrote:

import git
File "/opt/python/lib/python3.9/site-packages/git/__init__.py", line 8, in
from git.exc import * # @NoMove @IgnorePep8
File "/opt/python/lib/python3.9/site-packages/git/exc.py", line 10, in
from git.compat import safe_decode
File "/opt/python/lib/python3.9/site-packages/git/compat.py", line 32, in
from git.types import TBD
File "/opt/python/lib/python3.9/site-packages/git/types.py", line 18, in
from typing_extensions import TypeGuard # noqa: F401
ImportError: cannot import name 'TypeGuard' from 'typing_extensions' (/opt/python/lib/python3.9/site-packages/typing_extensions.py)

@Byron https://github.com/Byron Just got this with 3.1.19 and Python
3.9.6. Maybe we should yank it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gitpython-developers/GitPython/issues/1095#issuecomment-885378324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFJUSQ3XHNB6XABCFOWJSUDTZDQUVANCNFSM4VD23UVQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Yobmod]

😭
I've looked at loads of example libraries and instructions, and they're all different.

Encode/starlette have py.typed listed in setup.py package_data, but not in manifest.in, and that works.
That what mypy instructions say but it didn't work last release.

Encode/httpx (issue linked above) had same issue, so added py.typed to manifest.in.
So we did that, and it worked for installs from github, but it seems not from pypi.

I've now added it to both setup.py and manifest.in (DRY is so 1980s) to see if that fixes it (based on https://stackoverflow.com/questions/3596979/manifest-in-ignored-on-python-setup-py-install-no-data-files-installed)

If that doesn't work, the only thing I can think of is something to do with the installed lib having a different name? (git vs gitpython).

Maybe should release a beta with the fixes I commited this morn? Then I can install from pypi. I can't think of how else to test this?
Or I'll release as a clone package and test that one.

[Yobmod]

Yeah, CI doesn't error because it only happens with pypi install into already existing environments, but CI is making fresh venvs with the newest available libs each time

[mdantonio]

mypy docs (see the very last paragraph in https://mypy.readthedocs.io/en/latest/installed_packages.html#creating-pep-561-compatible-packages) suggests:

global-include *.typed

in MANIFEST.in

But i think that include py.typed should work as well...

(However, none of the files in the MANIFEST.in are included in the package... it seems that the MANIFEST is ignored at all)

[Byron]

Yeah, CI doesn't error because it only happens with pypi install into already existing environments, but CI is making fresh venvs with the newest available libs each time

Could we put a a new test on the list of things to do that builds the package and tries to install it, without uploading it to pypi? I am asking because faulty releases are now the norm instead of the absolute exception, and some prior warning would definitely be valuable.

I am also asking the folks from streamlit in case they want to contribute such a check.

But i think that include py.typed should work as well...
(However, none of the files in the MANIFEST.in are included in the package... it seems that the MANIFEST is ignored at all)

The whole packaging setup might need some cleanup. When I did that more than 10 years ago it was already terrible to work with and a mess from the start. This didn't improve much over the years apparently but maybe there is better ways now that are less error prone.

[Yobmod]

We could make a github actions that installs the lowest versions of dependancies, pushes a release to testPyPI in one step, then installs from there in the next step and does some tests. https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/. So in the last couple of releases it would not have updated typing_extensions and gotten the ImportError.

I finally found the problem in setup.py! After about 20 uploads to testpypi with different settings.

• Manifest.in was working properly, and picks up the listed files (including py.typed) for sdist and puts them in the tarball with the package and egg-info.
• But bdist was not picking up py.typed even with use_package_data=True or package_data={'git': ['py.typed']}
• The problem was find_packages() was returning [].
• It turns out exclude=(("tests*")) has to be a sequence of excluded folders, but didnt have a trailing comma to make it a tuple
• Changing it to tuple or list, and it now returns ['git', 'git.repo', etc] and bdist picks up py.typed from the 'git' folder as it should.

Edit: That fixed the py.typed issue, but typing-extensions v3.10.0.0 still giving problems. Testpypi says no distribution found when its a dependancy in requirments.txt or setup.cfg even without the python version limit.
Ive reverted it to v3.7.4.3 and removed Typeguard throughout.

[Byron]

Absolutely fantastic work! I definitely couldn't have done it!

After about 20 uploads to testpypi with different settings.

This looks like you saved us 20 or so broken releases, great that putting this kind of test on CI worked so well.

It turns out exclude=(("tests*")) has to be a sequence of excluded folders, but didnt have a trailing comma to make it a tuple

I suppose compile-time typing would have helped ;). No, it's a nice show of how single-character bugs usually are the hardest to find.

Testpypi says no distribution found when its a dependancy in requirments.txt or setup.cfg even without the python version limit.

Probably something related to dependency resolution and complex interplay between test.py, the python version as well as typing extensions.

Maybe once python 3.6 can be dropped, it will get better, easier? Maybe it's worth to create an issue on their end to see if somebody knows what this is.

[Yobmod]

I suppose compile-time typing would have helped ;). No, it's a nice show of how single-character bugs usually are the hardest to find.

Yeah, if setuptools was type-hinted, we would have found it out immediately!

I'm just testing the install on windows, then can be released

[Byron]

GitPython https://pypi.org/project/GitPython/3.1.20/ was just released.

Let's hope it works out, otherwise it will be yanked. I should be able to respond to issues within the next 12h or so.

[mdantonio]

It works! :smiley:

$ ls /usr/local/lib/python3.9/dist-packages/git | grep py.typed
py.typed

and mypy started to report type errors, e.g.

from git import Repo

repo = Repo(10)

...

error: Argument 1 to "Repo" has incompatible type "int"; expected "Union[str, PathLike[str], None]"

Thank you very much!

[mdantonio]

Type checked all my functions that are using gitpython (removed several Any and cast, I'm super happy!) and this is the only problematic case that I found:

I have a function that can return gitobj.active_branch.name, but mypy complains:

Returning Any from function declared to return "Optional[str]"

At runtime i got a str from active_branch.name so probably it is still un-annotated

[kasium]

Hey,

thanks a lot for the progress here. However here is an issue I hit:
@Byron

    import git
venv/lib/python3.7/site-packages/git/__init__.py:42: in <module>
    from git.config import GitConfigParser  # @NoMove @IgnorePep8
venv/lib/python3.7/site-packages/git/config.py:48: in <module>
    from typing import OrderedDict
E   ImportError: cannot import name 'OrderedDict' from 'typing' (/usr/lib64/python3.7/typing.py)

This was added in 3.7.2 but I run on 3.7.1. Can you please improve the check in the file and include the patch level as well or you could get this type from typing_extensions

[Byron]

I wonder if it's even possible to get this right for everyone.

Can you please improve the check in the file

I tried but it looks like it's not the obvious fix that's needed here. From what I know typing_extensions came with their own issues, but @Yobmod will know more.

Maybe pin your version to the previous known one in the meantime, let's keep the bait out and see if there is other issues that we try to address in the next release, before yanking this one.

[kasium]

Thanks for your feedback and fast answer. It seems like that your fix seems sufficient for me

[Byron]

You are welcome! It's interesting this works for you at the CI seems to be having a problem.
Probably it's just me, I will leave fixing this to the professionals :D.

[Yobmod]

I think the more version specific guard is the correct fix, mypy just complaining because it can't cope with the same variable being defined 3 different ways for different pythons.

[Yobmod]

Type checked all my functions that are using gitpython (removed several Any and cast, I'm super happy!) and this is the only problematic case that I found:

I have a function that can return gitobj.active_branch.name, but mypy complains:

Returning Any from function declared to return "Optional[str]"

At runtime i got a str from active_branch.name so probably it is still un-annotated

What is the gitobj?
The only active_branch.name in the code base is an attribute of Repo, but that is typed (as Union[str, PathLike[str], which ive changed to just str).
Or if it was a SymbolicReference of some type, that has a name property that was missing a type, now added.

For some of the Git comand wrappers, they are too dynamic to add types (so default to Any), until/unless someone adds specific methods/overloads for each command.

[mdantonio]

gitobj is an instance of Repo

Here a minimal test case to reproduce the issue:

from git import Repo

def get_active_branch(gitobj: Repo) -> str:
    return gitobj.active_branch.name

gitobj = Repo(".")
print(get_active_branch(gitobj))

$ mypy --warn-return-any active_branch_test.py 
active_branch_test.py:4: error: Returning Any from function declared to return "str"
Found 1 error in 1 file (checked 1 source file)

[Yobmod]

Ah, no error for me.

What does mypy show for reveal_type(gitobj.active_branch)?
For me it is 'Head' from your snippet
and reveal_type(gitobj.active_branch.name) is str.

So maybe adding type hint to symbolicReference.name has fixed it in main? (Head <- Reference <- SymbolicReference)

[mdantonio]

$ mypy --warn-return-any active_branch_test.py 
active_branch_test.py:5: note: Revealed type is "git.repo.base.Repo"
active_branch_test.py:6: note: Revealed type is "git.refs.symbolic.SymbolicReference"
active_branch_test.py:7: note: Revealed type is "Any"
active_branch_test.py:8: error: Returning Any from function declared to return "str"
Found 1 error in 1 file (checked 1 source file)

[Yobmod]

Thx for checking. Then it is fixed for next release :)

[Byron]

Thanks everyone for bearing with us. Yesterday after the second issue was filed I yanked 3.1.20, so it looks like that the issues around older python versions and ordered dict is all there is for now.

@Yobmod What do you think about setting up CI to test patch releases as well, to have more probes when determining that the type related changes are actually working for everyone? Typing seems to be a minefield when trying to support as many python versions as we do, and I have a feeling we have to beef up our testing game in order to avoid all these hassles in the future.
My suggestion is to setup CI to test the oldest patch level on each minor python version. If it's not supported by GitHub workflows, we might use pyenv or something similar and implement that ourselves. Maybe there are python actions that do that already though.
Just some thoughts of mine, I am looking forward to learning about yours.

[kasium]

Hi all,

just a short idea: typing imports could be wrapped into if TYPE_CHECKING and then all types must be wrapped into quotes. With this you could avoid these kind of issues like with python 3.7.1.

In addition there is also a checker for flake8: https://github.com/asottile/flake8-typing-imports

foo.py

from typing import OrderedDict

Execute: flake8 --select TYP --min-python-version 3.7.0 foo.py

foo.py:1:1: TYP001 guard import by `if TYPE_CHECKING:`: OrderedDict (not in 3.7.0, 3.7.1)

EDIT: I'm happy to file a PR to add flake8-typing-imports if you think this makes sense

[gaborbernat]

When running in strict mode one gets failures for the repository not exporting explicitly its properties:

Module "git" does not explicitly export attribute "Git"; implicit reexport disabled

See https://github.com/python/mypy/issues/10198#issuecomment-841175878 for more details.

[Yobmod]

Hi all,

just a short idea: typing imports could be wrapped into if TYPE_CHECKING and then all types must be wrapped into quotes. With this you could avoid these kind of issues like with python 3.7.1.

In addition there is also a checker for flake8: https://github.com/asottile/flake8-typing-imports

foo.py

from typing import OrderedDict

Execute: flake8 --select TYP --min-python-version 3.7.0 foo.py

foo.py:1:1: TYP001 guard import by `if TYPE_CHECKING:`: OrderedDict (not in 3.7.0, 3.7.1)

EDIT: I'm happy to file a PR to add flake8-typing-imports if you think this makes sense

We already have a similar library in requirements-dev.txt (flake8-type-checking), but it has too many false positives to use in CI.
So feel free to swap it for flake8-typing-imports and if it works (and once all errors are fixed/ignored) it can be added to test-requirements.txt.

The issue for Orderdict though is that we are subclassing it, so one import has to be runtime available. Maybe we can put the typed versions in the if TYPE_CHECKING block after the runtime import? It will still need py version guards for mypy though? it just wont break at runtime.

[Yobmod]

When running in strict mode one gets failures for the repository not exporting explicitly its properties:

Module "git" does not explicitly export attribute "Git"; implicit reexport disabled

See python/mypy#10198 (comment) for more details.

Yep, we have a lot of * imports that rely on implicit-reexport for mypy. For end-users this should not cause problems, but its a known mypy bug: https://github.com/python/mypy/issues/8754.

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

[gaborbernat]

I personally think for reasons explained in https://github.com/python/mypy/issues/10198#issuecomment-841175878 this is not a mypy bug 🤔 and I don't think due to that will be fixed 🤔 At best I'd imagine in py.typed you'd have some information about the package using implicit re-export but you cannot assume all packages will use that. Implicit re-export is a convenient but dangerous feature, and should be IMHO an opt-in not an always on thing.

[kasium]

The issue for Orderdict though is that we are subclassing it, so one import has to be runtime available. Maybe we can put the typed versions in the if TYPE_CHECKING block after the runtime import? It will still need py version guards for mypy though? it just wont break at runtime.

Yes, I guess it will only prevent runtime issues but during type checking time, the imports must work. However, normal users might get more stable releases

So feel free to swap it for flake8-typing-imports and if it works (and once all errors are fixed/ignored) it can be added to test-requirements.txt.

Let me check and open a PR if possible. Please note, that these libraries are doing different things. The current one is more style related, while flake8-typing-imports checks import from typing based on the current python version

[kasium]

So, I opened https://github.com/gitpython-developers/GitPython/pull/1309.
Please note that gitpython says it's compatible with >= 3.6.0 but it's only compatible with > 3.6.1! See the PR for details.

[Yobmod]

Seems useful already!

So we have to decide:

• Drop 3.6 now,

• or i'll pull out NamedTuple defaults (there are only a couple of uses) for now.

(NoReturn and Deque are available in typing_extensions, so that can be easily fixed.)

I was planning to wait until 3.10 released to drop 3.6.
But I see it had it's final release this June and i have noticed big packages already dropping it (numpy and pandas did last year and therefore most of scientific packages).

[Byron]

Dropping py 3.6 seems like the path of least resistance.
I suggest to go with that.

[gaborbernat]

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

See https://github.com/python/mypy/issues/8754#issuecomment-891109179, seems this is not a bug in mypy, but a feature and the issue lies in this package, not the type checkers (both mypy and pyright seem to assume that implicit re-export is off).

[Yobmod]

If someone wants to change all the * imports to match __all__ for each module it will fix it, but otherwise have to wait for the bug to be fixed. Current workaround is if using --strict, also use --implicit-reexport.

See python/mypy#8754 (comment), seems this is not a bug in mypy, but a feature and the issue lies in this package, not the type checkers (both mypy and pyright seem to assume that implicit re-export is off).

Its fixable here, but there are 156 implicit-reexports errors if its activated in the codebase atm, so it wont be done for the next few releases unless someone volunteers.
I've added implicit_reexport = true to our settings, but I don't know if that will be overriden downstream?

[kasium]

Is there any update how you want to proceed with this?

[Yobmod]

@Byron

I think i've done everything I planned for the next release.
3.6 is dropped, and all reported errors in typing fixed and tested against an expanded test matrix. (Github actions only goes as early as 3.7.5 though).

Testpypi uploads fine (using 3.7.3), so ready to go!

[Byron]

Thanks a lot! Thanks to a 'hickup' this was ultimately released as https://pypi.org/project/GitPython/3.1.23/ .