Pytest: Support for static typing

[markedwards]

Is there any plan to add static type hints to the Pytest code base, or to support/bless a typeshed implementation? Or is this already solved and my Google skills have failed me?

[RonnyPfannschmidt]

its currently unsolved and hasnt been discussed in depth by the wider set of pytest core contributors,

its not clear how and in what time frames to progress

[nicoddemus]

@markedwards I suppose you mean adding support so things like @pytest.fixture can accept type hints right?

I don't know, it feels to get the most bang for the buck we would need to be Python 3 only to be able to use native type hints instead of hints in comments (I'm aware it is possible to use type hints in Python 2). Perhaps when we drop Python 2 we can start thinking about this?

[The-Compiler]

I think @markedwards means adding type annotations to pytest's API (in the core or in the typeshed repository), so users of pytest can e.g. use mypy to tell them if they're using pytest's API incorrectly.

[markedwards]

Thanks everyone. Yes @The-Compiler that’s what I meant. I’m sure there are wrinkles that I haven’t considered, so I’ll just leave this as a general request for type hint support, and the developers can determine what exactly that should mean.

[nicoddemus]

I think @markedwards means adding type annotations to pytest's API (in the core or in the typeshed repository), so users of pytest can e.g. use mypy to tell them if they're using pytest's API incorrectly.

I expressed myself badly, sorry, that is what I meant. (reading my reply again I can see how it actually means something completely different).

I think one can contribute the pytest API to the typeshed repository right away without having to change pytest, I've done so for the deprecated python-skeletons repository (the work done there can be used as a starting point btw).

[gaborbernat]

@nicoddemus here's my use case, imagine I want to type hint my test code:

def test_magic(tmpdir, caplog, monekypatch):
    pass

To be able to do type hint I need the fixture types to be exposed publicly (otherwise I need to import private stuff from Pytest which leaves me exposed; e.g. CaptureFixture in the above example).

# private type imports; oh no!!!
from _pytest.capture import CaptureFixture
from _pytest.monkeypatch import MonkeyPatch
from py._path.local import LocalPath

def test_magic(tmpdir: LocalPath, caplog: CaptureFixture, monekypatch: MonkeyPatch):
    pass

Maybe we should have beside the https://github.com/pytest-dev/pytest/blob/master/src/pytest.py a pytest.fixture_types a module that exposes those? Users are already relying on their interfaces in the tests, so those are already somewhat public; not?

I also propose to split these two issues:

1. type hint pytest code
2. provide stubs to type hint test code written against pytest.

[gaborbernat]

@RonnyPfannschmidt @nicoddemus @asottile ping :-)

[asottile]

my 2c:

• exposing a convenient public module for the private types seems like a good idea 👍
• providing stubs in typeshed also seems like a good idea 👍 (iirc all that's needed here is pytest's project sign off)
• personally I really dislike the type-comments approach to typing so I'd be -0 on pytest itself starting to adopt mypy annotations until we can stop supporting python 2
• all this said, I usually don't bother annotating my tests 🤷‍♂️

[pior]

At work, we have more and more fully typed code-bases, and we type our tests as well.

👍 for exposing the private types and typing the public api !

Also: typeshed doesn't accept third-party stubs anymore, they are moving to a decentralized solution with the PEP-561.
While also discussing a third option: https://github.com/python/typeshed/issues/2491

To add some context, here is an interesting stub project: https://github.com/dropbox/sqlalchemy-stubs

[Zac-HD]

Checklist:

• [ ] Create a stub file src/pytest.pyi for the public interface
• [ ] Create a src/py.typed file (per PEP 561) to indicate that the interface is type-checkable
• [ ] Check that this does not break everything by dumping the marker file into the root site-packages directory when installed.
• [ ] Add package_data={"": ['pytest.pyi', '__init__.pyi']}, to the metadata in setup.py
• [ ] Create a new public module pytest.types which exports all and only the internal types needed for type hinting.
• [ ] Clearly document that pytest.types is not guaranteed stable between minor versions of Pytest, so we don't get locked into a compatible but terrible early version.

[RonnyPfannschmidt]

that pytest.type thing can possibly get us into deep trouble - i i wonder how to sort that out

[CaselIT]

Just my 2 cents.
I think a pytest.fixture_types module to expose the private types as proposed before would be enough.

[nicoddemus]

that pytest.types thing can possibly get us into deep trouble - i i wonder how to sort that out

We should probably create proper abc types/interfaces and only expose that instead of the actual concrete implementation. This will also be a very healthy thing for the project as whole.

[RonnyPfannschmidt]

main problem with that is, we have no idea what those are or ought to be without experimenting and breaking

[nicoddemus]

What we have in the reference would be a good start, because it is what is defacto documented at least.

[RonnyPfannschmidt]

@nicoddemus true, but the docs cant break other peoples code, an exposed module can ^^ - so we gotta ensure the details

[asottile]

so it turns out that I fixed my remaining "gripes" with type comments -- notably they didn't play well with pyflakes (and therefore flake8) -- now that pyflakes recognizes PEP 484 type comments it's not so bad to add types in my opinion.

I decided to try it out on pre-commit/pre-commit-hooks, here's the results: https://github.com/pre-commit/pre-commit-hooks/pull/360

[bluetech]

Pytest is now Python>=3.5 only, so it is possible to use the type annotation syntax. I created an RFC PR #5575 which proposes adding mypy checking to the code base, which is nice in itself, but with the goal of eventually covering the entire public API with annotations, and enabling external users to depend on it (py.typed file, see PEP 561) without a separate stub file.

What do you think?

[bluetech]

The initial PR was merged (thanks @asottile for helping), now pytest runs mypy internally.

To kickstart the plan from the previous comment, I created another PR #5593 which starts adding some type annotations.

[christopher-hesse]

@bluetech Thanks for working on this! I'm using the pyright type checker and it's not finding the types in pytest. Should the py.typed file be added at this point?

[bluetech]

We are not quite ready to add a py.typed file, but we are making progress.

It would be helpful however if you (and anyone else interested) try adding the py.typed file manually and telling us how it goes (currently adding it is a little complicated due to the pytest package layout, we will need to sort it out sometime):

mkdir $VIRTUAL_ENV/lib/python3.7/site-packages/pytest
mv $VIRTUAL_ENV/lib/python3.7/site-packages/{pytest.py,pytest/__init__.py}
touch $VIRTUAL_ENV/lib/python3.7/site-packages/{pytest,_pytest}/py.typed

I'll be interested to know if you get errors you don't expect, and also if you are using some API which is not typed yet, which will help focus the effort.

Also note that I have only ever checked with mypy -- not sure how pyright will work out. Let us know that too!

[christopher-hesse]

The current module setup of pytest is indeed confusing. After trying this out it looks like pyright actually requires separate stub files: https://github.com/microsoft/pyright/blob/3d9c094918ebebac56b733d97b00392ea0d58805/docs/type-stubs.md

Pytorch does it this way, for instance: https://github.com/pytorch/pytorch/tree/master/torch/nn

I'm not certain pyright even cares about py.typed files.

[bluetech]

Our current strategy, as laid out in https://github.com/pytest-dev/pytest/issues/3342#issuecomment-509253652, is to use py.typed; perhaps pyright can add support for it? This way it will also work with many other packages which use py.typed.

[christopher-hesse]

As far as I can tell, py.typed and stub files aren't really related to each other. pyright looks like it supports librarys that have py.typed files as well as those without. It does however, only support stub files, not inline annotations, for libraries specifically. The stub files should contain the public API of the package.

Moving pytest.py to a package and adding py.typed both seem like good ideas regardless.

[nicoddemus]

Hi @bluetech,

We are not quite ready to add a py.typed file, but we are making progress.

Pardon my ignorance (I don't work much with mypy), what is missing us from adding py.typed currently in pytest? I believe all public APIs already contain type hints, no? What else is missing?

[asottile]

@nicoddemus I think we're mostly unblocked -- though I think we'll have to move pytest.py to be pytest/__init__.py since PEP561 has no provisions for py_modules (unless there's some .pyi trick I don't know about here)

[RonnyPfannschmidt]

+1 - i want a pytest package back, its also a place where we can put publicly available code/api in a sane manner

[nicoddemus]

Had a few minutes to spare so decided to tackle this: https://github.com/pytest-dev/pytest/pull/6304

[bluetech]

what is missing us from adding py.typed currently in pytest? I believe all public APIs already contain type hints, no? What else is missing?

I don't think all public API is covered, though that shouldn't necessarily hold us back.

My plan was:

• Expand the typings where possible with emphasis on public API.
• Enable check_untyped_defs so the tests self-validate the types.
• Work out types for py.path.local somehow, since it shows up in pytest's API a lot.
• Post a pinned issue to ask interested users to manually add py.typed files and tell us how it goes.
• Add the py.typed files in a feature release.
• At this point I imagine we'll start getting feedback which will require some fixes.

We need to be somewhat careful because

• Typing is not that stable yet - typing and mypy and typeshed do change & break stuff.
• The type annotations become public API which need to be reasonably stable once we add py.typed.

[bluetech]

Update on my typing branch:

Expand the typings where possible with emphasis on public API.

Most of the interesting/common public APIs are covered now, and most of the internal APIs as well.

Enable check_untyped_defs so the tests self-validate the types.

This is done now, for both src/ and testing/.

Work out types for py.path.local somehow, since it shows up in pytest's API a lot.

This is done in https://github.com/pytest-dev/py/pull/232, not integrated yet however.

---

My plan now is to clean up the commits and submit the branch for inclusion. It is too big for any regular review though, so we will have to see how to go about that.

---

In the meantime, if you use pytest and mypy and want to try it, you can do it with these two commands in your virtualenv:

$ pip install --upgrade git+https://github.com/bluetech/pytest@typing#egg=pytest
$ touch "$VIRTUAL_ENV"/lib/python*/site-packages/{pytest,_pytest}/py.typed

If there any type errors or failures you do not expect, let me know. Also if not :)

[bluetech]

Another status update (hopefully the last):

Most of the internals of pytest are typed. According to mypy coverage report, we are "16.46% imprecise". Note that pytest uses some dynamic mechanisms that cannot be really statically typed.

The public API of pytest is not clearly delimited, especially for plugins. But what we can measure easily is the exported pytest.* interface, and hooks. This is the progress on the public interface:

• [x] pytest.__version__
• [x] pytest.approx (partial)
• [x] pytest.Class (partial)
• [x] pytest.cmdline
• [x] pytest.collect (deprecated)
• [x] pytest.Collector (partial)
• [x] pytest.console_main
• [x] pytest.deprecated_call
• [x] pytest.exit
• [x] pytest.ExitCode
• [x] pytest.fail
• [x] pytest.File (partial)
• [x] pytest.fixture
• [x] pytest.FixtureLookupError
• [x] pytest.freeze_includes
• [x] pytest.Function (partial)
• [ ] pytest.hookimpl (pluggy)
• [ ] pytest.hookspec (pluggy)
• [x] pytest.importorskip
• [x] pytest.Instance (partial)
• [x] pytest.Item (partial)
• [x] pytest.main
• [x] pytest.mark
• [x] pytest.Module (partial)
• [x] pytest.Package (partial)
• [x] pytest.param
• [x] pytest.PytestAssertRewriteWarning
• [x] pytest.PytestCacheWarning
• [x] pytest.PytestCollectionWarning
• [x] pytest.PytestConfigWarning
• [x] pytest.PytestDeprecationWarning
• [x] pytest.PytestExperimentalApiWarning
• [x] pytest.PytestUnhandledCoroutineWarning
• [x] pytest.PytestUnknownMarkWarning
• [x] pytest.PytestWarning
• [x] pytest.raises
• [x] pytest.register_assert_rewrite
• [x] pytest.Session (partial)
• [x] pytest.set_trace
• [x] pytest.skip
• [x] pytest.UsageError
• [x] pytest.warns
• [x] pytest.xfail
• [ ] pytest.yield_fixture (deprecated)

This is the progress on hooks:

• [x] pytest_addhooks
• [x] pytest_plugin_registered
• [x] pytest_addoption
• [x] pytest_configure
• [x] pytest_cmdline_parse
• [x] pytest_cmdline_preparse
• [x] pytest_cmdline_main
• [x] pytest_load_initial_conftests
• [x] pytest_collection
• [x] pytest_collection_modifyitems
• [x] pytest_collection_finish
• [x] pytest_ignore_collect
• [x] pytest_collect_directory
• [x] pytest_collect_file
• [x] pytest_collectstart
• [x] pytest_itemcollected
• [x] pytest_collectreport
• [x] pytest_deselected
• [x] pytest_make_collect_report
• [x] pytest_pycollect_makemodule
• [x] pytest_pycollect_makeitem
• [x] pytest_pyfunc_call
• [x] pytest_generate_tests
• [x] pytest_make_parametrize_id
• [x] pytest_runtestloop
• [x] pytest_runtest_protocol
• [x] pytest_runtest_logstart
• [x] pytest_runtest_logfinish
• [x] pytest_runtest_setup
• [x] pytest_runtest_call
• [x] pytest_runtest_teardown
• [x] pytest_runtest_makereport
• [x] pytest_runtest_logreport
• [x] pytest_report_to_serializable
• [x] pytest_report_from_serializable
• [x] pytest_fixture_setup
• [x] pytest_fixture_post_finalizer
• [x] pytest_sessionstart
• [x] pytest_sessionfinish
• [x] pytest_unconfigure
• [x] pytest_assertrepr_compare
• [x] pytest_assertion_pass
• [x] pytest_report_header
• [x] pytest_report_collectionfinish
• [x] pytest_report_teststatus
• [x] pytest_terminal_summary
• [x] pytest_warning_captured
• [x] pytest_warning_recorded
• [x] pytest_doctest_prepare_content
• [x] pytest_internalerror
• [x] pytest_keyboard_interrupt
• [x] pytest_exception_interact
• [x] pytest_enter_pdb
• [x] pytest_leave_pdb

pytest has two public dependencies (exposed to users):

• [x] py (particularly py.path.local) - The next release will hopefully contain type stubs. (Released as py 1.9.0).
• [ ] pluggy - Not typed. I plan to add type annotations, however given the nature of the library they will not be very helpful, at least without a mypy plugin or such.

My plan is to finish whatever missing pieces we can and publish the types in the upcoming pytest 6.0.0 release.

[jamescooke]

@bluetech Thanks so much for all your hard work on adding typing :raised_hands:

I've got Pytest master working with type-checking in tox environments for Python versions 3.6, 3.7 and 3.8 using pip install and then touching py.typed in the tox {envdir}/lib/{python_version}/site-packages/pytest/ and _pytest/ dirs.

If there any type errors or failures you do not expect, let me know. Also if not :)

The project isn't using anything complex unfortunately - just @fixture, mark.parametrize(), mark.skip(), raises(), but they all pass mypy great :+1:

[adamtheturtle]

Nice work!

I have no type errors in my project but I'm a little confused by something.
Please let me know if there is a better place for this (including nowhere).

I was (perhaps wrongly) expecting after seeing this closed to be able to add a type for the request parameter in the following sample without import _pytest:

import pytest


@pytest.fixture(params=["a"])
def example_fixture(request) -> str:  # << type hint for request on this line
    return request.param

Is that possible?
If not - should I make an issue?

[bluetech]

Hi @adamtheturtle,

This is something that might be nice to have eventually, but is not supported as of now. In any case there is no way to achieve this in Python itself, so would require a mypy plugin at least. Some previous discussion on this here: #5981.

For now we are working on a prerequisite for this, which is making explicit annotations for (builtin) fixtures possible: #7469.

[The-Compiler]

@bluetech I assume explicit annotations is what @adamtheturtle meant with "I was expecting [...] to be able to add a type for the request parameter" above. To clarify, no mypy plugin would be needed for that (only #7469), but a mypy plugin could be used to make mypy aware of the type of fixtures without needing an explicit annotations.

[adamtheturtle]

Thanks @bluetech , @The-Compiler for your quick and thorough responses - I believe that #7469 is the issue which will resolve what I'm interested in.

[squaresmile]

With pytest 6.0.1, mypy doesn't complain about pytest anymore but I'm still getting untyped decorator for @pytest.mark.parametrize from mypy. I'm checking with disallow_untyped_decorators = True. Is this a problem with my setup or not in the scope of the typing PR?

tests\test_raw.py:70:2: error: Untyped decorator makes function "test_raw" untyped

Here is the test file and here is the mypy.ini file. Thanks a lot for typing work.

[bluetech]

Hi @squaresmile,

I can't reproduce this:

import pytest

@pytest.mark.parametrize("query,result", [("q", "r")])
def test_raw(query: str, result: str) -> None:
    pass

reveal_type(test_raw)

gives

x.py:7: note: Revealed type is 'def (query: builtins.str, result: builtins.str)'

If you can create a standalone reproduction, please open a new issue with it and I'll take a look.

[lschmelzeisen]

@bluetech Saving the code you provided to a file test.py and running the following reproduces this:

$ mypy --strict --pretty test.py
test.py:3: error: Untyped decorator makes function "test_raw" untyped
    @pytest.mark.parametrize("query,result", [("q", "r")])
     ^
test.py:7: note: Revealed type is 'Any'
Found 1 error in 1 file (checked 1 source file)

[lschmelzeisen]

Additionally, methods of capture classes do not seem to be type-annotated:

import _pytest

def test_raw(capsys: _pytest.capture.CaptureFixture) -> None:
    assert capsys.readouterr().out == ""

$ mypy --pretty test.py 
test.py:3: error: Name '_pytest.capture.CaptureFixture' is not defined
    def test_raw(capsys: _pytest.capture.CaptureFixture) -> None:
    ^
Found 1 error in 1 file (checked 1 source file)

How is this case supposed to be resolved?

[asottile]

@lschmelzeisen I believe you're meant to import _pytest.capture to make that symbol available

[bluetech]

Saving the code you provided to a file test.py and running the following reproduces this:

There were two issues here: #7589 and #7593. The first be fixed in the next mypy release, and the second in pytest 6.1.

[bluetech]

@lschmelzeisen I believe you're meant to import _pytest.capture to make that symbol available

Right. I would also add that if you import from _pytest we do not guarantee stability. It's OK as long as you are willing to adapt if/when it breaks. We are working on a stable solution for this for pytest 6.1: #7469.

[lschmelzeisen]

There were two issues here: #7589 and #7593. The first be fixed in the next mypy release, and the second in pytest 6.1.

Great, looking forward to 6.1 then. And by the way, big thanks to you and others that are working on integrating typing into pytest!

@lschmelzeisen I believe you're meant to import _pytest.capture to make that symbol available

Right. I would also add that if you import from _pytest we do not guarantee stability. It's OK as long as you are willing to adapt if/when it breaks. We are working on a stable solution for this for pytest 6.1: #7469.

Sorry, I meant to report the following, but was to quick for myself when copying code:

import _pytest.capture

def test_raw(capsys: _pytest.capture.CaptureFixture) -> None:
    assert capsys.readouterr().out == ""

$mypy --strict --pretty test.py 
test.py:4: error: Call to untyped function "readouterr" in typed context
        assert capsys.readouterr().out == ""
               ^
Found 1 error in 1 file (checked 1 source file)

And yes, I am willing to use the internal _pytest API and adapt on changes until a stable solution is available.

[bluetech]

Due to internal implementation difficulties, CaptureFixture is not properly typed yet. Among other things, capsys yields strings, while capsysbinary yields bytes, so at a minimum CaptureFixture will need to have a bytes/str type parameter. It needs some work to sort out.

[ssbarnea]

I am not sure why this issue is closed as, if I read correctly the last two messages I have reasons to believe it is not possible to yet to add typing information for capsys. Seems like a real bug, even if if fixing it depends on upstream changes being made.

[bluetech]

CaptureFixture is properly typed now actually. It is not officially exposed yet though -- the main effort has moved to #7469 now, although I procrastinate on it a bit.

[omry]

@lschmelzeisen I believe you're meant to import _pytest.capture to make that symbol available

Right. I would also add that if you import from _pytest we do not guarantee stability. It's OK as long as you are willing to adapt if/when it breaks. We are working on a stable solution for this for pytest 6.1: #7469.

Trying with pytest 6.1 and the latest mypy I still get the problem. Is this expected to work now?

$ pip freeze | grep "mypy\|pytest"
mypy==0.790
mypy-extensions==0.4.3
pytest==6.1.2
pytest-mock==3.3.1

$ cat test_strict.py

from pytest import mark

@mark.parametrize("a", [1, 2,3])
def test_foo(a: int) -> None:
    ...

mypy --strict output:

$ mypy --strict test_strict.py 
test_strict.py:3: error: Untyped decorator makes function "test_foo" untyped
Found 1 error in 1 file (checked 1 source file)

[bluetech]

@omry I can't reproduce this. Let me know if you can reproduce it in a clean virtualenv (preferably in a new issue).

[omry]

@bluetech, thanks for looking!
knowing it worked for you, I figured it out:
I had an old suppressing stub to avoid having to type ignore everything from pytest (which somehow did not work for pytest.mark.parametrize()):

$ cat .stubs/pytest.pyi 
def __getattr__(name): ...  # type: ignore

$ cat .mypy.ini 
[mypy]
python_version = 3.6
mypy_path=.stubs

After deleting the stub mypy outputs 183 new warnings about the ineffective # type: ignore statements I have there :).