Ptvs: pytest support

[zooba]

I'm not entirely sure how this should look, but we should investigate how to support/encourage pytest usage.

[joxn]

As discussed today, I'll take some of this on. I'll start with modifying the test runner to use pytest rather than unittest; once we have that we can talk about the right way to integrate into the UI.

[joxn]

Right, so, it looks like the protocol is that we get a list of test names on the command line -t 'TestName1' -t 'TestName2' ..., construct a test run out of those test names, and pass that into the UnitTest runner. Then, when each test is done running, the runner constructs a VSTestResult which reports back to VS.

The corresponding protocol in pytest is to implement a plugin that provides pytest_collection_modifyitems to set up the test run and pytest_runtest_logreport to report the result of a test. Seems like it should be pretty simple. The only issue is that it's not quite obvious how to handle being asked to run a test that pytest's collection protocol couldn't find. Possibly, just add a dummy test item with that name that calls pytest.skip.

Looking ahead to test collection, pytest provides a --collect-only mode; I'm going to guess that it will be pretty simple to implement an extension that runs pytest under IronPython just for collection purposes. Don't hate me.

<Module 'test_foo.py'>
  <Function 'test_foo[0]'>
  <Function 'test_foo[1]'>
  ...

Additionally, if we can provide a class that is duck typed like pdb.Pdb, we can have pytest use that class's set_trace to debug on test failures. This way, if pytest really screws with the debugging stack, we might be able to get triage debugging working quickly (though obviously it won't allow setting breakpoints in code under test or anything useful like that).

[joxn]

More notes on collection: the algorithm for figuring out what the base directory will be and what configuration files are controlling is pretty clearly documented here: http://doc.pytest.org/en/latest/customize.html

For PTVS it might be nice to provide a "Pytest here" context menu item in the solution pane, as well as implementing root directory discovery based on the above algorithm. But because discovery is so dependent on options specified in [pytest] config blocks and in conftest.py files, as well as test generation via @pytest.mark.parametrize and even custom test generation code, I don't think we'll be able to get around actually executing pytest to assemble the test list.

[zooba]

So on the protocol, I forgot that #1394 hasn't been fixed yet, but we really need to stop listing every single test on the command line, otherwise a big test suite will make it _really_ easy to hit the 32k limit. But using -t is what we currently have.

If running pytest under IronPython will work for collection that's fine, but I suspect it'll involve importing the test suite which will fail if libraries are missing. We'll probably need to actually run --collect-only to get the info, and merge it with our own analysis to figure out whatever is missing.

I'd like to be able to have a pdb.Pdb-like debugger, and we may be able to get it done with #750. More likely we'll just want to inject some functions/files into DONT_DEBUG or DEBUG_ENTRYPOINTS (like we already do in vs_testlauncher.py).

If you track down our targets files, you'll find some example code near the end of the web ones for adding commands under the project (right-click) Python menu. That's a quick way to get a simple top-level command in there, though it hardly counts as integration :)

[denfromufa]

looks like in pythonnet project the python tests are not discoverable with PTVS anymore due to pytest @vmuriart

[alexbk66]

What's the status of this issue?
It it actualy about using pytest in VS2017? Not clear to me.

[digitalbanana]

+1 @alexbk66
would really love it if this issue got some traction (or even a definitive answer)

[alexbk66]

Ok, it's 2019, any news?

[GaryVS]

Curious on the progress of this versus just unittest support in VS2017.

[B4rtware]

I would also appreciate a status update. The same problem exists inside VS2019: no pytest files are recognized.

[huguesv]

We still want to do this (we recognize it's a popular request) but have not scheduled it yet.

[B4rtware]

We still want to do this (we recognize it's a popular request) but have not scheduled it yet.

Good to hear that! Thanks for the quick response. I was not sure if it still considered being implemented or even on the radar.