Plasmapy: Enable skipping of slow tests when using pytest from the command line

[namurphy]

Some of our tests take noticeably longer than others, and often it's not necessary to run the full test suite. The pytest documentation describes how to mark tests as slow and control whether or not they get run using command line flags. It would be great to implement this for our tests. Our longest tests are maybe half a minute so this is not urgent, but it would be helpful to do this since our test suite is probably only going to grow.

My preference would be to skip the slow tests by default when running on the command line because most of the time we want the tests to run quickly so that we can find test failures quickly. We do want to run the full test suite for pull requests, but I'd suggest that we have at least one run of our test suite only attempt the quick tests so that we can detect test failures quickly.

I'd also find it helpful if we could set it up so that the slow tests are run last, and have an option that the slow tests are only run if all of the quick tests pass. That's a bit more work, so maybe we can leave that as a stretch goal. Thanks!

[StanczakDominik]

In other words, this would require us to change:

1. Find which tests are running the slowest
2. Tag them with @pytest.mark.slow, which is one time we probably won't have to argue about naming
3. Update docs for how to use this (see below)

I would be wary of skipping any tests as a default, though, as that would likely be more confusing to newcomers! Using pytest -m 'not slow' is simple enough, whereas I don't see any way easy way to tell Pytest "okay, run all the tests" with not slow being the default.

The ordering part can then be solved by simply using two commands: pytest -m 'not slow', pytest -m slow. Not as flashy as a custom solution, but I briefly looked and don't think anything exists for ordering tests based on mark as of now.

For any active development you should probably be running pytest plasmapy/transport or whichever module you're working on at the time any way. That's going to limit the issue of having slow tests elsewhere.

[StanczakDominik]

Oh ho ho, look what I've found:

https://stackoverflow.com/a/27899853/4417567

you can pass --durations=N to print the slowest N tests after the test suite finishes.

[namurphy]

I would be wary of skipping any tests as a default, though, as that would likely be more confusing to newcomers!

True enough! In that case, let's go with running all tests as the default.

[StanczakDominik]

Closed by #677!