Iris: Adopt py.test

[bjlittle]

⚙ Feature Request

There are currently a few approaches to running the iris tests as a developer:

1. using the bespoke iris.tests.runner e.g., python -m iris.tests.runner --default-tests
2. using the test command from setuptools e.g., python setup.py test
3. using unittest e.g., python -m unittest discover iris

These are all perfectly functional approaches, for now, to take to run the iris tests. However, there are a few problems with each approach, namely:

1. iris.tests.runner mechanism is bespoke and heavily relies on nose, which has been in maintenance mode for several years now i.e., it's dead-end technology :bomb:. Also using this mechanism is causing issues for automatically collating testing coverage statistics.
2. the use of the test command in setuptools was deprecated in version 41.5.0, (latest 50.3.0) and will be withdrawn in a future version i.e., the future is closer than you think
3. ummm unittest isn't pytest, not even close :wink:

Motivation

It's high time that iris adoped a modern, robust, sustainable, actively supported and scalable approach to testing through pytest and pytest-xdist... before it's too late, and this becomes a much bigger issue than it needs to be i.e., let's agree to plan to avoid this crisis... before it's a crisis.

[bjlittle]

@SciTools/iris-devs Thoughts?

[trexfeathers]

ummm unittest isn't pytest, not even close 😉

I would have said yes for this reason alone. Everything else makes me scared to _not_ switch!

[pp-mo]

a few approaches to running the iris tests as a developer:

All that is about the test runner, really, which is about the "total test suite definition", and the needs of CI.

For basic developer needs, I think you missed the really obvious:

• (4) python -m unittest discover ./lib/iris/tests/...
• (5) python /path/to/specific/test.py

For sure we should migrate to using pytest as the engine, but the _format_ of our tests has never depended on nose, it is all pure unittest, which pytest also supports quite happily.

So I don't think we need a wholesale rewrite. We can adopt pytest-specific methods as they seem useful,

I do also have a couple of lingering doubts about becoming pytest-specific, though :

• (1) at one time we likewise considered adopting nose styles, because it looked cool and much neater. In retrospect, that would clearly have been a mistake !
• (2) I'm still not sure how+whether we can preserve our existing rules for what to test, how to name them, and where to put them (or new equivalent) in the pytest style. For all the tiresome clunkiness of the existing scheme, it does allows us to group + organise things in a useful way, albeit with a bit of extra effort + verbosity.

[bjlittle]

@pp-mo Yeah, I wasn't intending to enumerate every possible way to run all the tests, it was just a means to justify my motivation and more importantly get some kinda rough gauge of what everyone else thinks.

• (4) python -m unittest discover ./lib/iris/tests/...

This is kinda (3) really, but let's not split hairs :wink:

• (5) python /path/to/specific/test.py

Yup, if you wanna run those isolated tests, and all shades of grey of running individual test classes within those, or test class tasks etc

[bjlittle]

* (2) I'm still not sure how+whether we can preserve our existing [rules for what to test, how to name them, and where to put them](https://scitools-iris.readthedocs.io/en/latest/developers_guide/tests.html) (or new equivalent) in the pytest style.  For all the tiresome clunkiness of the existing scheme, it does allows us to group + organise things in a useful way, albeit with a bit of extra effort + verbosity.

@pp-mo That a really good point. But at the end of the day I guess we can evolve those iris test name conventions etc as we roll forwards, and make some pragmatic decisions if we want to be more liberal (or not), and factor in the work what needs done and when, and in what order... jeez it's almost like we're gonna plan this :open_mouth:

This is a BIG change, and if we're serious about doing it (and I think we should be), how we go about it e.g., blitz it all in a focused sprint, do it piece-meal etc?

[bjlittle]

With pytest you can run unittest tests as-is, but that's a bit of a cop out (IMHO) and it kinda paints you into a corner such that you can't fully exploit all the goodness of pytest.

I think that the BIGGEST headache in migration from unittest to pytest for us will be discovering the suitable, common patterns to rewrite the mock base unit tests in pytest :exploding_head: ... but I did recently take a stab at this for tephi.

I think that it would be wise to get a rough ideal of common approaches we want to adopt for migrating particular testing patterns, otherwise we'll end up with a multitude of different approaches to doing the same thing... and that, I suspect, will be a maintenance headache and increase the bar on developer testing effort

[rcomer]

I think we have test examples that would look much cleaner using pytest parametrization, e.g. here two tests that are very similar and the class is subclassed for each of the four arithmetic operations. Or here where each test is looping through the operations. I assume we wouldn’t want to write anything like the latter now, as if your first operation failed the others won’t get tested.

Aside - was there an intention to move/replace all the old modules in the top level tests directory when the unit/integration subdirectories were introduced?

[bjlittle]

@rcomer I think that the home of certain tests requires some serious rationalisation and consideration, and for me this would all come under this work. So :+1: on that.

From what I recall (and this is going back some time, perhaps @pp-mo can help clarify) those are legacy tests... and it's a fine line between those tests and integration tests, and system test etc. In fact, the unit tests came later, and those have a little bit more rigour... so IMHO it all needs a bit of a rethink to restructure it in a consistent and coherent way - plus there is most likely a lot of duplicated dead wood to purge/revisit, so we should take the opportunity to do that too.

[bjlittle]

I'd also advocate that #3416 is also dealt with as part of this testing re-haul.

[pp-mo]

was there an intention to move/replace all the old modules in the top level tests directory when the unit/integration subdirectories were introduced?

those are legacy tests...

Since @bjlittle asked me directly, yes we intended to retire + replace the "legacy" tests gradually over time.
But re-writing old tests, which are still working fine, has often seemed too much of a thankless task, while busy getting something else done.