Pytest: conftest.py discovery issue on Windows

[soupault]

In scikit-image we are experiencing an issue with the documentation testing due to conftest.py discovery issues on Windows (we use Windows VMs on AppVeyor and Azure).

Basically, the structure of our testing env is the following:

skimage/
- subpackage0/
--- tests/...
--- (modules)
...
- subpackageN/
--- tests/...
--- (modules)
- setup.py
- conftest.py

Once the package is installed, we run the testing from another directory as follows:

export TEST_ARGS="-v --doctest-modules --cov=skimage"
pytest ${TEST_ARGS} --pyargs skimage

For an example of the (failing) build, see https://dev.azure.com/scikit-image/scikit-image/_build/results?buildId=344.

For Linux and MacOS builds on Travis-CI we use essentially the same scripts. However, it seems that the conftest.py file at the rootdir of the package is correctly located and taken into account, what makes us think that the issue is Windows-specific.

Please, let me know if you need any additional debugging info.

• [x] a detailed description of the bug or suggestion
• [x] *output of pip list from the virtual environment you are using: see the Azure link above -> section "Pre-testing"
• [x] pytest and operating system versions: pytest 4.4.1
• [ ] minimal example if possible: sorry, I don't have a Windows machine for this

[nicoddemus]

Hi @soupault,

I'm seeing failures like this:

Expected:
    (array([107432, 154712]), array([ 0. ,  0.5,  1. ]))
Got:
    (array([107432, 154712], dtype=int32), array([ 0. ,  0.5,  1. ]))

Is this what you refer to?

[soupault]

Hi @nicoddemus ,

Thank you for the prompt response!
Yes. Basically, in our conftest.py we fix numpy printoptions to "legacy" format (there are few more things, but they are not relevant). Those failing tests indicate that the printoptions have not been set (i.e. numpy is using the new formatting), what, in its turn, means that the conftest.py has not been found and/or properly used.

For more information on numpy printoptions, see https://docs.scipy.org/doc/numpy/reference/generated/numpy.set_printoptions.html (-> argument legacy) or numpy 1.14 release notes.

[nicoddemus]

Hi @soupault,

I've been able to reproduce the issue locally by following the Azure build script. Using the debugger I've found out:

1. The conftest.py is being discovered and imported as expected. I can see np.set_printoptions() being called.
2. no.get_printoptions() is returning the expected values just prior to calling the doctest.

I've added the following code to ensure this:

python def runtest(self): _check_all_skipped(self.dtest) import numpy assert numpy.get_printoptions()['legacy'] == '1.13' self._disable_output_capturing_for_darwin() failures = [] self.runner.run(self.dtest, out=failures) if failures: raise MultipleDoctestFailures(failures)

(this is the pytest code which executes the doctests)

As expected the assertion works. I've also changed that string to something else for sanity check, and all tests fail then as expected. This can be easily verified by adding this to any of the failing doctests:

    >>> assert np.get_printoptions()['legacy'] == '1.13'

This assertion passes for me on failing doctests (again for sanity check we can change the "1.13" string to something else, say "FOO", and then we see the assertion failing).

Given the above, I'm not sure what's happening here, numpy's print options seem to be set to the correct value at the time the doctests are being executed.

Any other hints on how I can debug this in numpy?

Also, did you start seeing this behavior after upgrading pytest? If so, from which version did you upgrade?

[soupault]

@nicoddemus Great analysis, thank you very much! At this point it rather looks like a numpy [past?] issue. I now have an idea that it might be related to the way numpy prints the arrays of _platform-dependent_ types 🤔. Let me investigate this a bit further.

This can be easily verified by adding this to any of the failing doctests

That is really smart! Silly me... 🤦‍♂

Also, did you start seeing this behavior after upgrading pytest? If so, from which version did you upgrade?

The AppVeyor configuration file is really old, so it is hard to say if the version of pytest makes any difference. I can check with several past versions and report to you back.

[nicoddemus]

The AppVeyor configuration file is really old, so it is hard to say if the version of pytest makes any difference. I can check with several past versions and report to you back.

Sounds good, thanks!

[soupault]

@nicoddemus It appears that the issue is, indeed, in the numpy print formatting - https://github.com/numpy/numpy/issues/13468. I don't see how it can be related to pytest anymore :).
An interesting case, nevertheless.

Thank you very much for your analysis and support, and I'm really sorry for taking your time!

[nicoddemus]

Thank you very much for your analysis and support, and I'm really sorry for taking your time!

Not at all, glad I could help! Cheers!