Obspy: macos CI fails _NoValueType

Created on 14 Nov 2021 · 10Comments · Source: obspy/obspy

I am opening this issue to document the debugging work for all versions python on the macos CI image which currently produce a slew of failures similar to this:

    Traceback (most recent call last):
      File "/Users/runner/miniconda3/envs/test/lib/python3.8/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest obspy.signal.cross_correlation.templates_max_similarity[6]>", line 2, in <module>
        tr.data += np.random.random(len(tr)) * tr.data.max() * 0.5
      File "/Users/runner/miniconda3/envs/test/lib/python3.8/site-packages/numpy/core/_methods.py", line 40, in _amax
        return umr_maximum(a, axis, None, out, keepdims, initial, where)
    TypeError: float() argument must be a string or a number, not '_NoValueType'

When I log into the CI system using the tmate action I cannot reproduce the issue, either using conda or the conda-free setup on the branch in #2722 (the tests run fine).

After Googling around a bit I am inclined to think this may be a numpy compatibility issue but why I can reproduce when directly ssh'ed into the CI container remains a mystery.

bug CI

Source

d-chambers

All 10 comments

I thought it might be connected to setuptools and problems with the example data files, because only the setuptools version changed (from 58.0.4 to 58.2.0) when this error started popping up. But looking at newer test reports, the problem now occurs for a zoo of different setuptools versions.

trichter on 15 Nov 2021

I found some time to do a little more research.

This post had a similar issue and several posters solved it by either upgrading (or downgrading) their version of numpy.

Apparently, _NoValueType is a singleton used by numpy as a special sort of None. See here.

Reloading numpy might be at least part of the issue, as I occasionally see reloading numpy warnings in the CI output. See this numpy PR

d-chambers on 1 Jan 2022

I have a working hypothesis.

I think the issue is caused by our testing reloading numpy somewhere in the following test command:

https://github.com/obspy/obspy/blob/c4faef6b2df110e844c28ca2aa9515888c019cb8/.github/workflows/pull_request.yml#L122

When numpy is reloaded it breaks the singleton-ness of _NoValue which is no longer recognized by numpy's C-level code, hence it is handled incorrectly by a variety of ufuncs since they have _NoValue as the default argument for several parameters.

Interestingly, when I ran the test suite with either obspy-runtests or just python obspy/scripts/runtests.py the issue disappears. So something about using coverage run causes the numpy reload. See this numpy PR which added the warning about reloading numpy originally.

Unfortunately, I don't see an easy fix. We would have to find some other way to collect coverage information.

Personally, I think we should simply abandon ObsPy's test runner and switch to pytest, full stop. This would break the test reporting server but I think it is a small price to pay to get the CI working again (and remove a huge maintenance burden from the code). I spent a significant amount of time already trying to make the pytest output compatible with the reporting server in #2489 but I think it is a dead end.

d-chambers on 1 Jan 2022

I think it's time to move to pytest & rather spend some time on making it report _somehow_ rather than to spend too much extra time on this now.