Astropy: import _erfa fails for numpy 1.14

[dhomeier]

On 3.2rc1 with numpy 1.14.5 installed import astropy.table fails in _erfa boiling down to this ufunc import error:

Python 3.7.3 (default, May  5 2019, 04:25:55) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy.table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.7/site-packages/astropy/table/__init__.py", line 41, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/sw/lib/python3.7/site-packages/astropy/table/column.py", line 26, in <module>
    from . import groups
  File "/sw/lib/python3.7/site-packages/astropy/table/groups.py", line 7, in <module>
    from .index import get_index_by_names
  File "/sw/lib/python3.7/site-packages/astropy/table/index.py", line 38, in <module>
    from astropy.time import Time
  File "/sw/lib/python3.7/site-packages/astropy/time/__init__.py", line 2, in <module>
    from .formats import *
  File "/sw/lib/python3.7/site-packages/astropy/time/formats.py", line 16, in <module>
    from astropy import _erfa as erfa
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

With numpy 1.16.3 this succeeds, but I did not find any note that NUMPY_LT_1_16 support has already been removed.

[pllim]

@dhomeier , did you try running the whole test suite? Is the error similar to #8672 ?

[dhomeier]

It's obviously somehow related to #8672, only here I believe numpy 1.14.5 is expecting in erfa_ufunc.ab.signature something like "(d3),(d3),(),()->(d3)", getting "(3),(3),(),()->(3)" instead.

I cannot run any tests, because the collection already fails on the _erfa import:

>>> astropy.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.7/site-packages/astropy/tests/runner.py", line 256, in test
    return runner.run_tests(**kwargs)
  File "/sw/lib/python3.7/site-packages/astropy/tests/runner.py", line 599, in run_tests
    from astropy.table import Table  # pylint: disable=W0611
  File "/sw/lib/python3.7/site-packages/astropy/table/__init__.py", line 41, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/sw/lib/python3.7/site-packages/astropy/table/column.py", line 26, in <module>
    from . import groups
  File "/sw/lib/python3.7/site-packages/astropy/table/groups.py", line 7, in <module>
    from .index import get_index_by_names
  File "/sw/lib/python3.7/site-packages/astropy/table/index.py", line 38, in <module>
    from astropy.time import Time
  File "/sw/lib/python3.7/site-packages/astropy/time/__init__.py", line 2, in <module>
    from .formats import *
  File "/sw/lib/python3.7/site-packages/astropy/time/formats.py", line 16, in <module>
    from astropy import _erfa as erfa
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

But they (almost) all succeed with numpy 1.16.3 installed.
BTW as it appears the test for NUMPY_LT_1_16 is done in erfa_generators.py when processing the C source, I am wondering what happens if the package is compiled and built on a system with numpy < 1.16 and later installed along numpy 1.16...

[dhomeier]

On Python 3.6 (but not 3.5!) the import fails with a different error:

Python 3.6.8 (default, May  5 2019, 04:21:11) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> from astropy._erfa import ufunc
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.6/site-packages/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/sw/lib/python3.6/site-packages/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ImportError: numpy.core.multiarray failed to import
>>> import numpy.core.multiarray
>>> import numpy.core._multiarray_umath
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

That makes sense insofar as the _multiarray_umath module is only available in numpy 1.16...

[dhomeier]

Similar failure built with icc 17.0.2 against numpy 1.15.4 on RH Linux:

Python 3.5.1 (default, Oct 21 2016, 21:37:19) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy.table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 ...
  File "/home/uni05/dhomeie/routines/python/astropy-3.2rc1/build/lib.linux-x86_64-3.5/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

Only when building against numpy 1.16 and later running with a lower numpy version installed, the ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' comes up instead.

[pllim]

building against numpy 1.16 and later running with a lower numpy version installed

I don't think this is a supported use case, so I think we can ignore the numpy.core._multiarray_umath error. 😅

[dhomeier]

Need to exclude this case for binary packages though...

[astrofrog]

@dhomeier how did you end up in the situation of using astropy compiler with 1.16 with an older numpy? Did you build a wheel yourself to use in a different environment?

[dhomeier]

Not a wheel, just a standard build from source which I then packaged as a Fink .deb. ( am also the maintainer of the Fink package). Thus, if numpy < 1.16 is not supported anymore, I could easily put that requirement into the dependencies, but the release info says nothing to that effect, and since Fink's numpy is only at 1.14.5 in stable, I began testing with that. The build starts (and completes) happily, as setup.cfg only has setup_requires = numpy>=1.13. But the failure to interface the older numpy ufuncs occurs when building as a regular user as well, both on Max OS X 10.12 and on RHEL.

[mhvk]

Hmm, it seems that my changes to _erfa are breaking things here - the problem seems to be that the preprocessing of erfa is done before a package is put together, instead of in the installation stage. I think it should be possible to ensure that the preprocessing remains numpy-version agnostic (i.e., only in compiling the erfa ufuncs is the numpy version checked). But is that better?

[dhomeier]

But at least for my builds from the PyPI sources both on OS X and RHEL apparently the numpy interface is not configured correctly for NUMPY_LT_1_16=True even in the build stage.
When building instead the git checkout of v3.2rc1, the dimension format is correct, but the _multiarray_umath error still persists (both with running setup.py test and installing first and importing from there):

Python 3.7.3 (default, May  5 2019, 04:25:55) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> import astropy.table
>>> from astropy._erfa import ufunc
>>> ufunc.ab.signature
'(d3),(d3),(),()->(d3)'
>>> astropy.test()
/sw/lib/python3.7/site-packages/matplotlib/__init__.py:886: MatplotlibDeprecationWarning: 
examples.directory is deprecated; in the future, examples will be found relative to the 'datapath' directory.
  "found relative to the 'datapath' directory.".format(key))
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.7.1

Running tests with Astropy version 3.2rc1.
Running tests in lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy.

Date: 2019-05-15T21:55:07

Platform: Darwin-16.7.0-x86_64-i386-64bit

Executable: /sw/bin/python3.7

Full Python Version: 
3.7.3 (default, May  5 2019, 04:25:55) 
[Clang 9.0.0 (clang-900.0.39.2)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.14.5
Scipy: 1.2.1
Matplotlib: 3.0.3
h5py: not available
Pandas: not available
Cython: 0.28.5
Scikit-image: not available
astropy_helpers: 3.1
Using Astropy options: remote_data: none.

Matplotlib: 3.0.3
Freetype: 2.6.1
rootdir: /Users/derek, inifile:
plugins: remotedata-0.3.1, openfiles-0.3.2, mpl-0.10, doctestplus-0.3.0, arraydiff-0.3, hypothesis-3.69.2
collected 11680 items / 33 errors / 15 skipped                                                                                                                                           

========================================================================================= ERRORS =========================================================================================
__________________________ ERROR collecting lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/convolution/tests/test_convolve.py __________________________
ImportError while importing test module '/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/convolution/tests/test_convolve.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/convolution/__init__.py:4: in <module>
    from .core import *  # noqa
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/convolution/core.py:23: in <module>
    from .utils import (discretize_model, add_kernel_arrays_1D,
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/convolution/utils.py:5: in <module>
    from astropy.modeling.core import FittableModel, custom_model
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/modeling/__init__.py:11: in <module>
    from . import models
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/modeling/models.py:10: in <module>
    from .projections import *
lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/modeling/projections.py:26: in <module>
    from . import _projections
E   ImportError: numpy.core.multiarray failed to import
------------------------------------------------------------------------------------ Captured stderr -------------------------------------------------------------------------------------
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

...

    from . import _projections
E   ImportError: numpy.core.multiarray failed to import
------------------------------------------------------------------------------------ Captured stderr -------------------------------------------------------------------------------------
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
==================================================================================== warnings summary ====================================================================================
/sw/lib/python3.7/site-packages/html5lib/_trie/_base.py:3
  /sw/lib/python3.7/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 33 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================== 15 skipped, 1 warnings, 33 error in 28.97 seconds ====================================================================
>>> import numpy.core._multiarray_umath
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

but only when installing without the --offline option; with the latter basically the same failure as in the PyPI builds:

Python 3.7.3 (default, May  5 2019, 04:25:55) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> import astropy.table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/__init__.py", line 41, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/column.py", line 26, in <module>
    from . import groups
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/groups.py", line 7, in <module>
    from .index import get_index_by_names
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/index.py", line 38, in <module>
    from astropy.time import Time
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/time/__init__.py", line 2, in <module>
    from .formats import *
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/time/formats.py", line 16, in <module>
    from astropy import _erfa as erfa
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

[mhvk]

Hmm, the _multiarray_umath warning is from convolution, so a different origin...

I'm now very confused!

[dhomeier]

[Update: please proceed to the next comment right away!]

Me too, as I could not find any direct reference to _multiarray_umath within the astropy source tree, but the import error seems to boil down precisely to the difference between numpy versions 1.14 and 1.16 in numpy/__ufunc_api.h

 static NPY_INLINE int
 _import_umath(void)
 {
-  PyObject *numpy = PyImport_ImportModule("numpy.core.umath");
+  PyObject *numpy = PyImport_ImportModule("numpy.core._multiarray_umath");
   PyObject *c_api = NULL;

   if (numpy == NULL) {
-      PyErr_SetString(PyExc_ImportError, "numpy.core.umath failed to import");
+      PyErr_SetString(PyExc_ImportError,
+                      "numpy.core._multiarray_umath failed to import");
       return -1;
   }

I have a suspicion that one of the pytest-* support modules might be to blame, possibly having been built with numpy 1.16 installed, but after wiping all dependencies and attempting a clean rebuild I am now running into the ufunc import error again.
First I had to manually remove everything inside ./build because after just a python3.7 setup.py clean installation failed with this

Extracting astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg to /Users/derek/lib/python3.7/site-packages
error: [Errno 63] File name too long: '/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.7/build/lib.macosx-10.12-x86_64-3.6/build/lib.macosx-10.12-x86_64-3.6/build/lib.macosx-10.12-x86_64-3.6/build/lib.macosx-10.12-x86_64-3.6/build/lib.macosx-10.12-x86_64-3.6/build/lib.macosx-10.12-x86_64-3.6/astropy/io/misc/asdf/data/schemas/astropy.org/astropy/coordinates/earthlocation-1.0.0.yaml'

but at the end there appears to be a clean install in

Installed /Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg
Processing dependencies for astropy==3.2rc1
Searching for numpy==1.14.5
Best match: numpy 1.14.5
Adding numpy 1.14.5 to easy-install.pth file

Using /sw/lib/python3.7/site-packages
Finished processing dependencies for astropy==3.2rc1

> PYTHONPATH=~/lib/python3.7/site-packages python3.7                                                                                                                        
Python 3.7.3 (default, May  5 2019, 04:25:55) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> import astropy.table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/__init__.py", line 41, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/column.py", line 26, in <module>
    from . import groups
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/groups.py", line 7, in <module>
    from .index import get_index_by_names
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/table/index.py", line 38, in <module>
    from astropy.time import Time
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/time/__init__.py", line 2, in <module>
    from .formats import *
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/time/formats.py", line 16, in <module>
    from astropy import _erfa as erfa
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/Users/derek/lib/python3.7/site-packages/astropy-3.2rc1-py3.7-macosx-10.12-x86_64.egg/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

so I think this is the problem to focus on rather than the _multiarray_umath one.

[dhomeier]

Doh, of course! In between the rebuilds I had unpacked the PyPI tarball inside the git source tree, just to double-check that it was really identical to the git v3.2rc1 tag, thus also extracting the C sources which are not under git control. Removing those .c files and re-cythonising everything in master, then checking out v3.2rc1 again finally produced a working build, and let the tests run with 4 failures in wcsaxes (and a couple thousand lines of warnings).
Duly, if I afterwards update numpy to 1.16.3, I am reproducing #8672:

    def test_output_dim_3_signature():
        if NUMPY_LT_1_16:
            assert erfa_ufunc.c2i00a.signature == "(),()->(d3, d3)"
        else:
>           assert erfa_ufunc.c2i00a.signature == "(),()->(3, 3)"
E           AssertionError: assert '(),()->(d3, d3)' == '(),()->(3, 3)'
E             - (),()->(d3, d3)
E             ?         -   -
E             + (),()->(3, 3)

IOW it seems the numpy version the package is configured for does not even depend on the one installed on the machine the package is built, but the one the tarball has been created on. Seems tricky to get around this, as I believe for standard builds from source Cython is not expected as a requirement...

[pllim]

@dhomeier , since you are now reproducing #8672, can we close this issue? As for the 4 wcsaxes failures, please open a new issue, if it is not already reported. Thanks!

[dhomeier]

Well, the issue remains that building a source checkout is not guaranteed to create modules that will work on the build machine, unless one is re-running cython for the C sources in question. But I could add this as a related case to #8672 if you prefer to have it all in one issue.

[pllim]

@dhomeier , if your error message is now exactly the same as #8672, I do indeed prefer to move the remaining discussion there. Thanks!

[dhomeier]

@plim, the error message if I simply run python setup.py build in the unpacked astropy-3.2rc1.tar.gz is still the original one above. If I manually remove astropy/_erfa/ufunc.c to have it recreated during the build, then install the package, then upgrade numpy to 1.16, the error message is the same as #8672.

@mhvk I overlooked that ufunc.c is not created from a .pyx but by erfa_generator.py, so just removing it to have it recreated from the ufunc.c.templ should at least fix the build from source without needing a local Cython installation. But as you pointed out above, that will only shift the problem to the compilation stage...

[mhvk]

@dhomeier - thanks for all the sleuthing - at least, we understand now where the problem lies. Remains to understand how best to solve it...

[bsipocz]

Reopening as we see this all over again with conda packages astropy installs when using numpy 1.13, 1.14, or 1.15.

[bsipocz]

To summarize the slack thread:

• we see this for any downstream CI where coordinates/time/table is imported (as all use erfa).
• it doesn't matter whether conda-forge or conda default packaged packages are used, both are producing the same issue.
• no issues with pip installed version combinations
• @mwcraig is looking into our conda build.

[mwcraig]

Good news: I understand the problem (or at least
Bad news: I can't fix it, it isn't really a conda problem. The same failure happens if you build from the released tarball. Even more fun, it will never happen when you build from a clean checkout of the source repo, as when wheels are built.

The problem:

The erfa ufunc code ufunc.c is auto-generated from a jinja template. It is regenerated if:

• the template file ufunc.c.templ is newer than the generated file ufunc.c, unless the the files are part of a release (as indicated by astropy/version.py).
• the file ufunc.c does not exist, as is the case in a source checkout.

There is a critical bit of the template at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 that needs to be included if building for numpy less than 1.16.

The problem is that the check for the numpy version is done at the time the sdist is generated, not at the time the package is compiled.

I'm guessing the sdist was done on a machine with numpy 1.16 or higher. Because of https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 the generated ufunc.c is missing the bit needed to build a working binary on numpy less than 1.16.

If the release had been done a machine with numpy less than 1.16 then I'm guessing that the building the release tarball against numpy 1.16 might fail? No, apparently it does not.

Not sure what the fix is here. The current approach at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 doesn't work because the numpy version at build time isn't known.

To reproduce this, grab the source tarball and setup.py install in a an environment with numpy 1.14. Then try import astropy._erfa and you'll get the error.

Ping @bsipocz @mhvk

[bsipocz]

I'm guessing the sdist was done on a machine with numpy 1.16 or higher

yes, that's correct.
Would you think regenerating the sdist with np 1.15 and uploading it as a post release (or even as 3.2.1) would solve the issue?

[bsipocz]

(as, I don't mind if I need to generate the sdist in an env with older numpy, we just need to add it to the release notes (and then I need to remember, and hope that we can get rid of the old version in the next LTS 😉 ))

[mhvk]

I'm so sorry to hear the problems caused by me trying to be clever and use the enhancement I made to numpy (which I made because of my of work on _erfa!).

As you correctly note, the code generated for an older numpy works fine for 1.16, so it does seem that the easiest for now is to do that.

Or is it possible to just not include the generated code at all, and add jinja2 as a build requirement?

[mwcraig]

In the short run I’d say do a post release from an older numpy is probably least disruptive.

The only potential downside of requiring jinja2 is the lack of a way to indicate what is required for building. We could add some import checks to setup and raise informative errors if anything is missing.

[bsipocz]

I think then I would go ahead and do another proper releases, that would in turn make the sunpy folks also happy to get their bugfixes released in 3.2.1.

[mhvk]

Sounds good. If we have numpy >=1.16 for 4.0, then the problem will go away...

[bsipocz]

yes, this is a good reason to have 1.16 :)

[bsipocz]

New releases have been made, and uploaded to pypi. The conda-forge packages are also ready, but it'll take a few days for them to end up on the default channels.

@mwcraig - please do check whether everything is now working as expected (I made the sdists this time with np 1.14.6).

[bsipocz]

(Also, please let me know either way, whether they work or not)

[bsipocz]

I've tested on the astroML repo that the new releases indeed work, so closing this issue now once and for all. Thank you everyone for trouble shooting and figuring out the solution.

[mwcraig]

The new conda package seems to work ( at least import astropy._erfa succeeds).

For the future should we run the conda-forge tests against the lowest supported version of numpy? If yes, can do a PR to add that to the conda-forge recipe. It is, I think, one of the only places in our infrastructure where we build, and do some limited testing of, astropy built from the source distribution.

[pllim]

should we run the conda-forge tests against the lowest supported version of numpy?

It cannot hurt. :+1: from me.

[bsipocz]

@mwcraig - yes, I think you're right, the wheels are built from the source tag (they will be build from sdist when we switch to the new machinery by @astrofrog, but even there I would be more comfortable to build from source tag rather than pypi released sdist).

[mhvk]

@mwcraig - that would be wonderful! Where does/would that testing happen?

[mwcraig]

The testing would happen at https://github.com/conda-forge/astropy-feedstock, and only after a release is made, so it isn't ideal. It might have helped identify the issue a little sooner, though.