Rasterio: Numpy header requirement in setup.py breaks cloud deploys

[sachsbl]

We're running into issues deploying apps with Rasterio into cloud platforms such as Cloud Foundry. The issue is that Rasterio's setup.py tries to import numpy, which implies that your environment already has numpy installed. That makes it very difficult/impossible to install all dependencies for our app in one pip install (which is what many cloud platforms expect). This requirement seems somewhat non-standard for python apps. Can this be changed? Is there a hard requirement that numpy be installed first? Thanks!

[perrygeo]

The numpy headers are required to build the C extension modules and need to be installed before setup. But I think we can follow the lead of the SciPy setup.py here, which contains some tricks for building scipy and installing numpy in one step.

[ocefpaf]

@perrygeo I took a stab at it b/c this bother me for a while too. See https://github.com/mapbox/rasterio/pull/1099 (but feel free to close it if you have a better way to implement that.)

[sgillies]

@sachsbl Rasterio is a bit different from other Python packages in that 1) it has C extension modules, and 2) the C extension modules depend on non-standard packages: not only Numpy and Cython but also GDAL and its universe of dependencies. Even if we changed Rasterio's setup script to go and fetch Cython and Numpy, we'd be left with the GDAL dependency issue, no?

@perrygeo @ocefpaf I don't follow how that code helps. I'm unable to imagine a case where you'd want to run the setup.py script and not build Rasterio's extension modules. It doesn't work without them.

[ocefpaf]

@perrygeo @ocefpaf I don't follow how that code helps. I'm unable to imagine a case where you'd want to run the setup.py script and not build Rasterio's extension modules. It doesn't work without them.

The goal is to delay the extension build to after pip installed numpy. Although, as you already mention, b/c there are no wheels for gdal the original problem will remain.

[sachsbl]

@perrygeo @ocefpaf At least in my application, the difference is that pip is capable of installing numpy and therefore its not part of the things we pre-compile and make available as binaries in the target cloud container. In our cloud ecosystem, the assumption is that anything pip can handle will be in a requirements.txt file, and any required binaries (GDAL) will be available as part of the cloud container's "buildpack". That approach seems relatively standard. I agree that the GDAL issue would remain even if we fixed numpy, but in my mind that is an acceptable split between things that must be compiled on a per-OS/per-system basis and things that pip can handle dynamically.

[ocefpaf]

@sachsbl my PR #1099 is what I use on my projects. It is based on pandas' approach to solve the problem. However, it is not working here. For some reason numpy is installed but not found by pkg_resources. Note that the CI configuration here won't catch that because numpy is installed before the extensions are built. I'll investigate what is going on and maybe we can at least make it easier for the numpy case.

[sachsbl]

@ocefpaf Thanks for taking a stab at this! I've been busy with some other stuff but I will also take a crack at it in the next week or so. I'm not sure the best approach but I'd like to see this working

[ocefpaf]

@sachsbl I am closing this b/c I could not make it work for rasterio. Not sure why b/c this work for most of my projects that suffers from the same issue :unamused:

Feel free to copy the changes here if you want to move this forward. Also, take a look at how matplotlib solves this. I believe their import trick might be the missing piece to make this work.

Note that to test this here it would require some changes to the CI. You can test it locally with an env without numpy though.

[sgillies]

@sachsbl are Rasterio's binary wheels not an option for you? There are none for Linux at 0.36 but if you pip --pre you'll get rasterio 1.0a9. It's technically a pre-release but is better and more bug-free than 0.36.

@ocefpaf Is it actually possible to delay the extension building until after the install_requires packages have been installed? I've never seen this documented anywhere.

[sachsbl]

@sgillies @ocefpaf do the binary wheels bundle numpy in there? I could try that. I've seen inconsistencies getting those to work on various Linux platforms. We are using 1.09a as we need the vsis3 stuff in there.

This project seems to be doing this also:

https://github.com/kwgoodman/bottleneck/blob/master/setup.py

Seems simpler than the matplotlib approach, which Im not sure I fuly understand....

[ocefpaf]

@sachsbl I am biased but I recommend you to use conda and the conda package. It should give you more control over the numpy/gdal/rasterio version you want with all pre-compiled binaries.

[sachsbl]

We are also working to modify our deployment process to just accommodate RasterIO as-is. It seems somewhat odd, but I understand the complexities of this issue and we are committed to using this project regardless.

[sachsbl]

closing due to lack of easy solution. we implemented a workaround to install numpy and rasterio in 2 separate steps via pip.