Nixpkgs: pythonPackages.gdal broken on x86_64-darwin

[elasticdog]

Issue description

This package is reporting that it is not supported on darwin systems. I discovered this issue when trying to update pythonPackages.ansible2, which had worked cleanly previously (I last updated probably a month ago). I looked through the nixpkgs Git log to see if any recent _gdal_ changes were pushed related to OS X specifically, but I did not see anything too obvious. Perhaps this commit 674aa05abca627d?

Steps to reproduce

$ nix-env -iA nixpkgs.gdal
error: Package ‘gdal-2.1.1’ in ‘/nix/store/3ix4qwxv1l4czrsb4ipy2wv08pxrs7mh-nixpkgs-16.09pre90351.e446b9f/nixpkgs/pkgs/development/libraries/gdal/default.nix:59’ is not supported on ‘x86_64-darwin’, refusing to evaluate.
a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build` or any other Nix command you can add
  { allowBroken = true; }
to ~/.nixpkgs/config.nix.

Technical details

• System: OS X El Capitan (10.11.6)
• Nix version: nix-env (Nix) 1.11.3
• Nixpkgs version: "16.09pre-git" (nixpkgs-unstable channel updated on 2016-09-01)

[FRidh]

pkgs/development/libraries/gdal/default.nix says platforms = stdenv.lib.platforms.linux;

[FRidh]

seems like its been like that for a long time. The question I think is more, how come ansible depends on gdal? Perhaps through django? cc @lancelotsix

[FRidh]

No, not through django. I don't see any dependency on gdal with nix-store -q --tree /nix/store/kp1niba7h662rks42icplrdsjxzbzqdp-python2.7-ansible-2.1.1.0

[elasticdog]

I hadn't thought about it being pulled in from a downstream dependency change. That said, I'm trying to git-bisect my way through to see where it might have happened. I don't see gdal as a dependency in the tree either, but wouldn't the stuff in the nix store only be what built successfully previously?

[lsix]

I am running nix-build -A pythonPackages.ansible2 locally, and it is pulling django. Then, I suppose that the commit responsible for the build breakage wold be 37ad0f8e6a9830acda2e4af417b7f8fd386a861d

I am trying to figure out why django is necessary to build ansible…

[lsix]

The dependency chain is:

• ansible2
• pycrypto
• cryptography
• hypothesis1
• django

The real question is: Is really gdal unavailable for darwin ? The platform tag comes from 0d6228fa which is the commit that introduced gdal (dated of 2010).

I do not have a darwin box to give it a thy. @elasticdog if you do have one, could you try to mark darwin as an accepted platform and try to build it all ?

[elasticdog]

I marked darwin as an accepted platform for gdal, then had to do the same for libgeotiff, but then nix-build -A pythonPackages.ansible2 is taking a very long time to complete (I will report back once it finishes either way). That said, I have no idea how to actually test that gdal and libgeotiff are working as expected, if they do compile.

The other question would be, why does hypothesis pull in Django? My understanding is that Hypothesis has no actual dependencies, but supports extra optional deps per the upstream docs, in this case to support testing Django models.

[elasticdog]

Building gdal eventually failed:

building 'osgeo._gdal' extension
creating build/temp.macosx-10.7-x86_64-2.7
creating build/temp.macosx-10.7-x86_64-2.7/extensions
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O0 -Wall -Wdeclaration-after-statement -Wextra -Winit-self -Wunused-parameter -Wmissing-prototypes -Wmissing-declarations -Wformat -Werror=format-secur
ity -Wno-format-nonliteral -Wshorten-64-to-32 -Wshadow -Werror=vla -Wdeclaration-after-statement -Wnull-dereference -Wno-error=unused-command-line-argument-hard-error-in-future -I../../port -I../../gcore -I..
/../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/nix/store/27w9z2z9idgz6g52kfhndhd5s329svhj-python-2.7.12/include/python2.7 -I/nix/store/6zkl0d2gsz7ia45w1q1m7xagg27zww31-python2.7-nump
y-1.11.1/lib/python2.7/site-packages/numpy/core/include -I/private/var/folders/dh/d4h25vhs6pv71k49_7ywltjw0000gn/T/nix-build-python2.7-gdal-2.1.1.drv-0/gdal-2.1.1/include -c extensions/gdal_wrap.cpp -o build/
temp.macosx-10.7-x86_64-2.7/extensions/gdal_wrap.o
clang-3.7: warning: argument unused during compilation: '-fno-strict-overflow'
warning: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused-command-line-argument'? [-Wunknown-warning-option]
extensions/gdal_wrap.cpp:2440:65: warning: declaration shadows a local variable [-Wshadow]
SWIG_Python_NewShadowInstance(SwigPyClientData *data, PyObject *swig_this)
                                                                ^
extensions/gdal_wrap.cpp:2181:18: note: previous declaration is here
static PyObject *swig_this = NULL;
                 ^
extensions/gdal_wrap.cpp:2515:51: warning: declaration shadows a local variable [-Wshadow]
SWIG_Python_SetSwigThis(PyObject *inst, PyObject *swig_this)
                                                  ^
extensions/gdal_wrap.cpp:2181:18: note: previous declaration is here
static PyObject *swig_this = NULL;
                 ^
extensions/gdal_wrap.cpp:3017:10: fatal error: 'stdexcept' file not found
#include <stdexcept>
         ^
3 warnings and 1 error generated.
error: command 'clang' failed with exit status 1
make[2]: *** [GNUmakefile:76: build] Error 1
make[2]: Leaving directory '/private/var/folders/dh/d4h25vhs6pv71k49_7ywltjw0000gn/T/nix-build-python2.7-gdal-2.1.1.drv-0/gdal-2.1.1/swig/python'
make[1]: *** [GNUmakefile:30: build] Error 2
make[1]: Leaving directory '/private/var/folders/dh/d4h25vhs6pv71k49_7ywltjw0000gn/T/nix-build-python2.7-gdal-2.1.1.drv-0/gdal-2.1.1/swig'
make: *** [GNUmakefile:104: swig-modules] Error 2
builder for ‘/nix/store/xw203cschbkqm6g0278yqp4ln1dgcp73-python2.7-gdal-2.1.1.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/2m8inss6gk5nagvdps7am95fpmgn3jl7-1.10-gis-libs.template.patch.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/64jxd1j3ghwvp730wypggfn2isg60wa8-python2.7-pep8-1.7.0’
cannot build derivation ‘/nix/store/dg21bp10qkpy2kzqhv4b5vi31zz7k8jx-python2.7-Django-1.10.drv’: 1 dependencies couldn't be built
killing process 1347
cannot build derivation ‘/nix/store/mcrz328imcsy9zcdj1q5xp4fdzrcj3dc-python2.7-hypothesis-1.14.0.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/idfkk84vjwvlcx359q4p1skhg4h8bksz-python2.7-pretend-1.0.8’
killing process 1349
cannot build derivation ‘/nix/store/wr1kywrlcamb13nhycqapzbfy5743f47-python2.7-cryptography-1.5.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/zm0icyi8b5nzq2wvlgdxnnkv9jc69j66-python2.7-singledispatch-3.4.0.3’
cannot build derivation ‘/nix/store/6n1wviz2y0v8lbv85g632fkw2cyc702b-python2.7-paramiko-2.0.2.drv’: 1 dependencies couldn't be built
killing process 1351
cannot build derivation ‘/nix/store/7iqcjl28296hwhgx80mn02y0dhvqk1wn-python2.7-ansible-2.1.1.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/7iqcjl28296hwhgx80mn02y0dhvqk1wn-python2.7-ansible-2.1.1.0.drv’ failed

[elasticdog]

Looking at the hypothesis1 package dependencies, shouldn't all of these be optional:

fake_factory django numpy pytz

???

[FRidh]

@lancelotsix fixing gdal is indeed the right solution.

Even so, I was thinking we could maybe split Django into multiple outputs. However, that seems like a lot of work, and for what really. We could make a Django-small where we drop such dependencies, or, even easier and in my opinion just fine, and that is to drop django as a dependency of hypothesis. If someone wants to use hypothesis to test Django models, then they can just include Django themselves. We just need to put a note in the hypothesis expression then.

[FRidh]

@elasticdog we cannot make everything optional as we would end up with a lot of combinations.

[FRidh]

@elasticdog Often we choose to have packages that include everything they could need, and sometimes the opposite strategy is chosen. If you want to have it differently, you can override the function and change it yourself. However, it gets a bit harder to propagate that change so that dependents will use that change as well.

[FRidh]

@elasticdog hypothesis doesn't depend anymore on its optional dependencies so you can install ansible2 without depending on gdal
c319c842b70440c8e068b88dee0d6a237509b9d8

[lsix]

@FRidh a django_small will probably cause a lot of problems. Many django modules have django as propagatedBuildInput, therefore someone cannot have both those modules and django_small without also pulling django. This is why I dropped the django_gis variant since it was not usable as is. Splitting django would also be a lot of troubles I guess.

I’m perfectly fine with dropping optional dependencies for hypotethis, but I’ll run that locally to check that we do not break too much things.

[elasticdog]

@FRidh awesome, although it looks like that change also needs to be applied to the hypothesis1 package, which is what Ansible eventually pulls in?

[copumpkin]

Wait, why should ansible2 depend on django? that seems wild to me.

[copumpkin]

Oh, you're saying that it doesn't anymore, so that seems fine. We should definitely fix gdal on darwin, but ansible shouldn't depend on django either 😄

[elasticdog]

Looks like you just removed the hypothesis1 package (via https://github.com/NixOS/nixpkgs/commit/81191b707dcfed55eab29f032dce1b3091d5f674), so that should work too...will give it a shot. Thank you @FRidh and @lancelotsix!

[lsix]

@copumpkin pythonPackages.cryptography (a legitimate dependency of ansible2) uses pythonPackages.hypothesis for its testing. It happens to be that pythonPackages.hypothesis can be used to test django models and provides a helper module for that (hypothesis.extra.django). For someone intending to use only hypothesis “core”, the django dependency is unnecessary. The previous approach was to include all the optional dependencies of hypothesis. Now on, if someone wants to use it to test django models, he will need to depend on hypothesis and add manually depend on django (which he will obviously if he wants to test django models anyway)

[copumpkin]

@lancelotsix makes sense, thanks :smile:

[knedlsepp]

I'm looking into this. Currently I found that the swig-python extension is not building because the builder uses clang compiling a C++ file, when it should actually use clang++, so obviously it can't find <stdexcept>.
_Edit:_
Looks like unsetting CXX is the culprit.
_Edit 2:_
Oddly enough removing the unset CXX is still not enough for the builder to pick clang++.

[knedlsepp]

This can be closed, now that https://github.com/NixOS/nixpkgs/pull/20242 is merged.

[langston-barrett]

@knedlsepp I'm still seeing the error about not finding <stdexcept>, even with nixpkgs rev adb178c0e5e7fea2bd2bfa8def7d4bd1d841b327 (Jul 18, 2017). Are we sure that gdal is fixed on OSX?

[knedlsepp]

Hydra at least builds it just fine https://hydra.nixos.org/job/nixpkgs/trunk/gdal.x86_64-darwin what's your exact settings used so you cannot use the Hydra binaries?

[langston-barrett]

@knedlsepp What do you mean by "settings"? Here's some build info:

• OSX: 10.11.6
• nix: 1.11.13
• nixpkgs: 17.09pre111304.5328102325
• clang --version

Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin```

And a huge build log: https://gist.github.com/siddharthist/74ccfb272ba32d0122f8ebb7139bf759

Thanks for your quick response and willingness to take a look at this!

[copumpkin]

@siddharthist it looks like an impurity to me. I see in your configure script output, you have this suspicious line:

checking for Expat XML Parser headers in /usr/include... found

You can then see it passing -I/usr/include into all subsequent compiler invocations, which has the effect of bringing a bunch of system headers (often incompatible with the Nix ones as you're seeing) into scope.

It may not be obvious to you (based on your clang --version above), but Nix makes a point of using almost nothing from the host system. That includes system headers, Xcode, the macOS command-line tools, or much else you can think of. We have our own toolchain, headers, libraries, and so on.

For a while we had a working sandbox system on macOS that automatically isolated all builds from the tempting impurities of /usr but that broke and nobody's had time to fix it, so for now we generally do our best to encourage packages not to look outside the Nix store, but sometimes they'll try to be clever and poke where we don't want them to. The solution is generally to tell them explicitly where to find the thing they're looking for (e.g., in your case, perhaps passing a --without-expat to configure) to stop them from poking where they shouldn't.

[knedlsepp]

I see. This is more of a python36 issue. While nix-shell -p gdal and nix-shell -p python27Packages.gdal work for me, nix-shell -p python36Packages.gdal is broken. So are other compiled python libraries like: pandas [edit: Ok. This one seems broken for a different reason]. I think this is an incarnation of the "setuptools ignores that you need clang++ for C++ and simply uses the clang frontend." (Which works for the xcode clang, because it includes C++ headers, but not for ours)

[langston-barrett]

@copumpkin Thanks for explaining! I knew that was the case on Linux, I'm just starting to mess with Nix on OSX. Can we re-open this issue?

[knedlsepp]

Currently trying to fix this via: https://github.com/knedlsepp/nixpkgs/tree/fix-python3x-distutils
I'm doing it analogously to #19585. Looks good so far. Just checking if I also need to patch numpy.

[copumpkin]

Reopening for now!

[langston-barrett]

@knedlsepp You probably already realize this but just in case: I have the same error when building with Python 3.5, so it looks to be an issue not just with 3.6.

[knedlsepp]

Yup. Fix for 3.4+ on the way.

[Ekleog]

(triage) Is this still an issue? Current gdal appears to be linux ++ darwin, though I don't have any darwin machine to test.

[knedlsepp]

Should be fine by now: https://hydra.nixos.org/job/nixpkgs/trunk/python36Packages.gdal.x86_64-darwin

[Ekleog]

Great! :)