Nixpkgs: Theano package not working once installed

[krtab]

Describe the bug
Theano installation is not working:

nix-shell -p 'python3.withPackages(ps: with ps; [ Theano ])' --run "python -c 'import theano'"

returns

You can find the C code in this temporary file: /run/user/1000/theano_compilation_error_fvjx9y_e
Traceback (most recent call last):
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/gof/lazylinker_c.py", line 81, in <module>
    actual_version, force_compile, _need_reload))
ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.211, but found None. Extra debug information: force_compile=False, _need_reload=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/gof/lazylinker_c.py", line 105, in <module>
    actual_version, force_compile, _need_reload))
ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.211, but found None. Extra debug information: force_compile=False, _need_reload=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/__init__.py", line 110, in <module>
    from theano.compile import (
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/compile/__init__.py", line 12, in <module>
    from theano.compile.mode import *
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/compile/mode.py", line 11, in <module>
    import theano.gof.vm
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/gof/vm.py", line 674, in <module>
    from . import lazylinker_c
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/gof/lazylinker_c.py", line 140, in <module>
    preargs=args)
  File "/nix/store/pf0dk0bjk0sgmwsskyqh57g79y3jrrbn-python3-3.7.6-env/lib/python3.7/site-packages/theano/gof/cmodule.py", line 2396, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: Compilation failed (return status=1): impure path `/run/user/1000/ccX90FGJ.o' used in link. collect2: error: ld returned 1 exit status. 

To Reproduce
Run

nix-shell -p 'python3.withPackages(ps: with ps; [ Theano ])' --run "python -c 'import theano'"

Expected behavior
The module should be loaded

Notify maintainers
@bcdarwin

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 5.7.4-arch1-1, Arch Linux, noversion`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.6`
 - channels(arthur): `""`
 - channels(root): `"nixpkgs-20.03.2310.fb6c3a6831c"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Related issues:

• #73051

[bcdarwin]

This is unfortunate, but I don't use Theano these days (feel free to take over as maintainer if you like).

(Note that getting CUDA to work on a non-NixOS system is when I last checked a bit of a black art as well ...)

[krtab]

This is unfortunate, but I don't use Theano these days (feel free to take over as maintainer if you like).

(Note that getting CUDA to work on a non-NixOS system is when I last checked a bit of a black art as well ...)

I'm only using it because of pymc3, so I can't say that I understand a whole lot about it.
@ilya-kolpakov : Maybe this issue is of interest to you as maintainer of pymc3?

Regarding maintaining theano, it seems from the blame that @twhitehead and @abbradar have been working on it recently, maybe one of them would like to become maintainer? Sorry if mentioning like that is rude.

[danieldk]

There is a discussion about this issue in: https://github.com/NixOS/nixpkgs/issues/45463 with some workarounds.

(Doesn't solve the problem, but provides useful background on why this error happens.)

[twhitehead]

Yes indeed. The issue, as noted in that thread, is that nix-shell sets

TMP=/run/user/<uid>
TMPDIR=/run/user/<uid>
TEMP=/run/user/<uid>
TEMPDIR=/run/user/<uid>

while the build support wrapper assumes temporary directories are under /tmp. Unsetting these temporary environment variables before running python should address this.

A better long term solution though would be if the build support wrapper used the value of $TMP (if set) instead of just always assuming /tmp. I think changing the line linked to above from

        "${p:0:4}" != "/tmp" -a \

to

        "${p:0:${#TMP}${TMP:-4}}" != "$TMP${TMP:-/tmp}" -a \

would likely do the trick. The issue would be that it would likely require a mass rebuild.

[krtab]

Hi!

Thanks for your input and your work on nixpkgs. I'm not very familiar with how things work governance wise but if I understand correctly the change you suggest should be discussed with more members of the Nix team before being implemented, do you know how we should proceed?

[risicle]

@krtab nah, you just create a PR and the discussion happens there. If people don't like it in the end, they don't merge it.

Further to this issue, I think any dependence on an environment variable should be avoided. Environment variables are somewhat controllable through wrapping when the package is a binary/executable, but for a python module we have no way of knowing/controlling what the env is going to look like when theano is used. And we shouldn't expect people using a nix-provided theano to have to make any special provisions env-wise to make their application work.

To me it seems we should be trying to avoid using a wrapped compiler at all in this package.

[twhitehead]

Just to follow up on this, this should now be fully addressed with the merge of #97597.

[twhitehead]

As a follow up to what compiler to use.

Before my original patch to use the buildtime compiler, it used the one out of your environment. This was problematic as there is not guarantee that the code produced with that compiler was compatible with code produced with the other compiler.

This was especially true when running nix on a non-NixOS system (such as our super computer cluster at Compute Canada, where it wouldn't run at all), but it can also be true on NixOS as there are C++ compatibility issues that makes mixing things not recommended. To get an idea of the issues, see, for example,

https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html
https://lwn.net/Articles/632118/

At the time I figured capturing the buildtime compiler was the ultimate solution to this mess, as it would be 100% compatible with the package itself. Since learning about cross compilation, I now know we actually want the runtime version of the buildtime compiler. Ugh. So far no one cross compiled theano and noticed this though. :grin: