Spinalcordtoolbox: `PYTHONNOUSERSITE=True` leaks into shell scripts run by `sct_run_batch`

Created on 26 Jan 2021  ·  31Comments  ·  Source: spinalcordtoolbox/spinalcordtoolbox

Description

I am using sct_run_batch and in the shell script called, I call a command from gradient_unwarp. I downloaded SCT 5.1.0 and when calling this line :
~
gradient_unwarp.py ${file_t1}.nii.gz ${file_t1}_gradcorr.nii.gz siemens -g ${PATH_GRADCORR_FILE}/coeff.grad -n
~
This error occurs:
~~~

  • gradient_unwarp.py sub-1000394_T1w_raw_RPI_r.nii.gz sub-1000394_T1w_raw_RPI_r_gradcorr.nii.gz siemens -g /mnt/y/sebeda/coeff.grad/ICM/coeff.grad -n
    Traceback (most recent call last):
    File "/usr/local/bin/gradient_unwarp.py", line 11, in
    from gradunwarp.core import (globals, coeffs, utils)
    File "/usr/local/lib/python3.7/dist-packages/gradunwarp/core/__init__.py", line 8, in
    from .unwarp_resample import Unwarper
    File "/usr/local/lib/python3.7/dist-packages/gradunwarp/core/unwarp_resample.py", line 8, in
    import numpy as np
    ModuleNotFoundError: No module named 'numpy'
    ~~~
    I tried runnig the same command in my terminal, there was no error. I went back to SCT v5.0.1 and there was no error, the command worked.
    I am running on WSL1 with Ubuntu 18.04.

Steps to Reproduce

See README.md (preprocessing section)

  1. Install SCT 5.1.0

  2. Initialize shell variable with path to gradient correction file
    ~
    PATH_GRADCORR_FILE=
    ~

  3. Run preprocessing (see preprocess_data.sh)
    ~
    sct_run_batch -jobs -1 -path-data -path-output ~/ukbiobank_preprocess -script preprocess_data.sh -script-args $PATH_GRADCORR_FILE
    ~

Expected behavior: Using SCT v5.0.1 or running the command in the terminal:
~
gradient_unwarp.py sub-1000032_T1w_raw_RPI_r.nii.gz sub-1000032_T1w_raw_RPI_r_gradcorr.nii.gz siemens -g /mnt/y/sebeda/coeff.grad/ICM/coeff.grad -n
gradunwarp-INFO: Parsing /mnt/y/sebeda/coeff.grad/ICM/coeff.grad for harmonics coeffs
gradunwarp-INFO: Evaluating spherical harmonics
gradunwarp-INFO: on a 60^3 grid
gradunwarp-INFO: with extents -300.0mm to 300.0mm
gradunwarp-INFO: along x...
...
~

Actual behavior: Here is the complete log file of the processing error.
err.preprocess_data_sub-1000394.log

compat bug sct_run_batch
Source
picture sandrinebedard

Most helpful comment

it just needs to set to nothing. (PYTHONNOUSERSITE=)

It works!!! 🎉

picture sandrinebedard on 29 Jan 2021
🎉2

All 31 comments

So, the error seems to originate from the gradunwarp package. Did you make sure to install this package with all the dependencies? Can you run it without error from the Terminal?

jcohenadad picture jcohenadad on 26 Jan 2021

Did you make sure to install this package with all the dependencies?

Yes!

Can you run it without error from the Terminal?

Yes, there are no errors when I run it from the terminal or using SCTv5.0.1 @jcohenadad

sandrinebedard picture sandrinebedard on 26 Jan 2021

ah yes sorry-- you already mentioned earlier that it works as a standalone from the Terminal... this is very strange because sct_run_batch is essentially a wrapper for shell commands.
What is the output of:
~
which gradient_unwarp.py
~

jcohenadad picture jcohenadad on 26 Jan 2021

The output is:
~
/usr/local/bin/gradient_unwarp.py
~

sandrinebedard picture sandrinebedard on 27 Jan 2021

hum, i would have to dig deeper-- @joshuacwnewton @kousu @Drulex do you have any clue?

jcohenadad picture jcohenadad on 27 Jan 2021

A quick observation: gradunwarp installs itself using sudo python setup.py install.

This is (imo) a concerning way of doing things, because it forces folks to install gradunwarp (and its dependencies) to the system Python. Their installation method makes it harder to manage and isolate dependencies (among other concerns), and might be what lead to a situation like this. I'm reminded of this comic.


EDIT: I've retracted my recommendations, see @Drulex's comment below

To make sure numpy is present in the same location as gradunwarp, could you try installing numpy using sudo? ~~

> sudo pip3 install numpy

~~ This should ensure that numpy is installed to your system Python's package directory (which should be /usr/local/lib/python3.7/dist-packages/).

joshuacwnewton picture joshuacwnewton on 28 Jan 2021

To make sure numpy is present in the same location as gradunwarp, could you try installing numpy using sudo?

I would highly discourage using sudo pip install anything because it's a major security risk. That's basically running arbitrary code from the internet as root.

Instead gradunwarp should be installed without root privileges.

Drulex picture Drulex on 28 Jan 2021

Instead gradunwarp should be installed without root privileges.

That was my thinking, too. But, then this happens:


Compile log hidden, click to show 

joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ python3 setup.py install
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
building extension "gradunwarp.core.interp3_ext" sources
building extension "gradunwarp.core.legendre_ext" sources
building extension "gradunwarp.core.transform_coordinates_ext" sources
building data_files sources
build_src: building npy-pkg config files
running build_py
package init file 'gradunwarp/core/tests/__init__.py' not found (or not a regular file)
package init file 'gradunwarp/core/tests/__init__.py' not found (or not a regular file)
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/local/bin/gradient_unwarp.py to 755
error: [Errno 1] Operation not permitted: '/usr/local/bin/gradient_unwarp.py'

Because of this, I went along with the sudo restriction, but... yeah. On second thought, I think it would be better to take a closer look at ukbiobank-spinalcord-csa's dependency on gradunwarp.

joshuacwnewton picture joshuacwnewton on 28 Jan 2021

But have you tried the --prefix option? From https://github.com/Washington-University/gradunwarp#install:

image

Drulex picture Drulex on 28 Jan 2021

For example installing to local directory:


Compile log hidden, click to show 

[drulex@gentoo-p53 tmp]  $  git clone [email protected]:Washington-University/gradunwarp.git
Cloning into 'gradunwarp'...
remote: Enumerating objects: 378, done.
remote: Total 378 (delta 0), reused 0 (delta 0), pack-reused 378
Receiving objects: 100% (378/378), 76.08 KiB | 1.77 MiB/s, done.
Resolving deltas: 100% (206/206), done.
[drulex@gentoo-p53 tmp]  $  cd gradunwarp/
[drulex@gentoo-p53 gradunwarp] [master] $  python setup.py install --prefix ./
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
building extension "gradunwarp.core.interp3_ext" sources
building extension "gradunwarp.core.legendre_ext" sources
building extension "gradunwarp.core.transform_coordinates_ext" sources
building data_files sources
build_src: building npy-pkg config files
running build_py
package init file 'gradunwarp/core/tests/__init__.py' not found (or not a regular file)
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/gradunwarp
creating build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/core/__init__.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/core/coeffs.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/core/globals.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/__init__.py -> build/lib.linux-x86_64-3.7/gradunwarp
copying gradunwarp/core/utils.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/core/unwarp_resample.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
copying gradunwarp/core/gradient_unwarp.py -> build/lib.linux-x86_64-3.7/gradunwarp/core
creating build/lib.linux-x86_64-3.7/gradunwarp/core/tests
copying gradunwarp/core/tests/test_utils.py -> build/lib.linux-x86_64-3.7/gradunwarp/core/tests
package init file 'gradunwarp/core/tests/__init__.py' not found (or not a regular file)
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'gradunwarp.core.interp3_ext' extension
compiling C sources
C compiler: x86_64-pc-linux-gnu-gcc -pthread -fPIC

creating build/temp.linux-x86_64-3.7/gradunwarp
creating build/temp.linux-x86_64-3.7/gradunwarp/core
compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c'
extra options: '-O3'
x86_64-pc-linux-gnu-gcc: gradunwarp/core/interp3_ext.c
In file included from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from gradunwarp/core/interp3_ext.c:10:
/usr/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gradunwarp/core/interp3_ext.c: In function ‘interp3’:
gradunwarp/core/interp3_ext.c:97:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
   97 |     itr_v = (PyArrayIterObject *) PyArray_IterNew(result);
      |                                                   ^~~~~~
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/interp3_ext.c:97:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/interp3_ext.c:98:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
   98 |     itr_r = (PyArrayIterObject *) PyArray_IterNew(R);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/interp3_ext.c:98:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/interp3_ext.c:99:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
   99 |     itr_c = (PyArrayIterObject *) PyArray_IterNew(C);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/interp3_ext.c:99:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/interp3_ext.c:100:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  100 |     itr_s = (PyArrayIterObject *) PyArray_IterNew(S);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/interp3_ext.c:100:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/interp3_ext.c:123:9: warning: returning ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’} from a function with incompatible return type ‘PyObject *’ {aka ‘struct _object *’} [-Wincompatible-pointer-types]
  123 |  return result;
      |         ^~~~~~
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. build/temp.linux-x86_64-3.7/gradunwarp/core/interp3_ext.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/gradunwarp/core/interp3_ext.cpython-37m-x86_64-linux-gnu.so
building 'gradunwarp.core.legendre_ext' extension
compiling C sources
C compiler: x86_64-pc-linux-gnu-gcc -pthread -fPIC

compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c'
extra options: '-O3'
x86_64-pc-linux-gnu-gcc: gradunwarp/core/legendre_ext.c
In file included from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from gradunwarp/core/legendre_ext.c:10:
/usr/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gradunwarp/core/legendre_ext.c: In function ‘legendre’:
gradunwarp/core/legendre_ext.c:86:25: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘long int’ [-Wformat=]
   86 |         printf("but mu=%d and nu=%d\n", mu, nu);
      |                        ~^               ~~
      |                         |               |
      |                         int             long int
      |                        %ld
gradunwarp/core/legendre_ext.c:86:35: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
   86 |         printf("but mu=%d and nu=%d\n", mu, nu);
      |                                  ~^         ~~
      |                                   |         |
      |                                   int       long int
      |                                  %ld
gradunwarp/core/legendre_ext.c:99:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
   99 |     itr_x = (PyArrayIterObject *) PyArray_IterNew(x);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/legendre_ext.c:99:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/legendre_ext.c:100:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  100 |     itr_r = (PyArrayIterObject *) PyArray_IterNew(result);
      |                                                   ^~~~~~
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/legendre_ext.c:100:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/legendre_ext.c:108:32: warning: format ‘%f’ expects argument of type ‘doubl ’, but argument 2 has type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’} [-Wformat=]
  108 |             printf("but got x=%f\n", x);
      |                               ~^     ~
      |                                |     |
      |                                |     PyArrayObject * {aka struct tagPyArrayObject_fields *}
      |                                double
gradunwarp/core/legendre_ext.c:157:12: warning: returning ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’} from a function with incompatible return type ‘PyObject *’ {aka ‘struct _object *’} [-Wincompatible-pointer-types]
  157 |     return result;
      |            ^~~~~~
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. build/temp.linux-x86_64-3.7/gradunwarp/core/legendre_ext.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/gradunwarp/core/legendre_ext.cpython-37m-x86_64-linux-gnu.so
building 'gradunwarp.core.transform_coordinates_ext' extension
compiling C sources
C compiler: x86_64-pc-linux-gnu-gcc -pthread -fPIC

compile options: '-I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -c'
extra options: '-O3'
x86_64-pc-linux-gnu-gcc: gradunwarp/core/transform_coordinates_ext.c
In file included from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from gradunwarp/core/transform_coordinates_ext.c:10:
/usr/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gradunwarp/core/transform_coordinates_ext.c: In function ‘transform_coordinates’:
gradunwarp/core/transform_coordinates_ext.c:102:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  102 |     itr_x = (PyArrayIterObject *) PyArray_IterNew(X);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:102:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/transform_coordinates_ext.c:103:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  103 |     itr_y = (PyArrayIterObject *) PyArray_IterNew(Y);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:103:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/transform_coordinates_ext.c:104:51: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  104 |     itr_z = (PyArrayIterObject *) PyArray_IterNew(Z);
      |                                                   ^
      |                                                   |
      |                                                   PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:104:51: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/transform_coordinates_ext.c:105:52: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  105 |     itr_xm = (PyArrayIterObject *) PyArray_IterNew(Xm);
      |                                                    ^~
      |                                                    |
      |                                                    PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:105:52: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/transform_coordinates_ext.c:106:52: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  106 |     itr_ym = (PyArrayIterObject *) PyArray_IterNew(Ym);
      |                                                    ^~
      |                                                    |
      |                                                    PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:106:52: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
gradunwarp/core/transform_coordinates_ext.c:107:52: warning: passing argument 1 of ‘(PyObject * (*)(PyObject *))*(PyArray_API + 784)’ from incompatible pointer type [-Wincompatible-pointer-types]
  107 |     itr_zm = (PyArrayIterObject *) PyArray_IterNew(Zm);
      |                                                    ^~
      |                                                    |
      |                                                    PyArrayObject * {aka struct tagPyArrayObject_fields *}
gradunwarp/core/transform_coordinates_ext.c:107:52: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject_fields *’}
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. build/temp.linux-x86_64-3.7/gradunwarp/core/transform_coordinates_ext.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/gradunwarp/core/transform_coordinates_ext.cpython-37m-x86_64-linux-gnu.so
running build_scripts
creating build/scripts.linux-x86_64-3.7
copying and adjusting gradunwarp/core/gradient_unwarp.py -> build/scripts.linux-x86_64-3.7
changing mode of build/scripts.linux-x86_64-3.7/gradient_unwarp.py from 644 to 755
running install_lib
creating lib
creating lib/python3.7
creating lib/python3.7/site-packages
creating lib/python3.7/site-packages/gradunwarp
copying build/lib.linux-x86_64-3.7/gradunwarp/__init__.py -> .//lib/python3.7/site-packages/gradunwarp
creating lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/transform_coordinates_ext.cpython-37m-x86_64-linux-gnu.so -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/legendre_ext.cpython-37m-x86_64-linux-gnu.so -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/interp3_ext.cpython-37m-x86_64-linux-gnu.so -> .//lib/python3.7/site-packages/gradunwarp/core
creating lib/python3.7/site-packages/gradunwarp/core/tests
copying build/lib.linux-x86_64-3.7/gradunwarp/core/tests/test_utils.py -> .//lib/python3.7/site-packages/gradunwarp/core/tests
copying build/lib.linux-x86_64-3.7/gradunwarp/core/gradient_unwarp.py -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/unwarp_resample.py -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/utils.py -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/globals.py -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/coeffs.py -> .//lib/python3.7/site-packages/gradunwarp/core
copying build/lib.linux-x86_64-3.7/gradunwarp/core/__init__.py -> .//lib/python3.7/site-packages/gradunwarp/core
byte-compiling .//lib/python3.7/site-packages/gradunwarp/__init__.py to __init__.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/tests/test_utils.py to test_utils.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/gradient_unwarp.py to gradient_unwarp.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/unwarp_resample.py to unwarp_resample.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/utils.py to utils.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/globals.py to globals.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/coeffs.py to coeffs.cpython-37.pyc
byte-compiling .//lib/python3.7/site-packages/gradunwarp/core/__init__.py to __init__.cpython-37.pyc
running install_scripts
creating bin
copying build/scripts.linux-x86_64-3.7/gradient_unwarp.py -> .//bin
changing mode of .//bin/gradient_unwarp.py to 755
running install_data
copying gradunwarp/core/interp3_ext.c -> .//lib/python3.7/site-packages/gradunwarp/core/
copying gradunwarp/core/transform_coordinates_ext.c -> .//lib/python3.7/site-packages/gradunwarp/core/
copying gradunwarp/core/legendre_ext.c -> .//lib/python3.7/site-packages/gradunwarp/core/
running install_egg_info
Writing .//lib/python3.7/site-packages/gradunwarp-HCP_1.2.0-py3.7.egg-info
running install_clib
customize UnixCCompiler
[drulex@gentoo-p53 gradunwarp] [master] $  PYTHONPATH+=./ ./bin/gradient_unwarp.py -h
usage: 
gradient_unwarp infile outfile manufacturer -g <coefficient file> [optional arguments]

positional arguments:
  infile                The input warped file (nifti or mgh)
  outfile               The output unwarped file (extension should be
                        .nii/.nii.gz/.mgh/.mgz)
  {siemens,ge}          vendor (either "ge" or "siemens" for now)

optional arguments:
  -h, --help            show this help message and exit
  --version, -v         show program's version number and exit
  -g GRADFILE, --gradfile GRADFILE
                        The .grad coefficient file
  -c COEFFILE, --coeffile COEFFILE
                        The .coef coefficient file
  -w, --warp            warp a volume (as opposed to unwarping)
  -n, --nojacobian      Do not perform Jacobian intensity correction
  --fovmin FOVMIN       the minimum extent of harmonics evaluation grid in
                        meters
  --fovmax FOVMAX       the maximum extent of harmonics evaluation grid in
                        meters
  --numpoints NUMPOINTS
                        number of grid points in each direction
  --interp_order ORDER  the order of interpolation(1..4) where 1 is linear -
                        default
  --verbose
[drulex@gentoo-p53 gradunwarp] [master] $

Of course you would need to update PYTHONPATH permanently and add the installation directory to PATH

Drulex picture Drulex on 28 Jan 2021

Sorry. I think I've distracted from the core problem here. While the system-wide gradunwarp installation is concerning, I don't think where gradunwarp is installed relates to the missing numpy dependency.

Chances are numpy is missing because it's has been installed in the system location instead of the user location (depends on the OS I guess), but using the non-root install would put it in the same place as any user pip installed package.

Drulex picture Drulex on 28 Jan 2021

Ah... right. If gradunwarp was installed via sudo, then it won't look in @sandrinebedard's user site-packages, it will try and look in /root/.

joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ sudo python3 -m site
sys.path = [
    '/home/joshua/Downloads/gradunwarp-1.2.0',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

Good point.

joshuacwnewton picture joshuacwnewton on 28 Jan 2021

@sandrinebedard Could you please try re-installing gradunwarp using a local installation?

If this works for you, I also think we should write a warning to avoid the sudo installation of gradunwarp and its dependencies, and add it to ukbiobank-spinalcord-csa's dependency section of the README.


Aside about gradunwarp's installation choices

gradunwarp is awkward to uninstall because it uses a distutils-based installation and not pip/setuptools. Typically, we wouldn't go near sudo rm to uninstall a Python package.

In fact, the original gradunwarp used numpy.distuils for its own installation. As far as I know, numpy.distutils is meant for internal numpy usage only. The most popular fork has an open PR from 2015 to fix this (https://github.com/Washington-University/gradunwarp/pull/1), but... I digress.

First, uninstall the existing gradunwarp. 

> sudo rm -rf /usr/local/lib/python3.7/dist-packages/gradunwarp*
> sudo rm -f /usr/local/bin/gradient_unwarp.py

Then, install gradunwarp to the ~/.local/ directory. This is also conveniently the location where pip would normally install packages to.

> cd Downloads/gradunwarp-1.2.0/
> python3 setup.py install --prefix ~/.local

Then, if you haven't already, add ~/.local/bin to your $PATH environment variable.

> echo 'export PATH="$HOME/.local/bin/:$PATH"' >> ~/.bashrc

Finally, install numpy, scipy, and other dependencies to your local Python installation using pip, rather than the system-wide installation. Then, retry sct_run_batch.

joshuacwnewton picture joshuacwnewton on 28 Jan 2021

Could you please try re-installing gradunwarp using a local installation?

@joshuacwnewton I am trying this right now, I'll get back to you!

sandrinebedard picture sandrinebedard on 28 Jan 2021
👍1

I tried re-installing gradunwarp using a local installation and with sct_run_batch, I get this:
~~~

  • gradient_unwarp.py sub-1000032_T1w_raw_RPI_r.nii.gz sub-1000032_T1w_raw_RPI_r_gradcorr.nii.gz siemens -g /mnt/y/sebeda/coeff.grad/ICM/coeff.grad -n
    Traceback (most recent call last):
    File "/home/sabeda/.local/bin/gradient_unwarp.py", line 11, in
    from gradunwarp.core import (globals, coeffs, utils)
    ModuleNotFoundError: No module named 'gradunwarp'
    ~~~

And gradient_unwarp.py still works fine on the terminal.

sandrinebedard picture sandrinebedard on 28 Jan 2021

Aha! The reason this issue is occurring is because:

  • #3070 set PYTHONNOUSERSITE=True to prevent local site-packages/ installations from interfering with SCT.
  • Unlike SCT, however, in some cases gradunwarp may rely on the local site-packages/ directory. So, as a side effect, gradunwarp can no longer see its locally-installed dependencies, because the change to PYTHONNOUSERSITE is inherited by the call to gradunwarp.
  • That's also why it works via the terminal/v5.0.1 -- because PYTHONNOUSERSITE is left unchanged in those cases.

This is a weird problem to fix, because it's due to a clash between 2 Python-based packages that install themselves in non-standard ways:

  1. spinalcordtoolbox brings its own conda + Python distributions.
  2. gradunwarp uses distutils rather than setuptools.

Since neither are pip installable, neither are used with virtualenvs, which makes isolating dependencies harder (and reliant on hacky workarounds).

joshuacwnewton picture joshuacwnewton on 29 Jan 2021
❤1 👍1

Actually, to verify that this is indeed the problem, could you please temporarily edit the shell script as follows?

# Gradient distortion correction
- gradient_unwarp.py ${file_t1}.nii.gz ${file_t1}_gradcorr.nii.gz siemens -g ${PATH_GRADCORR_FILE}/coeff.grad -n
+ PYTHONNOUSERSITE=False gradient_unwarp.py ${file_t1}.nii.gz ${file_t1}_gradcorr.nii.gz siemens -g ${PATH_GRADCORR_FILE}/coeff.grad -n
file_t1="${file_t1}_gradcorr"

If this change fixes the problem, then we can go ahead and think about a more long-term solution.

joshuacwnewton picture joshuacwnewton on 29 Jan 2021

@joshuacwnewton I edited the shell script as you described, but the problem is still there :cry: :

~~~

  • PYTHONNOUSERSITE=False
  • gradient_unwarp.py sub-1000032_T1w_raw_RPI_r.nii.gz sub-1000032_T1w_raw_RPI_r_gradcorr.nii.gz siemens -g /mnt/y/sebeda/coeff.grad/ICM/coeff.grad -n
    Traceback (most recent call last):
    File "/home/sabeda/.local/bin/gradient_unwarp.py", line 11, in
    from gradunwarp.core import (globals, coeffs, utils)
    ModuleNotFoundError: No module named 'gradunwarp'
    ~~~
sandrinebedard picture sandrinebedard on 29 Jan 2021

wow @joshuacwnewton thank you so much for finding this bug! you would be an amazing detective

jcohenadad picture jcohenadad on 29 Jan 2021

@joshuacwnewton I edited the shell script as you described, but the problem is still there :cry:

+ PYTHONNOUSERSITE=False
+ gradient_unwarp.py sub-1000032_T1w_raw_RPI_r.nii.gz sub-1000032_T1w_raw_RPI_r_gradcorr.nii.gz siemens -g /mnt/y/sebeda/coeff.grad/ICM/coeff.grad -n
Traceback (most recent call last):
  File "/home/sabeda/.local/bin/gradient_unwarp.py", line 11, in <module>
    from gradunwarp.core import (globals, coeffs, utils)
ModuleNotFoundError: No module named 'gradunwarp'

Hmmm... just to clarify, is PYTHONNOUSERSITE=False on a separate line? It should be on the same line, like this:

PYTHONNOUSERSITE=False gradient_unwarp.py ${file_t1}.nii.gz ${file_t1}_gradcorr.nii.gz siemens -g ${PATH_GRADCORR_FILE}/coeff.grad -n

If they are on separate lines, the setting won't be available to child processes.

(The reason I suggest this syntax instead of export is to temporarily set the environment variable for just that one command.)

wow @joshuacwnewton thank you so much for finding this bug! you would be an amazing detective

Thank you! I might still be incorrect, though... I can't be certain until it's verified, hehe. :sweat_smile:

joshuacwnewton picture joshuacwnewton on 29 Jan 2021

Hmmm... just to clarify, is PYTHONNOUSERSITE=False on a separate line?

No, it is on the same line in my shell script.

sandrinebedard picture sandrinebedard on 29 Jan 2021
👍1

My apologies, I gave the wrong suggestion. Rather than set to False, it just needs to be set to nothing. (PYTHONNOUSERSITE=)

PYTHONNOUSERSITE= gradient_unwarp.py ${file_t1}.nii.gz ${file_t1}_gradcorr.nii.gz siemens -g ${PATH_GRADCORR_FILE}/coeff.grad -n


Click to see my testing of this

I'm using a dummy file here just to get to the point where imports are attempted, so I don't have to download the exact files.

joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ gradient_unwarp.py Firefox_wallpaper.png Firefox_wallpaper.png siemens -g Firefox_wallpaper.png -n
Traceback (most recent call last):
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 123, in <module>
    args = argument_parse_gradunwarp()
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 62, in argument_parse_gradunwarp
    raise IOError(args.infile + ' not found')
OSError: Firefox_wallpaper.png not found
joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ PYTHONNOUSERSITE=True gradient_unwarp.py Firefox_wallpaper.png Firefox_wallpaper.png siemens -g Firefox_wallpaper.png -n
Traceback (most recent call last):
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 11, in <module>
    from gradunwarp.core import (globals, coeffs, utils)
ModuleNotFoundError: No module named 'gradunwarp'
joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ PYTHONNOUSERSITE=False gradient_unwarp.py Firefox_wallpaper.png Firefox_wallpaper.png siemens -g Firefox_wallpaper.png -n
Traceback (most recent call last):
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 11, in <module>
    from gradunwarp.core import (globals, coeffs, utils)
ModuleNotFoundError: No module named 'gradunwarp'
joshua@XPS-15-9560:~/Downloads/gradunwarp-1.2.0$ PYTHONNOUSERSITE= gradient_unwarp.py Firefox_wallpaper.png Firefox_wallpaper.png siemens -g Firefox_wallpaper.png -n
Traceback (most recent call last):
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 123, in <module>
    args = argument_parse_gradunwarp()
  File "/home/joshua/.local/bin/gradient_unwarp.py", line 62, in argument_parse_gradunwarp
    raise IOError(args.infile + ' not found')
OSError: Firefox_wallpaper.png not found

(I think "False" gets interpreted as a non-empty string, which == True for certain comparisons.)

joshuacwnewton picture joshuacwnewton on 29 Jan 2021

it just needs to set to nothing. (PYTHONNOUSERSITE=)

It works!!! 🎉

sandrinebedard picture sandrinebedard on 29 Jan 2021
🎉2

@sandrinebedard Excellent!

Would you mind if I closed this, then opened an issue in ukbiobank-spinalcord-csa to talk about the gradunwarp installation?

While it might be possible to fix this on the SCT side of things, I think it might be more useful to address this on the gradunwarp side of things, considering there are multiple concerns with how that package is installed.

joshuacwnewton picture joshuacwnewton on 29 Jan 2021

Would you mind if I closed this, then opened an issue in ukbiobank-spinalcord-csa to talk about the gradunwarp installation?

Sounds good!

sandrinebedard picture sandrinebedard on 29 Jan 2021

Closing now. :slightly_smiling_face:

Just to be clear, though: I think the PYTHONNOUSERSITE= fix should be viewed as a temporary solution. A more long-term solution would be to make sure gradunwarp's dependencies are installed somewhere different than the site-packages/ directory.

I've written a more in-depth explanation about this in points 4. and 5. of https://github.com/sct-pipeline/ukbiobank-spinalcord-csa/issues/31.

joshuacwnewton picture joshuacwnewton on 31 Jan 2021
👍1

Reopening because @kousu has raised some good points in https://github.com/sct-pipeline/ukbiobank-spinalcord-csa/issues/31#issuecomment-771647698.

On our side, I think setting PYTHONUSERSITE=True and other global variables like it is not a good solution. Most apps, reasonably, assume that they are running on a standard-ish system. Breaking those assumptions makes SCT an incompatible thing like Matlab that people have to containerize to keep running. If we're going to force people to containerize we should at least provide the docker image ourselves.

For another hacky solution: We could try to isolate PYTHONUSERSITE to just SCT itself, maybe? Like, make sure every: subprocess.run([...], env={k: os.environ[k] for k in set(os.environ.keys()) - {'PYTHONUSERSITE'}}). That would be a hacky workaround but at least it wouldn't leak out beyond SCT anymore.

joshuacwnewton picture joshuacwnewton on 2 Feb 2021

On our side, I think setting PYTHONUSERSITE=True and other global variables like it is not a good solution

We could try to isolate PYTHONUSERSITE to just SCT itself, maybe?

For what it's worth, I think the change we make to PYTHONNOUSERSITE _is_ generally well isolated: It's only set in the context of SCT scripts, and I believe as soon as an SCT script exits, the environment variables return to normal.

https://github.com/neuropoly/spinalcordtoolbox/blob/255140ade8632524122e09ea53908bb56cf10ca3/spinalcordtoolbox/compat/launcher.py#L27-L28

https://github.com/neuropoly/spinalcordtoolbox/blob/255140ade8632524122e09ea53908bb56cf10ca3/spinalcordtoolbox/compat/launcher.py#L44

The leakage only occurs with sct_run_batch, because it's called via its CLI launcher (which sets the environment variable). Then, sct_run_batch calls a shell script, which then calls a Python-based program, which _then_ requires access specifically the user site-packages/ (which we had disabled).

I think we still want to isolate SCT itself from user site packages, since this is patching over a conda issue specifically. (See https://github.com/neuropoly/spinalcordtoolbox/issues/3067 and https://github.com/conda-forge/python-feedstock/issues/171),

But, we also want to allow site packages specifically for commands in sct_run_batch. Hm...

joshuacwnewton picture joshuacwnewton on 2 Feb 2021

We could try to isolate PYTHONUSERSITE to just SCT itself, maybe?

For what it's worth, I think the change we make to PYTHONNOUSERSITE _is_ generally well isolated: It's only set in the context of SCT scripts, and I believe as soon as an SCT script exits, the environment variables return to normal.

Yes! Environment vars are well designed in that respect!

I think we still want to isolate SCT itself from user site packages, since this is patching over a conda issue specifically. (See https://github.com/neuropoly/spinalcordtoolbox/issues/3067 and https://github.com/conda-forge/python-feedstock/issues/171),

😩

kousu picture kousu on 3 Feb 2021

What a gordian knot.

kousu picture kousu on 3 Feb 2021
👀1 👍1

I was overthinking this problem. We should limit PYTHONNOUSERSITE to SCT alone, as @kousu recommends.

joshuacwnewton picture joshuacwnewton on 24 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mrhardisty picture mrhardisty  ·  61Comments
jcohenadad picture jcohenadad  ·  30Comments
gmotzespina picture gmotzespina  ·  34Comments
Drulex picture Drulex  ·  25Comments
stephaniealley picture stephaniealley  ·  19Comments