Isis3: Difficulties installing other packages into the `isis3` conda environment

[michaelaye]

The current strict requirements table leads to difficulties for conda to resolve the dependency tree when trying to install anything on top of the isis3 environment.

For example (h/t to @rbeyer ):

conda activate isis3
conda install gdal

the conda requirement resolution leads to the suggestion to remove the isis3 package.

This is most likely caused by the strict requirements instruction of the isis3 package:

dependencies:
   armadillo 8.200.0
   blas ==1.1 openblas
   bullet ==2.86.1 0
   bz2file 0.98
   bzip2 ==1.0.6 1
   cmake ==3.9.1 0
   cspice ==66 h470a237_3
   curl ==7.60.0 0
   doxygen ==1.8.14 0
   eigen ==3.3.3 0
   embree ==2.14.0 0
   geos 3.5.1
   geotiff ==1.4.2 1
   gmm 5.0
   gmp ==6.1.2 0
   gsl ==2.2.1 blas_openblas_3
   hdf5 ==1.8.18 2
   icu ==58.2 0
   jama 125
   jpeg ==9b 2
   kakadu 1
   krb5 ==1.14.2 0
   libpng >=1.6.34
   libprotobuf 3.5.2
   libtiff ==4.0.9 0
   libxml2 ==2.9.7 0
   make
   mesalib ==17.2.0 0
   mysql 5.7.20
   mysql-connector-c ==6.1.6 0
   nanoflann 1.2.2
   ninja ==1.7.2 0
   nn
   openblas ==0.2.19 2
   opencv
   openssl ==1.0.2n 0
   patchelf 0.9
   pcl 1.8.1
   pip 9.0.1
   protobuf 3.5.2
   python 3.6
   qhull ==7.2.0 0
   qt 5.9.6.*
   qwt 6.1.3.*
   setuptools 38.5.1.*
   sip 4.18
   sqlite ==3.13.0 1
   suitesparse ==4.5.4 blas_openblas_200
   superlu ==5.2.1 blas_openblas_201
   tnt ==126 0
   wheel 0.30.0
   x264 20131218
   xalan-c 1.11
   xerces-c ==3.1.4 0
   xorg-kbproto ==1.0.7 1
   xorg-libice
   xorg-libsm
   xorg-libx11 ==1.6.4 6
   xorg-libxi
   zlib ==1.2.11 0

How could this be addressed?

[jessemapel]

Relaxing the Anaconda dependencies may resolve this problem. That is both a big challenge and may not actually fix the problem because of difference between how gdal and isis3 are compiled.

Anaconda doesn't do a good job differentiating between different builds and compilation environments. For example, many of the ISIS3 dependencies are pinned to specific builds and labels on conda-forge because they have to be compiled with gcc 5+ for binary compatibility (this is why CentOS7 has its own environment). Substituting these for different builds that used gcc 4 would cause linking errors.

We are hoping to go to more standardized builds and getting everything onto conda-forge, but there is no guarantee that would be enough to prevent the clash with gdal.

[jessemapel]

For the moment I see two solutions:

The PATH editing solution

1. Install ISIS3 into one Anaconda environment
2. deactivate the environment and then run the old isis3Startup.sh script
3. activate your gdal environment

This basically adds the ISIS3 binaries to your path and then adds the gdal environment on top of it. You could do some creative activate.d and deactivate.d editing to automate this whenever you activate the gdal environment.

The manual build solution

1. Install gdal in an Anaconda environment. Save what $CONDA_PREFIX is for this environment.
2. Clone the ISIS3 repo.
3. Setup the ISIS3 dependencies in a new environment.
4. Build ISIS3.
5. Install ISIS3 into your gdal environment. Set the -DCMAKE_INSTALL_PREFIX part of your cmake configure command to what $CONDA_PREFIX was for your gdal environment, do not use the $CONDA_PREFIX for your ISIS3 dependencies environment.
6. Run the isis3VarInit.py script to setup your environment variables for the environment.

This will manually install the ISIS3 binaries, preference file, xmls, etc. into your gdal environment. This way, when you activate your gdal environment, the ISIS3 binaries will already be in your PATH and ready to go.

[michaelaye]

I really don't find the PATH manipulation solution too bad. But maybe we should provide some well working scripted solutions for likely scenarios. And, I'm still trying to figure out how to make things working on the fish shell, b/c the activation scripts are not being read by fish. Currently creating an conda GH issue about it, b/c they support fish in general.

[thareUSGS]

This is problem for me also, but perhaps an opportunity to enhance both GDAL and ISIS3. So during an ISIS3 build, is a conda env used? If so, can't we first create an env with existing versions of GDAL (and Python, AstroPy, PVL, SpiceyPy, Pysis, etc.) which will set an existing (and available) environment for those binaries. Now build ISIS3 based on those environments...? This feels like being as locked down as a statically built binary but at least it will allow for some related binaries to be available within an ISIS3 environment.

This also would have the benefit our GDAL (HiRISE) users to be able to run GDAL tools against the ISIS3-delivered Kakadu library (for Jpeg2000 support).

conda info isis3
does show that many items are seemingly locked in. If I were to guess, the conflicts between isis3 and gdal might be one or all of "geotiff ==1.4.2 1", "libtiff ==4.0.9 0", "libpng >=1.6.34", "hdf5", "geos 3.5.1", or "xerces-c". All of these we should be able to get ISIS3 to use a currently available version (compatible with other binaries). BTW, looking into why ISIS3 needs geoTiff, I am a little horrified we wrote our own GeoTiff parser instead of simply using GDAL...

Anyway, for a current workaround, I have a conda env for isis3 and different env for gdal (which has all the Python stuff like PVL, mathplotlib, scipy, ect.). I have tested a single script (bash) which, runs some isis3 stuff, activates gdal, runs gdal stuff, and back to isis3

(isis3) $ cat do.sh
#!/bin/bash

#set ISIS3 conda env
conda activate isis3
which cubeatt
cubeatt from=P22_009816_1745_XI_05S073W.lev2.cub to=out.cub
#run GDAL
conda activate gdal
gdal_translate --version
which gdal_translate
gdal_translate out.cub out.tif
#back to ISIS3
conda activate isis3
which cubeatt

(isis3) $ source do.sh
/work/users/thare/python/anaconda/envs/isis3/bin/cubeatt
GDAL 2.3.2, released 2018/09/21
/work/users/thare/python/anaconda/envs/gdal/bin/gdal_translate
Input file size is 6459, 10459
0...10...20...30...40...50...60...70...80...90...100 - done.
/work/users/thare/python/anaconda/envs/isis3/bin/cubeatt
(isis3) $

[jessemapel]

@thareUSGS Your proposed build a big environment is roughly equivalent to just installing ISIS3 into the environment.

[jessemapel]

@thareUSGS I also think most of those dependencies could be changed, possible excluding geos. ISIS is still using a relatively custom geos build.

[thareUSGS]

@thareUSGS Your proposed build a big environment is roughly equivalent to just installing ISIS3 into the environment.

@jessemapel Yes - but we would preemptively do this as "nicety" for the users, since it is something we can control from here.

[kberryUSGS]

Adding to this because I experienced a similar problem, and didn't see the package I was unable to add to the isis3 environment already listed on this ticket: I needed to set up a separate conda environment to use matplotlib, so I have one conda env for isis3 and then another for "default python" including matplotlib.

Looking at this issue again, this is actually almost exactly much the same thing that Trent said above about having separate python and isis3 environments, so never mind!

[michaelaye]

Interestingly, matplotlib installs while "only" downgrading isis3 from 3.6.0-py36_3 to 3.6.0-py36_0. But, due to a recent conda "feature" from the devil called "features" it mixes packages from the defaults channel with conda-forge channels. This can lead to unpredictable errors and I strongly recommend when working with the conda-forge channel to pin numpy and gdal to conda-forge so that no default channel packages are being mixed in that are available on conda-forge, like so:

conda config --env --add pinned_packages conda-forge::numpy
conda config --env --add pinned_packages conda-forge::gdal

After doing that, this is the suggestion that conda offers for matplotlib into isis3:

$ conda install matplotlib                                                                                          (isis3)
Solving environment: done

## Package Plan ##

  environment location: /Users/klay6683/miniconda3/envs/isis3

  added / updated specs:
    - matplotlib


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    isis3-3.6.0                |           py36_0       220.1 MB  usgs-astrogeology
    libuuid-1.0.3              |                1          12 KB  conda-forge
    graphite2-1.3.12           |       h7d4d677_1          86 KB  conda-forge
    harfbuzz-1.9.0             |       h08d66d9_0         1.0 MB  conda-forge
    opencv-3.4.1               |   py36h40b0b35_1           7 KB
    fontconfig-2.13.0          |       h95c7cdb_0         245 KB
    matplotlib-2.2.3           |   py36h0e0179f_0         6.7 MB  conda-forge
    ------------------------------------------------------------
                                           Total:       228.1 MB

The following NEW packages will be INSTALLED:

    cycler:          0.10.0-py_1                  conda-forge
    graphite2:       1.3.12-h7d4d677_1            conda-forge
    kiwisolver:      1.0.1-py36h2d50403_2         conda-forge
    libuuid:         1.0.3-1                      conda-forge
    matplotlib:      2.2.3-py36h0e0179f_0         conda-forge
    pyparsing:       2.3.0-py_0                   conda-forge
    python-dateutil: 2.7.5-py_0                   conda-forge
    pytz:            2018.7-py_0                  conda-forge
    tornado:         5.1.1-py36h470a237_0         conda-forge

The following packages will be UPDATED:

    cairo:           1.14.6-4                     conda-forge       --> 1.14.12-he6fea26_5   conda-forge
    fontconfig:      2.12.1-4                     conda-forge       --> 2.13.0-h95c7cdb_0
    freetype:        2.7-1                        conda-forge       --> 2.9.1-h6debe1e_4     conda-forge
    glib:            2.51.4-0                     conda-forge       --> 2.55.0-h464dc38_2    conda-forge
    harfbuzz:        1.3.4-2                      conda-forge       --> 1.9.0-h08d66d9_0     conda-forge
    opencv:          3.3.0-py36_blas_openblas_200 conda-forge       [blas_openblas] --> 3.4.1-py36h40b0b35_1

The following packages will be DOWNGRADED:

    isis3:           3.6.0-py36_3                 usgs-astrogeology --> 3.6.0-py36_0         usgs-astrogeology

[michaelaye]

Hah, and I just realized that the same should be done for opencv, so adding

conda config --env --add pinned_packages conda-forge::openv

the changes are actually minor and isis3 is NOT being downgraded at all!

$ conda install matplotlib                                                                                          (isis3)
Solving environment: done

## Package Plan ##

  environment location: /Users/klay6683/miniconda3/envs/isis3

  added / updated specs:
    - matplotlib


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    matplotlib-2.1.0           |           py36_0         6.6 MB  conda-forge

The following NEW packages will be INSTALLED:

    backports:                     1.0-py_2             conda-forge
    backports.functools_lru_cache: 1.5-py_1             conda-forge
    cycler:                        0.10.0-py_1          conda-forge
    matplotlib:                    2.1.0-py36_0         conda-forge
    pyparsing:                     2.3.0-py_0           conda-forge
    python-dateutil:               2.7.5-py_0           conda-forge
    pytz:                          2018.7-py_0          conda-forge
    tornado:                       5.1.1-py36h470a237_0 conda-forge

Proceed ([y]/n)?

EDIT: Sorry for the accidental closing. @kberryUSGS , this should be a safe way to add matplotlib to the isis3 environment.

[michaelaye]

Motivated by this success I tried adding my standard science stack, and, alas apart from GDAL and geopandas, all these work!

conda install astropy scikit-image scikit-learn seaborn spiceypy numba numpy xarray shapely bokeh cartopy dask ipython ipympl ipyparallel isort jupyter_contrib_nbextensions pandas sphinx nbsphinx scipy pytest pytables nb_conda_kernels

with the only change of existing package:

The following packages will be UPDATED:

    tbb:                               4.4-hf7780a4_0                usgs-astrogeology --> 2019.2-h2d50403_0 conda-forge

Would be interesting to know if that would break isis3? Otherwise, this is rather complete, as I said, apart from geopandas and GDAL.

[jessemapel]

@michaelaye I don't remember why we're serving that library at the moment. Maybe I'll test updating that to conda-forge's build tomorrow.

[jlaura]

@jessemapel @michaelaye Is this a good opportunity to get an issue open, labelled as 'good for beginner' (maybe?), and then solicit to have some community members do some experimentation with this. I agree 100% that being able to get the scientific Python stack and GDAL into the ISIS3 env would be very powerful.

We know we want to be off as many snowflake builds as possible. I do not see us getting back onto the track of addressing this until, optimistically, sometime in the new year. Not a cop out, but not cycling back to our maintenance project for a while.

[jessemapel]

@jlaura That sounds good to me. I was going to mark this as will not fix because getting GDAL working with ISIS dependencies could be a serious problem.

[jessemapel]

I'm going to close this issue. We're doing some work on this under #648 but it probably make ISIS work with everything.

[rbeyer]

I'm reactivating this thread, not because I have a solution to the underlying problem, but because following some discussions today, I have written up some instructions for how to carry out the steps to have ISIS in one environment, but referenced from another.

Where should I PR these instructions, on the main README in the Installation section? Somewhere else?

[jlaura]

I think into the README is reasonable for now. Maybe as an advanced option in the Installation section?

We are hoping to have some funding next FY to work on our documentation in general, so it is likely to move around then no matter where it goes now.

[michaelaye]

I guess in the long run we might want isis3VarInit.py to be more intelligent and allow writing the activate/deactive scripts for other conda envs? Sounds like a nice PR we could do?