Isis3: Partial Download of Base Data does not provide dependencies for program std2isis

[gdmcdonald1]

ISIS version(s) affected: 4.1.1 AND 3.10.0

Description
Upon installing this week, and doing a partial download of the Base Data as suggested, when attempting to run std2isis.

I/O ERROR Unable to open [/home/USERNAME/anaconda3/envs/isis/data/base/templates/labels/CubeFormatTemplate.pft].

How to reproduce
Perform a clean install of Isis
conda activate isis
cd $ISIS3DATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/base .
std2isis from=test.tif to=test.cub

Possible Solution
It seems the base/template files are now hosted in the binary, but was this change not implemented for std2isis? I've verified I am able to run certain other programs such as maptemplate and qview.

Thank you for your help!

[rbeyer]

You're right, there do seem to be some changes:

> rsync isisdist.astrogeology.usgs.gov::isisdata/data/base/

[...]

drwxr-xr-x          4,096 2020/03/26 15:10:27 .
drwxr-sr-x          4,096 2019/02/28 14:15:07 dems
drwxr-xr-x          4,096 2020/07/07 03:02:08 kernelTesting
drwxr-sr-x          4,096 2014/04/01 14:59:45 kernels
drwxr-sr-x          4,096 2019/08/14 09:33:17 testData


> rsync isisdist.astrogeology.usgs.gov::isis3data/data/base/

[...]

drwxr-xr-x          4,096 2019/06/25 16:44:27 .
-rw-r--r--            181 2012/02/14 13:56:45 about.xml
drwxr-sr-x          4,096 2019/11/05 11:30:18 applications
drwxr-sr-x          4,096 2019/02/28 14:15:07 dems
drwxr-sr-x          4,096 2017/08/11 12:53:26 icons
drwxr-sr-x          4,096 2017/08/15 12:36:30 images
drwxr-sr-x          4,096 2014/04/01 14:59:45 kernels
drwxr-sr-x          4,096 2016/03/16 11:59:27 templates
drwxr-sr-x          4,096 2019/08/14 09:33:17 testData
drwxr-xr-x          4,096 2020/03/25 12:16:41 testing
drwxr-sr-x          4,096 2019/08/14 11:15:57 translations

My guess is that when the ASC made the new 'isisdata' area, they didn't get everything copied over, but maybe there is something else afoot. In the mean time, you can rsync the templates directory (and anything else that is complained about) from the isis3data location, I suppose.

[gdmcdonald1]

Thank you this is a big help! I saw no reference for an "isis3data" area in the installation guide so hopefully this will do for the time being.

[jessemapel]

isisdist.astrogeology.usgs.gov::isisdata/data is the data area for ISIS 4

isisdist.astrogeology.usgs.gov::isis3data/data is the data area for ISIS 3

hence the different names.

As of 4.1, we distribute much of what was in the base data area with the software itself. Did you see the same error with both ISIS 3 and ISIS 4?

[rbeyer]

Oh, right! So with the ISIS data area specified from the README (the just isisdata path), ISIS 4.1 and higher should work. I rsynced a fresh version of the new ISIS data, and can confirm that with ISIS 4.1+, your std2isis example runs just fine. So if you had the new ISIS data area version, were running ISIS 4.1+, and were getting errors, then that really is a problem, and we need more information to track this down. Can you confirm that?

However, if you had an older version of ISIS, and were using a single, shared location for your ISIS data area, the --delete in the rsync command could have bitten you. For example, I have been known to keep several versions of ISIS on my machine at the same time, but the data area is large and I don't want to duplicate that. So I mostly just have a single data area, have rsynced to it over the years, and have updated ISIS, and then just made sure my old $ISIS3DATA environment variable (now $ISISDATA) for each new version of ISIS points to that same spot.

If you had an ISIS 3 install, and had rsynced the isis3data version data to some location, it should have worked for all of the ISIS versions (even the new one). But if you saw the new instructions, and rsynced the new plain isisdata version to the same place where you had been putting the isis3data that --delete in the rsync command would have cleared out the things that are now not present in the new version (because the new version moves them elsewhere). This means that an ISIS 4.1+ pointed at that $ISISDATA directory should have worked, but ISIS 4.0.x or any ISIS 3 versions would have had the failure because the rsync --delete removed some files that they needed.

[gdmcdonald1]

Hi Ross, thank you for the detailed follow up!

I set up different conda environments for my ISIS3 vs ISIS4, so was not sharing data areas, but I think you're right in that I may have not had the needed data in the ISIS4 data area. Part of that may lie with, in ISIS 4: despite running the isis3VarInit.py script after installation the $ISISDATA variable takes me to /home/USER rather than /home/USER/anaconda3/envs/isis4/data. In ISIS3, $ISIS3DATA takes me appropriately to /home/USER/anaconda3/envs/isis/data.

B/c so far all the issues I've been having are with ISIS 4, for now I'm rsync'ing the isisdist.astrogeology.usgs.gov::isis3data/data/base that was suggested and trying to go forward just using ISIS3, but would you suggest otherwise? I can't tell from the Github info and https://isis.astrogeology.usgs.gov/ whether ISIS 4 is really now the preferred version for current general users. Thanks again!

[rbeyer]

Hmm.

ISIS 4.1.1 is the designated public release, so it should work reliably. Sorry you've run into issues, but thanks for being willing to engage. I appreciate that you probably just want to get on with using ISIS for your work, so if you have something working and just want to move on, that's totally fine.

However, from what you mention, it seems like the problem is maybe more with the environment variables than the data files themselves. If you already have separate ISIS 3 and ISIS 4 data areas and ISIS 3 and ISIS 4 conda environments, this actually would be pretty easy for you to test. To do so, make sure your "ISIS 3" data area is synced with the isis3data rsync path, and then also rsync your "ISIS 4" data area with the plain isisdata path.

This should "reset" your state to where you were when you reported the problem. This should mean that your ISIS 3 environment should work, but your ISIS 4 environment shouldn't, right?

Okay, now you mentioned that in your ISIS 4 environment, your $ISISDATA environment variable doesn't point to the right place, that's probably the key. In that environment, if you visit $CONDA_PREFIX/etc/conda/activate.d you'll find a bunch of scripts named env_vars.*, take a look at the one that is specific to the shell you use. That is the file that sets the environment variables every time you conda activate. Are these paths correct? If not, you could try re-running the isis3VarInit.py script again, and seeing if things are different. Maybe there is something goofy about the way that script is run. If you still aren't getting something that is right, you can always go in an change the appropriate env_vars.* file so that it correctly points to your ISIS 4 data area, or just use your typical shell commands to explicitly set the environment variable this time for testing. This should make things work. If so, we've isolated the problem to environment variables and not data files.

[gdmcdonald1]

Thanks, I'm willing to try and work things through especially with all your help---would rather be using the latest version unless there's a specific reason not to!

I checked env_vars.sh in my isis4 environment and see the following: #!/usr/bin/env sh
export ISISROOT=/home/USER/anaconda3/envs/isis4
export ISIS3DATA=/home/USER/anaconda3/envs/isis4/data
export ISIS3TESTDATA=/home/USER/anaconda3/envs/isis4/testData
export ALESPICEROOT=/home/USER/anaconda3/envs/isis4/aleData

Interesting that it lists ISIS3DATA instead of ISISDATA, I've verified that I am able to navigate there with cd $ISIS3DATA$. I also double checked the version # in /home/USER/anaconda3/envs/isis4 and found:

4.0.1 # Public version number
2020-03-04 # Release date
stable # Release stage (alpha, beta, stable)

so confirmed I'm at least not mixing versions in my isis4 environment as for why I'm seeing ISIS3DATA in that script. Thanks again, really appreciate your help!

[jessemapel]

Thanks, I'm willing to try and work things through especially with all your help---would rather be using the latest version unless there's a specific reason not to!

I checked env_vars.sh in my isis4 environment and see the following: #!/usr/bin/env sh
export ISISROOT=/home/_USER_/anaconda3/envs/isis4
export ISIS3DATA=/home/_USER_/anaconda3/envs/isis4/data
export ISIS3TESTDATA=/home/_USER_/anaconda3/envs/isis4/testData
export ALESPICEROOT=/home/_USER_/anaconda3/envs/isis4/aleData

Interesting that it lists ISIS3DATA instead of ISISDATA, I've verified that I am able to navigate there with cd $ISIS3DATA$. I also double checked the version # in /home/_USER_/anaconda3/envs/isis4 and found:

4.0.1 # Public version number
2020-03-04 # Release date
stable # Release stage (alpha, beta, stable)

so confirmed I'm at least not mixing versions in my isis4 environment as for why I'm seeing ISIS3DATA in that script. Thanks again, really appreciate your help!

This looks like you have ISIS 4.0 installed and not 4.1. We changed the environment variables and the data area in ISIS 4.1, I would expect your env_vars.sh script to be setting ISISDATA and ISISTESTDATA for version 4.1.

Can you run conda list isis in your ISIS 4 environment to see which version you have installed?

[rbeyer]

Yeah, I think that's the problem, too. So there are a couple of things going on. First is that the "ISIS3" string of characters was endemic to the codebase for more than a decade, and ticking the version up to ISIS 4.x still had some of these 'names' around. The second is that the format of the "ISIS data" area changed, but also not on a clean major version boundary. Maybe easiest to summarize like this:

ISIS version | ISIS Data Environment variable name | ISIS Data layout
---|---|---
3.x | $ISIS3DATA | isis3data
4.0.x | $ISIS3DATA | isis3data
4.1.0 | $ISISDATA | isisdata

See the problem? The environment variable name and the data area layout didn't change until the 4.1.0 release.

It looks like your "ISIS 4" conda environment is running 4.0.1, so that means that it will use the $ISIS3DATA env variable (which it seems to be doing), but it needs the "old" isis3data Data area layout. Pointing it at a "new" isisdata layout is what was causing the problems. So you can either point your 4.0.1 environment to the "old" isis3data area of your "ISIS 3" environment via:

export ISIS3DATA=/home/USER/anaconda3/envs/isis3/data

assuming you named that conda environment "isis3".

But really, you might as well get to the most recent release and just upgrade your ISIS 4 environment to ISIS 4.1.1 via:

conda update isis

in your ISIS 4 environment, rerun the init script (to get the right environment variables into your activate and deactivate scripts), deactivate and reactivate your isis4 environment, and now it should be pointing at the new plain isisdata layout, and should work.

[gdmcdonald1]

Thanks again for all of your help Ross and Jesse. Here are the latest snippets that seem to have still left me with 4.0.1. I wonder if it's possible if one of the channels is still pointing to this as the latest version?

conda update isis
...
Downloading and Extracting Packages
libffi-3.2.1         | 47 KB     | ##################################### | 100% 
tk-8.6.10            | 3.2 MB    | ##################################### | 100% 
_openmp_mutex-4.5    | 435 KB    | ##################################### | 100% 
gettext-0.19.8.1     | 3.6 MB    | ##################################### | 100% 
xz-5.2.5             | 343 KB    | ##################################### | 100% 
zlib-1.2.11          | 105 KB    | ##################################### | 100% 
libgcc-ng-9.2.0      | 8.2 MB    | ##################################### | 100% 
_libgcc_mutex-0.1    | 3 KB      | ##################################### | 100% 
wheel-0.34.2         | 24 KB     | ##################################### | 100% 
pip-20.1.1           | 1.1 MB    | ##################################### | 100% 
setuptools-49.1.0    | 938 KB    | ##################################### | 100% 
libgomp-9.2.0        | 816 KB    | ##################################### | 100% 
libstdcxx-ng-9.2.0   | 4.5 MB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

conda list isis
...
packages in environment at /home/george/anaconda3/envs/isis4:
Name                    Version                   Build  Channel
isis                      4.0.1                         0    usgs-astrogeology
````

````
conda config --show channels
...
channels:
usgs-astrogeology
conda-forge
defaults

[jessemapel]

@gdmcdonald1 Interesting. Can you try two other things:

conda search -c usgs-astrogeology isis

This should list all of the versions you can see on our channel

conda install -c usgs-astrogeology isis=4.1

This should try and force an update to 4.1

[gdmcdonald1]

Thank you again both so much for your help, this worked:

conda install -c usgs-astrogeology isis=4.1

Apologies for the delayed response, I wanted to verify several different functions worked before declaring victory.

My suggestions going forward (so hopefully you don't get similar requests for help) is to document on the installation page the differences in the ISIS Data Environment variable names and rsync areas (as per Ross's table) for 3 vs 4.0 vs 4.1, and also perhaps most importantly, make a note in the installation guide for users to verify they have 4.1 installed after installing, and including the above command as a fix if 4.0 gets installed. In my case,

conda install -c usgs-astrogeology isis

lead to having 4.0 installed. It's possible this was somewhat specific to my conda and linux distribution, but especially because I'm using Ubuntu 18.04 LTS I would imagine it's very possible this situation of 4.0 getting installed by default could happen to someone else and the above info would be useful. Thanks again, you've both been a huge help.

[jlaura]

@gdmcdonald1 @jessemapel Is this still an open issue? Looks like this is resolved with the data area info and that it might be a conda issue on the installation side?

[jessemapel]

@jlaura I would like to see Ross's table moved into the install instructions before this is closed.