Isis3: Vimscal leads to poor calibration of both Jupiter and Saturn spectra.

[ashwinbraude1]

Hi,

I'd like to re-open an old thread which I am no longer able to edit (refer to https://isis.astrogeology.usgs.gov/fixit/issues/5331 for more detail on the background and for graphical plots) as I am currently revisiting Saturn VIMS data and the original issue still hasn't been resolved in the latest release of ISIS3. I'd also like to keep this thread public as I am in contact with several other people who are also curious about the VIMS calibration.

As a reminder, I've been having trouble using Vimscal to calibrate both Jupiter and Saturn data, which results in very unphysical spectra. This problem did not exist in older releases of ISIS3, which means that Vimscal has been tampered with in some way that makes it no longer usable. I'd like to add several points to my original thread that may aid in diagnosing the problem. Much of the attention so far between RC19 and RC20 has been paid on trying to correct for small effects of spectral drift, but the effects of spectral drift on the calibration of the data are very small and do not seem to be the root cause of the poor calibration. We note that Vimscal performed using version 3.4 of ISIS appears to produce much more sensible spectra, and so the error must have cropped up somewhere between the release of versions 3.4 and 3.5. In addition, since the calibration files haven't changed that much between the two releases, the root cause of the problem appears to be in the software itself and not in the calibration cubes.

While I do not work with the IR part of the VIMS data myself, I have also been informed that later versions of Vimscal perform poor calibration of the IR spectra as well as the VIS spectra. Unlike the VIS spectra, the general shape of the IR spectra appears to be well-calibrated, but the I/F values are wildly overestimated.

Thanks,
Ashwin

[jessemapel]

@twilson271828 May have some comments on this as he was working on your ticket previously.

In your previous post you provided a plot from cubes calibrated using isis 3.4 and a cube calibrated with isis 3.5. Can you provide the 3.4 calibrated cube? Have you already given these to @twilson271828?

[ashwinbraude1]

Hi Jesse,

Thank you for your reply. The reference Jupiter spectrum calibrated using version 3.4 that I showed in the plots is available in the supplementary data of Carlson et al (2016, Icarus 274, 106-115), which you can access from this link without a paywall: https://www.sciencedirect.com/science/article/abs/pii/S0019103516001494#ec-research-data (the name of file is 'Cassini VIMS spectra of the GRS.txt', which is in the zip file). Unfortunately it only contains 3 spectra, I have just sent an email to the person who calibrated them in case he can supply you with additional spectra if you need. I'm afraid I don't have any spectra available for Saturn calibrated using version 3.4; the one in the original plots was from a different instrument and calibrated using different software.

Edit: Pat Fry has kindly provided me with the calibrated cubes (in I/F and specific energy) of the visible part of cube 1354610545, from which the aforementioned spectra were extracted, and which I have attached to this post. The IR part isn't available though, as the sampling mode was 'UNDER'. The pointing information is also inaccurate due to the lack of good NAIF kernels.

vims_cube_1354610545_version_3point4.zip

Ashwin

[patmfry]

Hi Ashwin, and to whoever might pick this up.
I have debugged the problem with the VIMS visible calibration. The problem is with the specEnergyFile. The wavelength-dependent functions in the cube ($cassini/calibration/vims/(calVervsion)/RC19-mults/____.cub) are two different functions depending on whether an IR or VIS cube is being processed. The IR part contains "sensitivity" that converts from DN to photons. There is also a multiplier in vimsCalibration0002.trn that must be applied. The visible part (bands 1-96) contains a wavelength-dependent multiplier that is to be applied to a visible sensitivity function ($cassini/calibration/vims/vis_perf_v????.cub). Inspection shows that the visible multiplier is near unity and has none of the spectral character contained in the previous vis_perf... file. So either the RC19-mults files were incorrectly produced or, more likely, the vimscal program run by the project had differing processing for the visible and ir bands. There is also a presumably erroneous multiplier that gets the near-unity visible sensitivity function close to the actual sensitivity function in the middle of the wavelength range. Without recreating the RC19-mults file is to change the software so that in calculateSpecificEnergy(), for visible bands (g_visBool), the specEnergyCube used is the product of the old perf file and the new multiplier file. The IR multiplier in the vimsCalibration.trn file should then be ignored or set to 1.

Ashwin, for existing cubes, the appropriate way to fix the specific energy or ioverf files is to divide by the (as far as I can tell arbitrary) scalar IR multiplier from the .trn file (39474.1268...) and multiply by the old vis_perf_v????.cub function (it is spatially independent).

I hope this is useful...
Regards,
Pat Fry

[ashwinbraude1]

Thank you Pat, I've just done this on a set of Jupiter spectra and I can confirm that they look much more sensible now! I'll leave this ticket open nonetheless until VIMSCAL is patched for the next edition of ISIS3.

[jlaura]

Vimscal leads to poor calibration of both Jupiter and Saturn spectra

[acpaquette]

@patmfry I am picking this ticket up for the current mission support sprint. Off the top of your head do you know what would be involved to update the mults file to appropriately fix this problem? Or is the product of the old perf file and the mult file, generating a new calibration file, a good way to handle this?

It would also be nice to have a cube that has the incorrect calibration values, along with a fixed cube with the changes you suggested above. I think I have corrected the values with the following fx call, if the cubes linked in the zip by @ashwinbraude1 have the incorrect values.
fx f1=/Users/acpaquette/Desktop/vims_cube_1354610545_version_3point4/V1354610545_2_vis_if_rc17.cub f2=$ISIS3DATA/cassini/calibration/vims/vis_perf_v0002.cub+1-96 to=/Users/acpaquette/Desktop/vims_cube_1354610545_version_3point4/V1354610545_2_vis_if_rc17_fx.cub equation="(f1/39474.12689222814) * f2"

I've just been making some assumptions about the data that is linked in this issue and could use some clarification. Thank you!

[patmfry]

acpaquette Not sure what best was to fix this is, new files or in software. The downside of creating new perf files is that there are a lot of them, My approach would likely be to mod vimscal calculateSpecificEnergy() and just have separate processing for VIS and IR, returning the product of two files for VIS cubes, As far as the mutiplier in the vimsCalibration...trn file, I guess one could make a higher version with VIS multiplier of 1 rather than 39474.
It looks like your fx call should do what it's supposed to. I'll try to attach bad and fixed cubes in a separate posting. They're inside FITS files, sorry.

[patmfry]

acpaquette Hopefully the zip archive contains
V1354610545_2_vis_if_RC19.fits and V1354610545_2_vis_if_RC19fix.fits
vimsv_goodbad.zip

[acpaquette]

@patmfry Second question, I can't run vimscal on the V1354610545_2_vis_if_RC19.fits file after ingesting it as a cube. It seems that the file is from 2000 and is no longer supported according to #2397. However I may be missing something.

[patmfry]

@acpaquette Not sure if there's some confusion here. Both the cubes I sent are already cailbrated by vimscal. Are you trying to calibrate one of those, or are you trying to calibrate v1354610545_2_vis.cub (output from vims2isis and spiceinit)? If the problem is that vimscal doesn't have multiplier and other files available for 2000, then I have a way around that. I've made symbolic links in the data/cassini/calibration/vims/RC19/RC19-mults to .../RC20//RC19-mults files, with appopriate names for vimscal to ingest them.
[patf@puck vims]$ ls -l RC19/RC19-mults/
total 85516
-rw-r--r-- 1 patf domain users 97 Aug 22 2017 AAA.readme.txt
lrwxrwxrwx 1 patf domain users 73 Feb 6 10:59 RC19.1999_v0001.cub -> ../../RC20/RC19-mults/VIMS1999.6000.calibration.multiplier.RC19_v0003.cub
lrwxrwxrwx 1 patf domain users 73 Feb 6 11:01 RC19.2000_v0001.cub -> ../../RC20/RC19-mults/VIMS2000.9610.calibration.multiplier.RC19_v0003.cub
lrwxrwxrwx 1 patf domain users 73 Feb 6 11:01 RC19.2001_v0001.cub -> ../../RC20/RC19-mults/VIMS2001.1200.calibration.multiplier.RC19_v0003.cub
-rw-r--r-- 1 patf domain users 5833714 Aug 22 2017 RC19.2004_v0001.cub
-rw-r--r-- 1 patf domain users 5833714 Aug 22 2017 RC19.2005_v0001.cub
-rw-r--r-- 1 patf domain users 5833714 Aug 22 2017 RC19.2006_v0001.cub
-rw-r--r-- 1 patf domain users 5833714 Aug 22 2017 RC19.2007_v0001.cub
-rw-r--r-- 1 patf domain users 5833714 Aug 22 2017 RC19.2008_v0001.cub

Sorry about the long lines. The same has to be done for .../solar-spectrum, .../wave-cal, ...band-wavelengths files. One could also use files from the RC20FINAL set, I suspect they are the same.

That said, I can also put together a set of recent Saturn files, which can be processed more easily. I'll get those on here soon.

[patmfry]

@acpaquette by the way, here is version of isis I was using for the previous processing...
[patf@puck isis3]$ cat version
3.9.1 # Public version number
2019-11-19 # Release date
stable # Release stage (alpha, beta, stable)

[patmfry]

vimsv_saturn_goodbad.zip

_vis.cub file is uncalibrated (vims2isis, spiceinit) cube, _vis_se_RC19.cub is calibrated (broken) radiance file, _vis_if_RC19.cub is calibrated (broken) I/F file. There are FITS versions of these last two file types as well. Those FITS files were processed similarly to your fx solution to fix the radiance and I/F files, those files endint in fix.fits.

[acpaquette]

I managed to get ahold of the original qub and was trying to process that in full but ran into an error that the there was no mult file for 2000. What you mentioned above helps clear things up! Would you happen to know the reasoning behind those few files being named differently from the rest?

I will start working with the Saturn data today and see how far I get. Thank you for the help @patmfry

[patmfry]

@acpaquette I think the reason for the lack of multiplier files for the cruise and Jupiter encounters was just that at the time the time-dependent wavelength/throughput files were created, the priority was getting those out for the immediate Saturn data processing; there probably wasn't time/money/enthusiasm by the VIMS calibration team for extending the analyses to the earlier data. Fortunately the later/final releases of calibration data do cover the earlier epochs, though the ISIS vimscal does not yet use those (RC20/RC20FINAL) files.

[acpaquette]

@patmfry I can replicate the Saturn image up to the calibration step but when I apply the fx fix I am seeing significant difference in the result. Is there anything different you are doing to apply the fix? Currently I am running:
vims2isis from=v1749924310_1.qub vis=v1749924310_1_vis.cub ir=v1749924310_1_ir.cub
(From original you provided)

spiceinit from=v1749924310_1_vis.cub web=false attach=TRUE cksmithed=FALSE ckrecon=TRUE ckpredicted=FALSE cknadir=FALSE spksmithed=FALSE spkrecon=TRUE spkpredicted=FALSE shape=SYSTEM startpad=0.0 endpad=0.0 url=https://services.isis.astrogeology.usgs.gov/cgi-bin/spiceinit.cgi port=443
(From original you provided)

vimscal from=/Users/acpaquette/Desktop/vimsv_saturn_goodbad/v1749924310_1_vis.cub to=/Users/acpaquette/Desktop/vimsv_saturn_goodbad/v1749924310_1_vis_cal.cub units=SPECENERGY flatfield=2006FLAT irorigdark=false dark=true
(v1749924310_1_vis_cal.cub == V1749924310_1_vis_se_RC19.cub)

fx f1=/Users/acpaquette/Desktop/vimsv_saturn_goodbad/v1749924310_1_vis_cal.cub f2=/Users/acpaquette/isis3data/cassini/calibration/vims/vis_perf_v0002.cub+- 1-96 to=/Users/acpaquette/Desktop/vimsv_saturn_goodbad/v1749924310_1_vis_cal_fx.- cub equation="(f1/39474.12689222814) * f2" mode=CUBES
(v1749924310_1_vis_cal_fx_flip.cub != V1749924310_1_vis_se_RC19fix.cub)
The output needs to be flipped as well, not sure why.

Does this look like the pipeline you have been using or is there some significant step I am missing? The resulting image I end up with isn't as clean as V1749924310_1_vis_se_RC19fix.cub. There is noticeable striping in the image on various samples which leads me to believe that we our processing steps are varying somewhere.

[patmfry]

@acpaquette It looks like you're doing the same processing I do, except I forgot that I do a destriping of the visible bands when I do the isis to fits conversion. The destriping is described in Adriani et al. 2007 ("The de-striping of the VIMS-V images...", see attached jpeg). Other than the cosmetic destriping, do your spectra looks fairly similar to my fixed spectra?

[acpaquette]

@patmfry They are fairly similar. The differences range from 0-334 with an average difference of 7. Would it be possible to process the image and ignore the de-striping step on your end?

[patmfry]

@acpaquette Sure. I'll probably get to it tomorrow morning.

[patmfry]

Here are SE and I/F files, no destriping, with and without fix..
vimsv_saturn_nodestripe_goodbad.zip

[acpaquette]

Sweet! I am getting almost the same result up to 4 sig figs. I should have something done by the end of the day. Thanks for being prompt and helping to get data to us @patmfry!