Isis3: hyb2onc2isis needs updating to deal with new Hayabusa2 FITS image labels

[foobarbecue]

ISIS version(s) affected: 3.9.1

Description
When running hyb2onc2isis with any of the FITS images JAXA released for their July 2018 orbits, from http://darts.isas.jaxa.jp/pub/hayabusa2/onc_bundle/browse/ , hyb2onc2isis fails with:

**PROGRAMMER ERROR** No value or default value to translate for translation group [InstrumentId] in file [/data/JPL_Stereo/Isis/data/hayabusa2/translations/hyb2oncInstrument.trn].

How to reproduce
Install ISIS 4.0.0, download hyb2_onc_20180710_060508_tvf_l2a.fit and run hyb2onc2isis from=hyb2_onc_20180710_060508_tvf_l2a.fit to=hyb2_onc_20180710_060508_tvf_l2a.cub

Possible Solution
Labels on newer Hyabusa images are presumably different from labels used in older images against which hyb2onc2isis was developed.

Additional context
Initially discussed at https://astrodiscuss.usgs.gov/t/problem-injesting-hyabusa2-data/374/6

[foobarbecue]

We could also fix some inconsistencies in the hyb2onc2isis docs regarding input file types at the same time.

[AustinSanders]

For reference, the related issue was resolved by adding a default value to the transformation file. It is likely that this will involve a similar fix.

[foobarbecue]

I started digging into this. I dumped a .fits label and compared to the .trn files.

Lots needs changing. I was able to figure out many of the changes, but some labels appear not to be present and so I just commented them out... probably the wrong approach. Now, hyb2onc2isis works but spiceinit fails with **ERROR** No Camera Kernels found for the image [hyb2_onc_20180801_094857_tvf_l2a.cub]

Where are these .trn files hosted, and how do I propose changes to them? The README in the same folder says "These translation files are maintained and archived on astrogit under the fy17_hayabusa2 repo."

In the meantime, here's a diff of the changes I made in order to get hyb2onc2isis to stop outputting errors:
diff.txt

edit: I also commented out the ImageNumber group in hyb2oncArchive.trn , but that didn't make it into the above diff.

[jessemapel]

@foobarbecue We're currently working to get those files pushed into the repo. For right now though, only the USGS ASC has access to make changes. We've got this issue slotted for work at the end of March so we'll take you're changes and start off of them getting this resolved.

The spiceinit error looks like it's caused by a lack of kernels. We only have kernels up to 2016 in $ISIS3DATA/hayabusa2/kernels/ck, I'm not sure why that is. @kberryUSGS do we have a kernel update script for Hayabusa 2?

Thanks for putting so much work into this, it'll help us get this resolved much easier.

[kberryUSGS]

We do have a kernel update script for Hayabusa2. It's pulling kernels from http://www.darts.isas.jaxa.jp/pub/hayabusa2/spice_bundle/spice_kernels/, but the most recent cks there are from 2016. I'm not sure if there is a more up-to-date source.

[jlaura]

@scsides Do you want to check with the team and see if they have updated CKs that we can pull?

[foobarbecue]

I see kernels up to 2019 are now available at https://www.darts.isas.jaxa.jp/pub/hayabusa2/spice_bundle/spice_kernels , and sisdist.astrogeology.usgs.gov rsync. Great, I'll see if I get spiceinit to work now.

[foobarbecue]

Almost!

(isis3) aacurtis@lmmp-pipeline-ubuntu:/trek.devdata/ryugu/l2a/20180801$ spiceinit from=hyb2_onc_20180801_155438_tvf_l2a.cub
Group = Kernels
  NaifFrameCode             = -37100
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = $hayabusa2/kernels/pck/pck00010.tpc
  TargetPosition            = $hayabusa2/kernels/tspk/de430.bsp
  InstrumentPointing        = ($hayabusa2/kernels/ck/hyb2_hkattrpt_2018_v02.b-
                               c, $hayabusa2/kernels/fk/hyb2_v14.tf)
  Instrument                = $hayabusa2/kernels/ik/hyb2_onc_v05.ti
  SpacecraftClock           = $hayabusa2/kernels/sclk/hyb2_20141203-20161231_-
                              v01.tsc
  InstrumentPosition        = Null
  InstrumentAddendum        = $hayabusa2/kernels/iak/hyb2oncAddendum0001.ti
  ShapeModel                = Null
  InstrumentPositionQuality = Unknown
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Error                     = "Can not find [BODY2162173_RADII] in text
                               kernels"
End_Group

Group = Kernels
  NaifFrameCode             = -37100
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = $hayabusa2/kernels/pck/pck00010.tpc
  TargetPosition            = $hayabusa2/kernels/tspk/de430.bsp
  InstrumentPointing        = ($hayabusa2/kernels/ck/hyb2_aocsc_2018_v02.bc,
                               $hayabusa2/kernels/fk/hyb2_v14.tf)
  Instrument                = $hayabusa2/kernels/ik/hyb2_onc_v05.ti
  SpacecraftClock           = $hayabusa2/kernels/sclk/hyb2_20141203-20161231_-
                              v01.tsc
  InstrumentPosition        = Null
  InstrumentAddendum        = $hayabusa2/kernels/iak/hyb2oncAddendum0001.ti
  ShapeModel                = Null
  InstrumentPositionQuality = Unknown
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Error                     = "Can not find [BODY2162173_RADII] in text
                               kernels"
End_Group
**ERROR** Unable to initialize camera model.

[jessemapel]

That's a pck error. Ryugu isn't in pck10, try downloading this: https://www.darts.isas.jaxa.jp/pub/hayabusa2/spice_bundle/spice_kernels/pck/hyb2_ryugu_shape_v20190328.tpc and passing it via the pck parameter.

[foobarbecue]

Well, I'm not sure why the change but now I'm getting:

(isis3) aacurtis@lmmp-pipeline-ubuntu:/trek.devdata/ryugu/l2a/20180801$ spiceinit from=hyb2_onc_20180801_174821_tvf_l2a.cub
Group = Kernels
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = $hayabusa2/kernels/pck/pck00010.tpc
  TargetPosition            = $hayabusa2/kernels/tspk/de430.bsp
  InstrumentPointing        = ($hayabusa2/kernels/ck/hyb2_hkattrpt_2018_v02.b-
                               c, $hayabusa2/kernels/fk/hyb2_v14.tf)
  Instrument                = $hayabusa2/kernels/ik/hyb2_onc_v05.ti
  SpacecraftClock           = $hayabusa2/kernels/sclk/hyb2_20141203-20161231_-
                              v01.tsc
  InstrumentPosition        = Null
  InstrumentAddendum        = $hayabusa2/kernels/iak/hyb2oncAddendum0001.ti
  ShapeModel                = Null
  InstrumentPositionQuality = Unknown
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Error                     = "PVL Keyword [NaifIkCode] does not exist in
                               [Group = Kernels]"
End_Group

Group = Kernels
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = $hayabusa2/kernels/pck/pck00010.tpc
  TargetPosition            = $hayabusa2/kernels/tspk/de430.bsp
  InstrumentPointing        = ($hayabusa2/kernels/ck/hyb2_aocsc_2018_v02.bc,
                               $hayabusa2/kernels/fk/hyb2_v14.tf)
  Instrument                = $hayabusa2/kernels/ik/hyb2_onc_v05.ti
  SpacecraftClock           = $hayabusa2/kernels/sclk/hyb2_20141203-20161231_-
                              v01.tsc
  InstrumentPosition        = Null
  InstrumentAddendum        = $hayabusa2/kernels/iak/hyb2oncAddendum0001.ti
  ShapeModel                = Null
  InstrumentPositionQuality = Unknown
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Error                     = "PVL Keyword [NaifIkCode] does not exist in
                               [Group = Kernels]"
End_Group
**ERROR** Unable to initialize camera model.

[jlaura]

hyb2onc2isis needs updating to deal with new Hyabusa2 FITS image labels

[kberryUSGS]

Hi @foobarbecue, the changes you recommended were merged in PR #3813 . I needed to make an additional change to fix the NaifIkCode error you experienced, see the change to Hayabusa2OncKernels.trn in the PR. The updated software will be available in the ISIS 4.1 release candidate, which is currently being worked on and I believe is slated to be released by the end of this week or early next week. Thank you again for the diff, it saved a lot of time!

After I was able to ingest your example Hayabusa2 fits file, to get the resulting cube spiceiniting, I did need to make some changes to the hayabusa2 kernels area. These changes will be available on the rsync server when the release candidate is released. If you'd like them earlier, I could also push an earlier update, just let me know!

[foobarbecue]

Wonderful! I will definitely use this in the next few days, if you are able to push the earlier kernel area update.

[jessemapel]

@foobarbecue We'll be pushing the most recent kernels we have as we've changed the data area around for 4.1. It looks like we have coverage up to beginning of 2019 currently.

[jessemapel]

@kberryUSGS Can this be closed?

[foobarbecue]

Hi, I just got around to trying this with the 4.1.0 RC and from spiceinit I get:

Error                     = "Can not find [BODY2162173_RADII] in text
                               kernels"

I don't need Hayabusa 1 kernels for Hayabusa2, do I?

[jessemapel]

The Hayabusa2 pck kerneldb isn't pulling in the ryugu PCK. Try adding PCK=$ISISDATA/hayabusa2/kernels/pck/hyb2_ryugu_shape_v20190328.tpc.

@kberryUSGS Is it okay if I manually update the kernel.db?

[kberryUSGS]

@jessemapel I already updated the kernel.db to pick up the new pcks, but they weren't pushed to the rsync server until just now. I pushed just the hayabusa2 kernel changes to the rsync server today.

@foobarbecue Try updating your hayabusa2 data area using the following for use with the ISIS 4.1.0 RC:

cd $ISISDATA
rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/hayabusa2 . --dry-run

I added the --dry-run above so you can see if the changes proposed by rsync are reasonable. If the results for the above rsync look reasonable, run without the --dry-run to actually pull down the changes.

You should see that PCK file (among other changes) come down in your rsync.

The rsync module name has changed from isis3data to isisdata for data for use with ISIS 4.1.0 RC. This information (and more!) will be in the ISIS 4.1.0 RC release notes when the RC is officially released.

[foobarbecue]

Thanks guys, after updating kernels from the isisdata rsync I was able to successfully run hyb2onc2isis followed by spiceinit. I then tried hyb2onccal and got:

(isis4) aacurtis@lmmp-pipeline-ubuntu:/trek.devdata/ryugu/l2a/20180801$ hyb2onccal from=hyb2_onc_20180801_100809_tvf_l2a.cub to=from=hyb2_onc_20180801_100809_tvf_l2a.cal.cub
**ERROR** PVL Keyword [SelectedImageAreaY4] does not exist in [Group = Instrument].

[foobarbecue]

Think I figured it out, working on a quick patch