Isis3: Cleanup before switching to Amazon Cloud for backups

[SgStapleton]

We are currently running cronjobs on flytrap for backup of the entire /usgs/cpkgs/isis3 directory. We will soon be moving to Amazon Cloud Storage and need to decide what can/needs to be backed-up here. We cannot backup everything in this directory as there is sensitive material in random files (such as ssh keys and login credentials). We also do not WANT to backup this entire directory as much of it is out of date. (We have old cvs files from before svn!!!)

We are currently in the process of getting a full and current copy of the directory into flytrap to enable nightly backups yet again (https://github.com/USGS-Astrogeology/ISIS3/issues/3206). Once this backup has completed successfully, it might be a good time to go in and clean out what is not needed in this directory, what needs to stay here, and what can go out on Amazon Cloud.

[jlaura]

@scsides Can you take a look at the directory and give some feedback on what we can move / purge?

[jlaura]

Also @all we appear to be cleared to execute this. I would love to make this happen during the next support / maintenance sprint.

[scsides]

First draft suggestion with caveats

WORK IN PROGRESS:

/usgs/cpkgs/isis3 -------------------------
bin_1.8 - NO
bin_svn - NO (consider moving to archive area)
cvs - NO (consider moving to archive area)
data - YES
    QUESTIONS:
      What services will be available (rsync, ftp, ...)?
      A: gitlab with lfs only
      Will we be able to obtain logs for downloads?
       A: Current process is for revision control only
    NOTES:
      spiceserver updates from here
      local kernel maint pushes to here
      clean up:
        old kernel.????.db files
        old moc_kernels.????.db files
        old daily, weekly, monthly kernels replace by higher order
        old predict spk and ck
        Possibly older version of IK
        Possibly older versions of IAK
      public web page doc point to isisdist.wr.usgs.gov and isisdist.astrogeology.usgs.gov
      ISIS3 github docs point to same rsync servers
      Re-write log parser

data_archive - NO (consider moving to archive area, or delete)
datalocal - NO (private data)
i3_cpkgs.du.txt - No (consider delete)
ipce_logs - NO (consider delete)
isis3beta_logs - NO (consider delete)
isis3mgr_scripts - NO, but clean up
isis3mgr_scripts_old - NO (consider moving to archive area, or delete)
isis3production_logs - NO (consider delete)
kernel_staging - NO (private data, staging area for spice maint)
logs - NO (consider delete after CI)
resources - NO (consider archiving, possible in GIT)
scripts.html - NO (spiceserver, kernel maint)
svn_dependencies.txt - NO (consider delete)
testData - YES
  NOTES:
      clean up obsolete OS dependent files -  SuSE, Old Darwin,..
      remove tests for apps that don't exist
      **remove private data**
      SUBDIRS - looks like some of these shouldn't be there at all
        bin - delete
        config - delete
        IsisDlm - Archive
        IsisPreferences - delete
        lib - delete
        license.txt - delete
        make - delete
        Makefile - delete
        scripts - delete
        src - CLEAN
        templates - delete
        version - delete

testDataBeta - NO (consider delete)
testDataIsis3.6.0 - NO (consider delete after under revision)
usertests - NO (consider moving to archive area)

[SgStapleton]

Just a note on the logs dir - this is where we are currently backing-up our nightly test logs, so we may need to find somewhere else to store these...? (This is also where the nightly emails are parsing from)

[scsides]

Agreed, I've added a note about CI

[SgStapleton]

Perfect.

[jlaura]

As our S3 buckets are now ready to go, I would like to get this pushed into the queue to potentially address next week during the support/maintenance sprint if possible.

[jlaura]

Just an update - the approval for GitLab is rolling through our security process now. Once we have that approval we will be good too go live. I am tagging this with 'Current Sprint' milestone in case we run short on items to complete. If we are able to get the directory into a state to push, I can work on getting the push to S3 done prior to the next support/maintenance sprint. If not, no big and we can do this inside the next sprint.

[SgStapleton]

First draft suggestion with caveats (In Progress)

Following are @scsides suggestions with progress completed. (This entire directory has been successfully backed-up in /work/projects/flytrap/isis-bank/isis3-vault/ )

WORK IN PROGRESS:

/usgs/cpkgs/isis3 -------------------------
bin_1.8 - NO --gone
bin_svn - NO (consider moving to archive area) --gone
cvs - NO (consider moving to archive area) --gone
data - YES
data_archive - NO (consider moving to archive area, or delete) --gone
datalocal - NO (private data) --gone
i3_cpkgs.du.txt - No (consider delete) --gone
ipce_logs - NO (consider delete) --gone
isis3beta_logs - NO (consider delete) --gone
isis3mgr_scripts - NO, but clean up --(see note below)
isis3mgr_scripts_old - NO (consider moving to archive area, or delete) --gone
isis3production_logs - NO (consider delete) --gone
kernel_staging - NO (private data, staging area for spice maint) --(see note below)
logs - NO (consider delete after CI) --(see note below)
resources - NO (consider archiving, possible in GIT) --gone
scripts.html - NO (spiceserver, kernel maint) --(see note below)
svn_dependencies.txt - NO (consider delete) --gone
testData - YES
testDataBeta - NO (consider delete) --gone
testDataIsis3.6.0 - NO (consider delete after under revision) --gone
usertests - NO (consider moving to archive area) --gone

[SgStapleton]

Special Notes: @scsides maybe you can answer a few of these questions/concerns?

datalocal - What is this and does it need to be preserved somewhere else (other than archive)?
A: Must remain as is or scripts that use it must be changed. CAN NOT be moved to S3/gitlab
isis3mgr_scripts - We have quite a bit tied-up in this dir as far a scripts are concerned and will need to locate how far this reaches, but I imagine we need to find another home for this as the /usgs/cpkgs/isis3 dir will soon be it's own git repo and we already have a repo established for our scripts (no point in saving them in two separate repos!)
A: New thinking is that only data and testdata will move to a repo dir and may be symlinked to preserve current structure
kernel_staging - I have a feeling that this is somehow used to implement kernel updates, but I may be wrong?
A: Yes, kernels updates for LRO, & Cassis use this area.
logs - These are our nightly build logs. Once CI is set-up, this can go away entirely, but until that happens we need to find a new home for these and make sure scripts are updated to reflect this.
A: New thinking should allow this to remain as is until it is no longer needed i.e., CI
scripts.html - What is this and what uses it? Where else should we locate this...?
A: This is the html documentation for the scripts in isis3mgr_scripts. It is generated nightly (cron job) from the -h option of each script. See: isis3mgr_scripts/documentScripts. The New Thinking applies

[SgStapleton]

I will be working on getting the data and testData dirs cleaned-up.

[SgStapleton]

@jlaura It looks like all of the extra dirs in /usgs/cpkgs/isis3/testData belong to you. Everything out of there can go besides src/

[jessemapel]

@SgStapleton datalocal is where we put proprietary data that we cannot release to the public. We need to keep it somewhere but we cannot put it out to the public.

[scsides]

Answers and comments added as "A:" to original questions

[SgStapleton]

Removed everything I can from /usgs/cpkgs/isis3. I am waiting for the go-ahead on where to archive what from here needs to be archived, and then they may be removed from here as well.

@jlaura @scsides and myself discussed housing the new local repository at the same level as the data and testData dirs and eventually changing the data and testData to simple symlinks that point to dataRepo/(data and testData), but decided that it would be best keeping two full physical copies until we work-out the kinks. Added a dataRepo dir to house our new local repository, but as I was copying over data/, I ran out of space. This would mean that we need more space allocated to /usgs/cpkgs/isis3.

Opened an informational issue https://github.com/USGS-Astrogeology/ISIS3/issues/3282

[SgStapleton]

List of what will be staying in /usgs/cpkgs/isis3 once archive is complete:

• data (will be converted to symlink eventually)
• datalocal (proprietary data that cannot exist in public-facing repository)
• dataRepo (our new local git repository)
• isis3mgr_scripts (local git repository)
• kernel_staging (needed for kernel download scripts)
• logs (used by nightly scripts - can be deprecated once CI fully in place)
• scripts.html (This file is produced nightly by scripts)
• testData (will be converted to symlink eventually)
• testDataIsis3.6.0 (I am actually not entirely sure when this can go...)

[kberryUSGS]

From one of the earlier posts, focusing on spice-related cleanup:

   clean up:
        old kernel.????.db files
        old moc_kernels.????.db files
        old daily, weekly, monthly kernels replace by higher order
        old predict spk and ck
        Possibly older version of IK
        Possibly older versions of IAK

~6,000 old kernel database files were cleaned up as part of the Isis Spice sprint (thanks @kaitlyndlee and @acpaquette !), but we didn't clean up any of the other categories.

@scsides @SgStapleton Did either of you end up doing any cleanup of the spice-related things listed above, and if not, would it still be useful to do so? I'm asking because if not, some of these could be added as potential side-tasks for the next Isis Spice sprint. It would be easy, for example, to add the moc_kernels to the list of kernel db files being automatically cleaned up (keep the last 10?)

[scsides]

@jlaura I have not, but it would be import to purge these and keep them purged before the move.

For the IK and IAK there are only a few and they are small. It would be nice to have these under revision control and then remove the older ones, so we can see how they changed if necessary.

[SgStapleton]

@kberryUSGS I did not do any of the cleanup yet within the directories that still exist there. They are untouched!

[kberryUSGS]

I'm cleaning up the extra directories in testData now:

From earlier in this issue:

SUBDIRS - looks like some of these shouldn't be there at all
        bin - delete
        config - delete
        IsisDlm - Archive
        IsisPreferences - delete
        lib - delete
        license.txt - delete
        make - delete
        Makefile - delete
        scripts - delete
        src - CLEAN
        templates - delete
        version - delete

Update: These are removed. Now working on cleaning up test data that no longer is used by any test. There are a lot of these.

[ascbot]

I am a bot that cleans up old issues that do not have activity.

Happy Birthday to this issue! :birthday:

Unfortunately, this issue has not received much attention in the last 12 months. Therefore, I am going to close it. Please feel free to reopen this issue or open a new issue sometime in the future. If this issue is a bug report, please check that the issue still exists in our newest version before reopening.