Pkg.jl: advice for recovering from corrupted depot in the manual

[tpapp]

Occasionally parts of the depot become corrupted (because of interrupted downloads/processes, bugs in Pkg, etc). While this is not part of the normal workflow, it would be great if the manual could have a section on how to recover from these situations, that could be linked eg on Discourse.

Eg

1. establish the depot path (examples below assume the default)
2. remove ~/.julia/packages
3. remove the global manifest,
4. restart julia and do a pkg> resolve?

[StefanKarpinski]

Might be worth adding a pkg> fsck command that goes through verifying things and trying to fix them and/or delete/recreate them.

[DilumAluthge]

1. Commit all changes in any packages that you have checked out for development
2. Push all changes to a remote (GitHub, GitLab, etc.)
3. rm -rf ~/.julia

And of course, if you don't have any packages checked out for development, then you simply do:

1. rm -rf ~/.julia

[giordano]

I'd never suggest to people to nuke the julia directory. Why not just move it?

[tpapp]

I agree with @giordano on this --- we should never recommend just deleting things.

Until pkg> fsck is added, it would be great to determine the gentlest minimum solution. It is needed occasionally to recover from various problems, and we should have a standard way.

My understanding is the following:

1. one can remove artifacts, clones, compiled, packages, registries, they will just be downloaded at no cost if the machine is online (we should mention this, some environments have no net connection)

2. when dev exists, it may contain uncommitted stuff, and it is generally harmless

3. logs, prefs, and config are generally not a source of inconsistent state, so they can be left alone

4. environments is really small, and should be just moved to a backup location, eg

   mv ~/.julia/environments ~/.julia/environments-backup-`date +%Y-%m-%d`
   

Is this correct? Did I miss anything?

[StefanKarpinski]

Can't think of anything else. However, logs can cause problems since manifest usage is tracked there and the file can become corrupted (invalid TOML due to multiple writers).

[fredrikekre]

For 1.: compiled is not downloaded but generated, safe to remove just like the others though.

[tpapp]

Thinking about this, it may be preferable to include something in Pkg instead of a shell script that people copy-paste. I am thinking of something like the following:

pkg> nuke
This command needs to be invoked with --yes to do anything; this is just a message.

It can be used to recover from an inconsistent state of the package depot by removing 
files that were downloaded or automatically generated. The removed files will be
saved to a backup directory (which is displayed).

Used with --environment, it also resets your default environment.

--yes would not autocomplete.

pkg> nuke --yes
Moving files to /home/tamas/.julia_depot_backup_2020-03-23_1042.

pkg> nuke --yes --environment
Moving files to /home/tamas/.julia_depot_backup_2020-03-23_1044, resetting environment.

[StefanKarpinski]

Was thinking of something similar myself.

[tpapp]

Since corrupted registry questions come up daily on Discourse, I am wondering how to move forward with this. In order of increasing difficulty:

1. write up a bunch of commands for users to copy/paste,
2. implement nuke above,
3. implement fsck (cf #1717)