Borg: Borg repository restore/reinstall instructions are needed

[jk-1]

In case of HW failure or media corruption you need to restore your borg repos from a backup media. Because the restored repo is almost always older than damaged repo you need to to something with the client cache to get the restored repo working.

I tried following scenarions to restore repo after an imaginary HW failure:

This is the original working repo soon to be damaged:

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
b01      Mon, 2017-12-11 10:32:37   "Initial-backup" 
b02      Mon, 2017-12-11 10:32:38   "Second-backup" 
b-mod-bckp-01    Mon, 2017-12-11 10:32:42   "This-is-mod-bckp-01" 
b-mod-bckp-02    Mon, 2017-12-11 10:32:43   "This-is-mod-bckp-02" 
b-mod-bckp-03    Mon, 2017-12-11 10:32:44   "This-is-mod-bckp-03" 

Repo is periodically backed up to another "secure" location, e.g. removable disk, tape etc.

~/tmp/RepoTest$ cp -avr repo01 repo01.1

Repo is in use, backups are stored into it.

borg create repo01::B03 dir01 --comment "B03"

~/tmp/RepoTest$ cp -avr repo01 repo01.2

borg create repo01::B04 dir02 --comment "B04"

~/tmp/RepoTest$ cp -avr repo01 repo01.3

borg create repo01::B05 dir02 --comment "B05"

Situation before crash:

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
b01      Mon, 2017-12-11 10:32:37   "Initial-backup" 
b02      Mon, 2017-12-11 10:32:38   "Second-backup" 
b-mod-bckp-01    Mon, 2017-12-11 10:32:42   "This-is-mod-bckp-01" 
b-mod-bckp-02    Mon, 2017-12-11 10:32:43   "This-is-mod-bckp-02" 
b-mod-bckp-03    Mon, 2017-12-11 10:32:44   "This-is-mod-bckp-03" 
B03      Mon, 2017-12-11 10:37:38   B03 
B04      Mon, 2017-12-11 10:38:36   B04 
B05      Mon, 2017-12-11 10:39:23   B05 

Then there is a HW failure or other serious problem so that a backup is needed:

1) Repo restore by simple delete & copy

~/tmp/RepoTest$ rm -rf repo01

~/tmp/RepoTest$ cp -avr repo01.1 repo01

This gives following error:

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
Cache is newer than repository - do you have multiple, independently updated repos with same ID?

2) Repo restore by simple delete & copy and borg cache delete

~/tmp/RepoTest$ rm -rf repo01

~/tmp/RepoTest$ cp -avr repo01.1 repo01

~/tmp/RepoTest$ borg delete repo01 --cache-only
~/tmp/RepoTest$ echo $?
0

This gives following error:

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
Cache is newer than repository - do you have multiple, independently updated repos with same ID?

3) Repo restore by borg delete and simple copy

~/tmp/RepoTest$ borg delete repo01
You requested to completely DELETE the repository *including* all archives it contains:
b01                                  Mon, 2017-12-11 10:32:37 [5e1bf5a7677972470307332f33bb2e683f178130022769c17a67d07202d6946b]
b02                                  Mon, 2017-12-11 10:32:38 [6d630d83a7944fb460a057164c2d5d243c482001a7a036834a0e4b674ab94eeb]
b-mod-bckp-01                        Mon, 2017-12-11 10:32:42 [67478151c2313331e76b1af20974bbc081a029bf69745fe660a8bac690b51e11]
b-mod-bckp-02                        Mon, 2017-12-11 10:32:43 [61c77f5934b2091b6deab168f8a8b92633cc9bbc2f4d6a6ea6bffb8ea504fe99]
b-mod-bckp-03                        Mon, 2017-12-11 10:32:44 [63d5c7456c904f9687a1a09a0d6e7755a5a6a70567c0cfb11481e0e75a355d3a]
Type 'YES' if you understand this and want to continue: YES

~/tmp/RepoTest$ cp -avr repo01.1 repo01

This gives following error:

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
Cache is newer than repository - do you have multiple, independently updated repos with same ID?

4) Repo restore by simple delete & copy & repo id modification

~/tmp/RepoTest$ rm -rf repo01
~/tmp/RepoTest$ cp -avr repo01.1 repo01

according to this: https://github.com/borgbackup/borg/issues/1559 modify the repo id by hand

~/tmp/RepoTest$ borg list repo01 --format="{archive} {TAB} {time} {TAB}{comment} {NEWLINE}"
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
b01      Mon, 2017-12-11 10:32:37   "Initial-backup" 
b02      Mon, 2017-12-11 10:32:38   "Second-backup" 
b-mod-bckp-01    Mon, 2017-12-11 10:32:42   "This-is-mod-bckp-01" 
b-mod-bckp-02    Mon, 2017-12-11 10:32:43   "This-is-mod-bckp-02" 
b-mod-bckp-03    Mon, 2017-12-11 10:32:44   "This-is-mod-bckp-03" 

Scenatio 4. works if the repo is NOT encrypted, but few questions arise about the scenario 4:

• How to do this if the repo is encrypted
• What happens to the cached data of the damaged repo? Will it remain in the cache forever?
• Are there any limitiations for the repo id? E.g. what is max length of the id? Should I modify one character in the id or append one character to the end?

And other related questions:

• What is the purpose of the "borg delete repo --cache-only" or did I use it somehow incorrectly because it did not help here?
• Why didn't the "borg delete repo" command delete related client cache? Here client and server are on the same machine.
• If I want to manually delete all e.g. repo01 specific cache files from ~/.cache/borg then how can I know which files to delete?

And finally: Should there be somekind of dedicated short "repo restore instructions" for this kind of (usually emergency) restore situations?

Thank you for your help.

[ThomasWaldmann]

"cache is newer" is of course expected for this scenario (if your copy is older than the original).

With this error message, the expectation is to solve it by deleting the cache, but the data used for this check is not deleted by that, see also #3427.

I suspect you use borg 1.1.x, right?

The error messages in current borg release are like this:

    class RepositoryIDNotUnique(Error):
        """Cache is newer than repository - do you have multiple, independently updated repos with same ID?"""

    class RepositoryReplay(Error):
        """Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)"""

[jk-1]

Borg version is 1.1.3

How can I delete cache file related to a certain repo? I thought that "borg delete repo01 --cache-only" would to that but it did not.

[ThomasWaldmann]

Yes, the security dir not getting deleted is likely a bug (or at least unexpected), see the other ticket.

[jk-1]

On v1.1.3 "borg delete repo" deletes repo dir and its sub directories.

[DimeNastov]

I have ran into this issue this morning after restoring repositories from archives.
Using borg v.1.1.4 this is the procedure that works for encrypted repositories, at least in my case:

1. execute on the client server: borg delete reponamehere --cache-only
2. rename the .config/borg/security to .config/borg/security.bak or something else. Do not delete it as you might need it for further debugging.

After this you should be able to do a" borg info" and "borg list" just fine, and the backup and restore continues to function as previously.

I hope this helps someone.

[ThomasWaldmann]

@DimeNastov you don't want to rename the whole security dir, but rather .config/borg/security/<repoid>. you can get the repo's id from repo_dir/config of the repository in question.

[tuxick]

I ran into this problem and solved it by (re)moving ~/.cache/borg/\ After this "borg list" no longer gave me this error but running check on repo seems wise.

[shyam334]

I ran into the same issue using v1.1.7 and Thanks for the workarounds.
Just wondering what is a proper fix for this (after seeing the comment in #3427).

[Quentin-M]

Same here after a crash+restart of my borg server.

[ThomasWaldmann]

Just as an update:

Scenario 3 (see also below) should work since borg 1.1.4 (having the fix for #3427) and is the recommended method if your repository is corrupted and you have a good, but older copy of it.

$ borg delete repo    # kill corrupted repository (and cache and security info for it)
$ cp -avr repo-good repo

[fullofcaffeine]

I have ran into this issue this morning after restoring repositories from archives.
Using borg v.1.1.4 this is the procedure that works for encrypted repositories, at least in my case:

1. execute on the client server: borg delete reponamehere --cache-only
2. rename the .config/borg/security to .config/borg/security.bak or something else. Do not delete it as you might need it for further debugging.

After this you should be able to do a" borg info" and "borg list" just fine, and the backup and restore continues to function as previously.

I hope this helps someone.

I had a similar issue recently when I needed to sync from remote (using rclone) because I accidentally pruned my local copy of some important archives. I cancelled before the data was rcloned to remote, so I just rcloned it back to local (downloading the full repo would take forever, this repo is ~1TB, rcloning only the diff took a min or so). The solution described above worked for me. Thanks!

[fantasya-pbem]

@ThomasWaldmann So what is a doc article expected to be like? Describe scenario 3 as best practice (for >= 1.1.4)? Should we add the "rename security dir" method too? (I don't think so, 1.1.3 is quite old nowadays.) So maybe just another FAQ entry?

[maxammann]

@fantasya-pbem as far as I understood it is ok do to borg delete reponamehere --cache-only and deleting entries from ~/.config/borg/security, right?

I mean when using only a single client which backups to the remote then this procedure is equal to deleting the whole repository and copying it back from the remote to the client.

EDIT: From a security perspective :)