Borg: Allow operations on read-only filesystems

[lxp]

I think it makes sense to allow repository checks on read-only filesystems.
I would already be happy, if I could force borg to not try to lock the repository during a check (with all it's implications).
Especially, in case of a check without repair, the worst thing that could happen should be that the check fails. Or am I wrong?

$ borg check /srv/repository
Failed to create/acquire the lock /srv/repository/lock.exclusive ([Errno 30] Read-only file system: '/srv/repository/lock.exclusive').
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1498, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1452, in run
    return args.func(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 72, in wrapper
    with repository:
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 80, in __enter__
    self.open(self.path, self.exclusive, lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 161, in open
    self.lock = UpgradableLock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 261, in acquire
    with self._lock:
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 118, in __enter__
    return self.acquire()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 141, in acquire
    raise LockFailed(self.path, str(err)) from None
borg.locking.LockFailed: ('/srv/repository/lock.exclusive', "[Errno 30] Read-only file system: '/srv/repository/lock.exclusive'")

Platform: Linux debian 4.5.0-0.bpo.1-amd64 #1 SMP Debian 4.5.1-1~bpo8+1 (2016-04-20) x86_64 
Linux: debian 8.4 
Borg: 1.0.2  Python: CPython 3.4.2
PID: 8075  CWD: /root
sys.argv: ['/usr/bin/borg', 'check', '/srv/repository']
SSH_ORIGINAL_COMMAND: None

[ThomasWaldmann]

Currently not possible. A writer wait until there are no readers, readers wait until there is no writer.

[lxp]

I understand that it is implemented like that now, but isn't it possible to add a --no-lock option or something like that?
I know that there are no writers, if the filesystem is read-only and no lock is around.
Of course, a filesystem remount could make the filesystem later read-write and a writer could start writing because there is no reader lock around. However, with a --no-lock option I could knowingly accept that risk.
Do you expect the repository to get damaged if a writers starts acting on the repository while a check is running? I really do not care if the check fails in that for me unlikely situation.

[ThomasWaldmann]

Maybe not. But before introducing a --ignore-lock one should be very sure.

BTW, is your repo read-ony so client's can't delete anything usually? Maybe "append-only" repo mode is something for you, see docs.

[lxp]

In my case I am doing a daily local backup to a different disk, which I remount rw before the backup and back ro afterwards. I remount ro to protect against an accidental rm -rf /. I also like to do frequent repository checks, but I want to keep the rw time as short as possible. Therefore, I like to do the repository check while the filesystem is mounted ro.

I think append-only repo mode is not really useful for me, as I am doing a local backup and also want to use prune.

The option I am suggesting should not ignore locks completely. It should only work for read locks and still should check and fail if there is already a write lock.
I would suggest a different option naming, as users should not confuse this with the break-lock command. Maybe something like --no-read-lock would be a good name.

[ThomasWaldmann]

A workaround might be to only have repo/data/ on a read-only filesystem while the rest is r/w.

[lxp]

Yeah, another workaround might be to use overlayfs on the whole repo during checks, but I didn't try it yet.

[zoot]

I have a similar issue.

Trying to access a remote borg repository for list, info, mount etc. fails when the remote repository resides on a ZFS snapshot.

I'm backing up using borg with daily pruning for encryption and remote ZFS snapshots for retention (and backup destruction protection).

The crucial reason for doing so, is to protect the backed up data on the remote backup server from an attacker who has access to the host doing the backing up. With read-only ZFS snapshots, the attacker is unable to delete remote backups.

Thus far, I've attempted borg list, which fails with:

Remote: borg.locking.LockFailed: ('/data/backup/.zfs/snapshot/daily_2017-05-23/home/lock.exclusive', "[Errno 30] Read-only file system: '/data/backup/.zfs/snapshot/daily_2017-05-23/home/lock.exclusive'")

I'm using this on rsync.net's service. It's not ideal having to request that their support staff manually restore onto a writable part of the filesystem for recovering historic backups; I'd like to be able to automate finding historic borg archives and mounting them.

@lxp wrote:

The option I am suggesting should not ignore locks completely. It should only work for read locks and still should check and fail if there is already a write lock.

I agree.

Surely borg doesn't require a read lock on a repository which resides on a read-only filesystem, because no other process can write to that repository?

Perhaps borg could detect whether the repository resides on a read-only filesystem and silently disable read locks, as they're superfluous in this case.

[zoot]

@lxp and @ThomasWaldmann, I have a relatively elegant workaround, for now.

It creates a directory on a writable filesystem, linking the snapshot's contents into it.

The below assumes you're only permitted to run remote ssh commands as user@server, without full shell access. This works with rsync.net

[user@client:~]

repo='foo'
snap='daily_2017-05-23'

ssh user@server "mkdir $repo.snap"

targets=$(ssh user@server "ls .zfs/snapshot/$snap/$repo")

for t in $targets; do
    ssh user@server "ln -s ../.zfs/snapshot/$snap/$repo/$t $repo.snap/$t"
done

borg list user@server:$repo.snap

[ThomasWaldmann]

hmm, add to FAQ?

[enkore]

rofs is it's own error code (EROFS); thus we could disregard locking iff it fails with this specific error.

Even though Borg requires repositories to not span file systems, this should be behind an explicit flag (e.g. --rofs as a common option; rejected/ignored for writing operations; passed via borg_cmd to remotes.).

check is a different story from just accessing the data and would be more complex.

[realityexists]

Perhaps borg could detect whether the repository resides on a read-only filesystem and silently disable read locks, as they're superfluous in this case.

I'd be wary of this, because a network filesystem may be read-only for you, but read-write for another user.

Probably better to just log a warning that the check may incorrectly report corruption if the repository is modified while it runs. (chkdsk on Windows says something like this, for example.)

[ThomasWaldmann]

Changed the title to be more generic. Also applies to list and extract. (from #2771)

[ghost]

I would also greatly appreciate this. I intentionally keep my backup drive readonly until my nightly borg job kicks off.

[awasu]

+1 for this. I also mount backups read-only, for safety. And if I archive a repo to DVD, it's completely unusable without copying everything to disk first, I can't even query it to find out what it is!

If somebody makes changes to a repo while it's being queried without a lock, at worst the query will show the wrong results, or crash, which is not good, but it won't damage the repo. Maybe abort and fail if a lock file suddenly appears during query processing...?

[ThomasWaldmann]

if you do some clever mounting or symlinking, you can have repo/ rw and repo/data/ ro.

[awasu]

if you do some clever mounting or symlinking, you can have repo/ rw and repo/data/ ro.

I think the point people are making is that it's a bit annoying to have to jump through these kind of hoops just to get a listing of what's in a repo, especially if you know that the repo is not in use. If that assumption turns out to be wrong, the query may not work properly, but presumably the repo won't be damaged. And you could semi-reliably detect if someone had taken a write lock, unbeknownst to you e.g. if a lock file suddenly appears, or the timestamp on the index file changes.

Mounting backups read-only is a Good Thing - I do it to protect myself from misteaks, other people seem to be doing it for security reasons - and mounting part of the repo writable doesn't address these. Somebody suggested using an overlay file system, but that's getting getting ridiculously complicated for something as simple as a "borg list" :-)

[kosli]

same problem over here.... having the backup repo always only mounted rw during backup.
e.g. a montoring system that checks every x minutes the repo to see when the latest backup happend won't work on the ro repo with borg list.

[davidgiven]

I'd like this as well --- my use case is wanting to burn backups onto read-only media. (And have just bought a bluray burner for precisely this purpose.)

The workaround is pretty trivial:

mount /dev/cdrom /mnt
cp -rs /mnt /tmp/backup
borg list /tmp/backup

...but it'd be nice not to have to do this. I don't like using non-standard workflows as part of my backup pipeline. If the flag were called something like --read-only-media and then actually checked to make sure the medium really was read only, then that ought to be reasonably robust (but of course the lock's so easy to circumvent that it's not really possible to prevent the user from shooting themselves in the foot).

(It could also suppress the 'The repository was previously located at...' warning in this case, too.)

[kosli]

Another option is to use an OverlayFS mounted over the read-only media. Haven't tested it yet, but should be straightforward. @davidgiven so you wouldn't need to copy data around, or create the symlinks.

[davidgiven]

That's enormously heavyweight, though. I really shouldn't have to do that.

[wzyboy]

I have a similar issue: host A mounted an NFS as read-write while host B mounted the same NFS as read-only. When I try to do borg list borg extract or other "read-only" operations, I cannot do it on host B. I use host B for recovery options so I do not want to remount,rw, to avoid accidental removal of backup data. (actually the user account on host B does not have sudo permission to remount the NFS)

And I think davidgiven has a more legit use case: checking / extracting archives from a read-only media.

The solution provided by zoot works for me. Thanks!

[Mayeu]

Here is another use case where no lock (or a configurable lock folder) would come handy.

I am trying to access a backup made on an external drive using an ext4 filesystem on macOS. I use ext4fuse to have read only access to the disk (it does not have any write access implemented).

But the previous link solution does not work in this case. When I access a file directly from my drive, it works:

$ cat /Volumes/Borg\ Backup/backup/README
This is a Borg Backup repository.
See https://borgbackup.readthedocs.io/

But if I try with a link, it fails:

$ ls -l borg-link
total 0
lrwxr-xr-x  1 m  staff  35 Dec 13 10:50 README -> /Volumes/Borg\ Backup/backup/README
...
$ cat borg-link/README
cat: borg-link/README: No such file or directory

There must be something weird with FUSE and symbolic links. I assume this would also be the case on any OS using this ext4 fuse implementation.

(So I am currently copying my terabyte of backups in another macOS friendly drive for now 😅.)

[abclution]

To add to this request, this is the scenario I wish I could accomplish.

• borg does the backup
• after completion, I zfs snapshot the borg repo
• I would like to be able to borg mount/list the zfs snapshoted (ro) repo
• This way borg can run again/continue running on the real rw repo as I have only mounted the snapshot of it.

I believe zoots workaround will also work for me but I have not tried it.

[LeBrad]

Another use that this request would allow is the mounting of a borg repository that is inside a mounted borg repository.

Making backups of backups seems like a valid usage for borg, especially when bundling the backups of multiple borg users, and where some are encrypted and others are not. The current behavior prevents borg mount from working on recursive repos.

Here's the familiar error:

borg mount /root/mounty/backup01-2019-04-05T18\:36\:14.362268/srv/storage/backups/nfs01/ /root/mountymounty/
Failed to create/acquire the lock /root/mounty/backup01-2019-04-05T18:36:14.362268/srv/storage/backups/nfs01/lock.exclusive ([Errno 30]
Read-only file system: '/root/mounty/backup01-2019-04-05T18:36:14.362268/srv/storage/backups/nfs01/lock.exclusive').
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1355, in do_mount
    return self._do_mount(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 139, in wrapper
    with repository:
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 189, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 392, in open
    self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait, kill_stale_locks=hostname_is_unique()).acquire()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 353, in acquire
    with self._lock:
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 114, in __enter__
    return self.acquire()
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 138, in acquire
    raise LockFailed(self.path, str(err)) from None
borg.locking.LockFailed: Failed to create/acquire the lock
/root/mounty/backup01-2019-04-05T18:36:14.362268/srv/storage/backups/nfs01/lock.exclusive ([Errno 30] Read-only file system:
'/root/mounty/backup01-2019-04-05T18:36:14.362268/srv/storage/backups/nfs01/lock.exclusive').

Platform: Linux backup01 4.19.0-2-amd64 #1 SMP Debian 4.19.16-1 (2019-01-17) x86_64
Linux: debian buster/sid
Borg: 1.1.9  Python: CPython 3.7.2+
PID: 15280  CWD: /root/mounty/backup01-2019-04-05T18:36:14.362268                                                                        
sys.argv: ['/usr/bin/borg', 'mount', '/root/mounty/backup01-2019-04-05T18:36:14.362268/srv/storage/backups/nfs01/',
'/root/mountymounty/']
SSH_ORIGINAL_COMMAND: None

[tuxcrafter]

Can we disable the need for locking on an already read only mount, it does not make sense to require the locking. (backups to cdroms, or other read-only media), in this case backups or other borg-backups.

[kristous]

I do monitoring of my borg backups based on borg list. While a backup is running my monitoring fails of course.

I would really appreciate a --no-lock (or something similar) for borg list. I don't think that a list could break anything.

[jms19]

Another one here. Like others I'd like to be able to monitor backups without giving myself the rights to write them. It's not a read-only filesystem in my case but still.

[Catscrash]

I would like that as well and would like to add one more usecase:
My Borg Repositories are cloned to backblaze. If I want to do a desaster recovery from backblaze, I can mount the bucket with rclone mount. The API Key for restoring purposes is read-only, which makes sense usually. Unfortunately I can't use borg mount or even borg list on this anymore, since it's read-only. It would really help having this, as I wouldn't have to download the complete repository to just restore one file

[wzyboy]

My Borg Repositories are cloned to backblaze. If I want to do a desaster recovery from backblaze, I can mount the bucket with rclone mount.

I have this exact use case. I had always bragging to my friends that I can use rclone mount + borg mount to restore any files I want from BackBlaze. But I never actually tried it. Now I read your comment, it occurred to me that I cannot double-mount. :joy:

[aleixq]

@lxp and @ThomasWaldmann, I have a relatively elegant workaround, for now.

It creates a directory on a writable filesystem, linking the snapshot's contents into it.

The below assumes you're only permitted to run remote ssh commands as user@server, without full shell access. This works with rsync.net

[user@client:~]

repo='foo'
snap='daily_2017-05-23'

ssh user@server "mkdir $repo.snap"

targets=$(ssh user@server "ls .zfs/snapshot/$snap/$repo")

for t in $targets; do
    ssh user@server "ln -s ../.zfs/snapshot/$snap/$repo/$t $repo.snap/$t"
done

borg list user@server:$repo.snap

Sorry for necrobumping I end up with similar solution to work with borg + Hetzner zfs storagebox snapshots.
In /mnt/backup-server/.zfs/snapshot/*/borg-backups there are different per app borg repos in ro mode, after running the script you'll end up with the repo but data dir suffixed by snapshot time in the specified rw directory.
To avoid issues as #4631 I end up copying all files but data from actual working borg repo. (don't know if it will be a problem...)

So it
:

#!/bin/bash
shopt -s extglob

BORG_ACTUAL_DIR=/mnt/backup-server/borg-backups/
BORG_SNAPS_LOCKABLE_DIR=/mnt/backup-rw
BORG_SNAPS_RO_DIRS=/mnt/backup-server/.zfs/snapshot/*/borg-backups

# Start removing past snapshot repos links.(not the data sure!)
# rm -r ${BORG_SNAPS_LOCKABLE_DIR}/* #commented to avoid copypaste problems 
# Get list of snapshots to take the snap Time
for snaptime_path in $BORG_SNAPS_RO_DIRS; do
  SNAPTIME=$(basename $(dirname $snaptime_path))
  # Get list of repos and copy in LOCKABLE DIR all but data, that is only linked 
  for repo in $(ls $snaptime_path); do
    SNAPNAME="${repo}__${SNAPTIME}"
    mkdir $BORG_SNAPS_LOCKABLE_DIR/$SNAPNAME
    # Doing this will end up with cache older than actual repo and security warning
    #cp -r $snaptime_path/$repo/!(data) $BORG_SNAPS_LOCKABLE_DIR/$SNAPNAME/
    # We copy from actual borg dir instead
    cp -r $BORG_ACTUAL_DIR/$repo/!(data) $BORG_SNAPS_LOCKABLE_DIR/$SNAPNAME/
    ln -s $snaptime_path/$repo/data $BORG_SNAPS_LOCKABLE_DIR/$SNAPNAME/data
    echo -e "repo ${BORG_SNAPS_LOCKABLE_DIR}/${SNAPNAME} done. To list repo do: \n\tborg list ${BORG_SNAPS_LOCKABLE_DIR}/${SNAPNAME}"
  done
done;

As example:

• rw borg repo : /mnt/backup-server/borg-backups/www
• ro borg repo snapshotted: /mnt/backup-server/.zfs/snapshot/*/borg-backups
• available lockable repos after script:
  
  
  
  • /mnt/backup-rw/www__2020-01-22T21-35-24
  
  
  • /mnt/backup-rw/www__2020-01-22T21-45-44
    
    And so on e.g. borg list /mnt/backup-rw/www__2020-01-22T21-45-44 can be done...
  
  

[gav-]

I can confirm that it is possible to copy the top-level files (hints, index, etc) from a read-only mount of a borg repo, and then symlink the read-only data directory.

Using this technique I was able to use rclone mount --read-only of wasabi S3 storage, set up a dir with the information above, and then borg mount that directory to browse files. It's a bit clumsy, and a little slow, but it's not an infeasible method to recover a few important files.

$ mkdir -p mnt/borg-ro mnt/borg-rw mnt/browse
$ rclone mount --read-only wasabi:borgbackup mnt/borg-ro
$ cp mnt/borg-ro/* mnt/borg-rw/
$ ln -s ../borg-ro/data mnt/borg-rw/
$ borg mount mnt/borg-rw mnt/browse

[ThomasWaldmann]

fixed by #5048.