Borg: evaluate Lars's vision for backups

[anarcat]

i keep on coming back to this article from Obnam's author, Lars Wirzenius. He makes a very strong points that backups should be way more transparent for our users and while i understand this may not be our priority at first, it would be an interesting end goal for borg.

The gist of it:

• #325: continuous backups
• automatic detection and configuration of external drives
• automatic backup configuration ("this is a single user system, so i'll just backup /home and /etc")
• #314: GUI, with notification area for start/stop of backups ("it is now safe to remove your USB drive") and restore interface (probably just help the user mount the drive and drag-and-drop for restores)
• #315: config files for portable configs that can be sent by a third party without enabling arbitrary code execution
• #120: asymetric crypto, so the decryption key can be stored offline
• allow restore during operating system install

Stuff we already have:

• remote server support (although that is only SSH for now, see #102 for S3, for example)
• encrypted backups
• automatic cache exclusions (provided they follow the CACHEDIR.TAG standard)
• efficient large file support
• FUSE mounts

thoughts?

[rumpelsepp]

automatic detection and configuration of external drives

That one is _really_ important. I backup regularly on an external HDD and it might not be connected every time when the cron job fires.

[anarcat]

@rumpelsepp i'd appreciate if you would create an issue detailing your use case, then. :)

[ThomasWaldmann]

Quite some of that is just out-of-scope for borgbackup (or any non-feature-creep backup tool).

I've use obnam for (very) short, but neither performance (about a magnitude slower than attic back then) nor its crypto usage (forking a gpg once per file to do symmetric crypto) nor the contribution workflow (send a patch to a mailing list) impressed me. That's how I found attic. :)

So, while I like to look at alternative approaches, I am a bit selective when adopting them.

[anarcat]

The summary here isn't a description of obnam, or isn't necessarily related to obnam, other than the original blog post happens to have been written by obnam's author.

I don't know if all those features are out of scope for borg: 4 of the 7 missing features already have issues in our own issue queue. Furthermore, while some things like "automatic configuration of external drive" may not belong in the borg codebase itself, it could be part of a debian package that would ship udev shims. other parts, like #325 (continuous backups) do need borg's collaboration and would be harder or more wasteful to do with an external tool.

i think it's still useful to evaluate those ideas to provide a good and smooth user experience. of course our priority is to make the backend solid, but i like to think of this as a long term project. :)

[ThomasWaldmann]

just as a note: I'ld appreciate if all topics that we consider for borg are taken to separate issues and discussed there and removed from the list, so we can close this issue when we have decided.

[rumpelsepp]

i'd appreciate if you would create an issue detailing your use case, then. :)

My use case is, that I backup on a external hdd and quite frequently I have problems with stale lockfiles. When these files are present, borg hangs forever when I retrigger the backup; that sucks… :/

[ThomasWaldmann]

@rumpelsepp this is completely unrelated to THIS issue. See #136, #157, #210 and #285.

[rumpelsepp]

this is completely unrelated to THIS issue

Yeah, you are right... Sorry for that.

[anarcat]

@rumpelsepp i still believe your use case would be useful to describe:

automatic detection and configuration of external drives

[RonnyPfannschmidt]

IMHO a tool for HD config/triggering should be distinct from the actual backup tool

Separation of concerns is critical

[ThomasWaldmann]

I'll close this issue as too unfocussed soon - if you'ld like to keep something of it, create individual, focussed and non-vague tickets.

[enkore]

automatic detection and configuration of external drives

That one is really important. I backup regularly on an external HDD and it might not be connected every time when the cron job fires.

I do this with udev and a systemd unit. If you always use the same physical port (same USB port, or you have a SATA disk frame) you can create a udev rule that starts a systemd unit (or just runs a script).

$ cat /etc/backups/40-backup.rules  # hardlinked in the right place
ACTION=="add", SUBSYSTEM=="block", DEVPATH=="/devices/pci0000:00/0000:00:11.0/ata1/host0/target0:0:0/0:0:0:0/block/*", TAG+="systemd", ENV{SYSTEMD_WANTS}="automatic-backup.service"

The long "/devices/pci0000:00/0000:00:11.0/ata1/host0/target0:0:0/0:0:0:0/block/*" string can be constructed from /sys, but it's easier to just use udevadm monitor and plug the drive in and to just copy it.

The script can then just mount the drive based on a similar hardware path (/dev/disk/by-path/…-partX) and do the backup. It can also automatically unmount and detach (spin-down, hdparm -Y) the drive when it's finished.

[enkore]

If you can't identify backup drives by physical path, you could still automate it by having udev match against the UUID (or label) of the file system.

[ThomasWaldmann]

btw, we have an "extra" repo now for stuff that is related somehow, but should not go into borg repo.

[enkore]

Maybe deployment.rst might be reorganized into a section with, well, basically "guides" to explain common setups?

[ThomasWaldmann]

I'ld rather keep such stuff separate. There are many different setups, even many common ones.

[enkore]

Maybe deployment.rst might be reorganized into a section with, well, basically "guides" to explain common setups?

Quick note for people from the future: I've done this some time ago. https://borgbackup.readthedocs.io/en/latest/deployment.html