Borg: allow use of a config file for different backup jobs

Created on 20 Oct 2015 · 23Comments · Source: borgbackup/borg

i think it would be worthwhile for borg to support reading config files. as the create commandline options get more and more complicated, the shell pipeline grows and grows. it is unlikely to hit the ARG_MAX limit (around 2MB on my system) but still, it seems a little counter-intuitive to have very long command lines like this.

i feel it would help in sharing scripts and best practices to allow borg to use config files. we could ship sample configs for home directories, full system backups, etc.

i would personnally use it for different backup targets: onsite, offsite, and different purge frequencies and so on.

when developping bup-cron, i fixed this by sucking in the config file through the commandline parser, simply. this means that:

borg create --stats foo::bar baz

becomes the config file:

repository=foo::bar
path=baz

Right now i use the following shell wrapper to call borg:

#! /bin/sh -e

# this script should not exist. there should instead be a borg cron
# subcommand that reads a freakin config file.
#
# that would need to be implemented in borg/archiver.py, with a hook
# in run() and a def do_cron() that would call do_create() and others.

# features missing from bup-cron:
# snapshots
# parity
# syslog
# timing

REPOSITORY=/media/anarcat/BOOK/borg

# check for more recent bup in /usr/local/
PATH=$PATH:/usr/local/bin

# custom tag for manual backups
tag=$1

# this is how you bootstrap the repo
# borg init $REPOSITORY # todo: -e keyfile?

if [ ! -d $REPOSITORY ]; then
  mail -s "backups failed" root <<EOF
$REPOSITORY not mounted, nightly backups failed

you're own you're own buddy.

email fired from $0
EOF
else
  borg create --do-not-cross-mountpoints        \
      --stats                                    \
      $REPOSITORY::$(hostname)-$(date +%Y-%m-%d)$tag \
      / /boot /usr /var /home                    \
      --exclude-caches                           \
      --exclude "/home/*/.cache"                 \
      --exclude "*/.Trash-*"                     \
      --exclude "*/[Cc]ache/*"                   \
      --exclude "*/.bitcoin/blocks/*"            \
      --exclude "*.vmdk"                         \
      --exclude "/tmp/*"                         \
      --exclude "*/build-area/*"                 \
      --exclude "/proc/*"                        \
      --exclude "/dev/*"                         \
      --exclude "/sys/*"                         \
      --exclude "/var/cache/*"                   \
      --exclude "/var/tmp/*"                     \
      --exclude "/var/log/*"
fi

as the comments describe, i was considering implementing this as a bup cron command that would be designed to run under cron. it could also purge old backups - so it would combine create and purge in one single pass.

the way i did this is with fromfile_prefix_chars (i used fromfile_prefix_chars='@') and then inspect files in standard locations (by injecting @foo.conf in the commandline at startup). the config files checked were /etc/bup-cron.conf, ~/.bup-cron.conf and ~/.config/bup-cron.conf.

Config file locations could be:

/etc/borg.conf
~/.config/borg.conf
~/.borg/config

Or maybe we could define multiple "jobs" that would simply be ran in sequence in directories:

/etc/borg/<jobs>.conf
~/.config/borg/<jobs>.conf

... etc.

enhancement

Source

anarcat

Most helpful comment

potential duplication of code (or implementation of features I don't need) in other projects is nothing I care much about.

but integrating stuff into borg that is usually located outside of it and adding config file parsing would cause a lot duplication and additional work here. if you add or change a feature, you currently have to edit 1 place (argparse related code). With additional config file, you would have to change 2 places.

also, if you think what would be needed to integrate everybody's backup related scripting needs into borg + configfile, you'ld be just reinventing shell scripting and atticmatic and what else and integrate everything into borg. this would just lead to an endless stream of feature requests until everybody's need are integrated (which would otherwise be a non-issue because people would just write a script as needed).

so, in short, I am still rejecting this as out of scope and even harmful.

ThomasWaldmann on 20 Oct 2015

👍3

All 23 comments

this is https://github.com/jborg/attic/issues/18

anarcat on 20 Oct 2015

and yes, i know about atticmattic but it seems overkill and duplication of code to have a separate software to handle what is basically a fundamental aspect of borg, which is configuration of behaviour...

anarcat on 20 Oct 2015

potential duplication of code (or implementation of features I don't need) in other projects is nothing I care much about.

so, in short, I am still rejecting this as out of scope and even harmful.

ThomasWaldmann on 20 Oct 2015

👍3

if you add or change a feature, you currently have to edit 1 place (argparse related code). With additional config file, you would have to change 2 places.

no... the way i suggest this be implemented does _not_ require additional changes when new options are introduced. the options in the config file would be the same as on the command line and you'd still only have to change the argparse builder when you introduce new options. that's the whole point of fromfile_prefix_chars.

and people can still write their own shell scripts as well, of course. enabling config files doesn't force people to use them.

anarcat on 20 Oct 2015

so if it is the same in the config file as on the commandline, what's the point?

ThomasWaldmann on 20 Oct 2015

the point is that non-technical users don't need to learn the shell and they can copy-paste examples from others more easily. having a config file enables shared configs more easily, in my opinion, as instead of running arbitrary codes from strangers, you have a limited set of things the config can do, controlled by borg.

it would also allow one command to both purge and create.

i wouldn't want people to run random shell scripts as root, as a general practice. copy-pasting configs seems like a better idea.

anarcat on 20 Oct 2015

OK, I agree on creating a 1:1 equivalence between commandline and config file (using the same "verbs") if it does not require maintaining code at 2 places.

I don't agree on adding any additional functionality to be triggered via the config file (like pre or post commands, like "scripting" multiple commands, doing mounts or snapshots, cron-like stuff or whatever).

Also, this is not for 0.28.

ThomasWaldmann on 20 Oct 2015

BTW, something much more needed than that (IMHO) is a config file for defaults (if one is not happy with the builtin ones).

ThomasWaldmann on 20 Oct 2015

👍1

how about just doing prune alongside create? can i get that in? :)

i agree that snapshots and mounts may be feature creep for now (although we have #225 for par2, for example)...

anarcat on 21 Oct 2015

no. :) and if you look at the faster cache sync PR, you'll see one reason why.

when sharing a repo, if you prune often (and it really deletes an archive), you will have expensive full cache rebuilds often. thus, create should happen often and prune only now and then to get better performance most of the time.

ThomasWaldmann on 21 Oct 2015

i did look at it, but i am a little confused by that code and couldn't review it properly, so i don't understand the reference. :)

anarcat on 21 Oct 2015

There is (at least) one drop in replacement for argparse, called ConfigArgParse, that adds additional support for setting flags via env vars and config files. I would expect it to be properly maintained in the future since the _Let's Encrypt_ client depends on it. It has build-in support for defining default config files and user defined config files.

sophie-h on 17 Feb 2016

@qua-bla what does that give us over the builtin fromfile argument to argparse?

environment variables are interesting - but we are already using those completely separately - not sure about merging the two...

anarcat on 17 Feb 2016

Only a few but imho important features @anarcat

Defining default_config_files=['/etc/borgbackup/borgbackup.conf', '~/.config/borgbackup/borgbackup.conf'] which also enables to set system wide defaults as mentioned by @ThomasWaldmann
Use of proper --config option that also appears in the help output (instead of @ or other fromfile_prefix_chars)
Usual config format that only allows named arguments (instead of --option=)

Not sure, but the support for environment variables might simplify to bash script around custom needs?

sophie-h on 17 Feb 2016

I have to admit that it there are few problems

ConfigArgParse has simply no support for positional arguments (bw2/ConfigArgParse#48). So you cannot set archive and paths via the config.
The set of options that the subparsers expect are different. ConfigArgParse (and argparse's _@_) is not happy with unknown options. If I put exclude into a global config, I cannot run init without an error. Partially, config files for sub commands make sense (e.g. serve) and they are supported. However, I'm not sure if config files per sub command + maybe one overall config is sane.

sophie-h on 18 Feb 2016

close/wontfix?

enkore on 19 Jul 2016

Some ideas maybe make sense, some other maybe do not.
I'ld have to actually see the code and its consequences.

ThomasWaldmann on 19 Jul 2016

Pulling in a dependency just for this doesn't seem feasible, especially since "ConfigArgParse" isn't widely packaged. If it's possible without extra dependency...

enkore on 19 Jul 2016

Just for reference: I hacked together a crude shell script for that purpose. It is pretty ugly right now, but thought I might share it anyway: https://github.com/dragetd/borg_auto