Libretime: Problem: Setting up a backup routine needs doxing and automating

[hairmare]

Issue

The current Backing up the server has a hint that suggests you should do your backups using cron or similar. The page then recommends backing up to /src/airtime/database_backups which in most cases will be on the same volume the actual data is on. The Storage Backup part of the docs tells users that they need to backup /srv/airtime incrementally without giving much details. It doesn't tell us that we should also consider backing up the actual system.

The utils/airtime-backup.py script already automates some of this. It's not very configurable and is lacking documentation.

Proposed Solution

Update the docs with real backup and restore instructions and maybe automate optionally setting these up with the installer. I see the interactive installer asking us if we should set up a cron or systemd timer that calls a fixed airtime-backup.py.

Tasks

• [ ] add example for using cron and/or systemd timers to backup the database
• [ ] use a separate partition for all backup needs instead of /srv/airtime/...
• [ ] add example on how to incrementally backup the storage partition
• [ ] explain how to use the /systemstatus page at «Settings», «Status» to figure out how much space you will need for a backup
• [ ] add some info on full system backups (using amanda or a similar solution. Because you get the public folder for free as well (https://github.com/LibreTime/libretime/issues/184#issuecomment-297421473))
• [ ] document that all of these need to be manually tested at a regular interval for them to make sense
• [ ] Add some configuration possibilities to airtime-update
• [ ] Refactor airtime-update so it reads from the global config in /etc/airtime/airtime.conf as well as grabbing the needed audio file paths from the database (cc_music_dirs)
• [ ] Refactor airtime-update so it gets installed like other python apps
• [ ] Update installer so admin gets asked if he/she would like to start airtime-update on a cron/timer and do so

[hairmare]

@Robby262 I updated this to cover your points from https://github.com/LibreTime/libretime/issues/184#issuecomment-297417573

[Robbt]

So going back to this.
We have a very basic python based backup script that backs up the database and file storage into a new directory called airtime_backup.

I think that what would be ideal would be something that specifically offers off-site backup and some kind of versioning. At the very least we need to provide simple instructions for how to configure the backup to point to an external hard drive or storage. rsync would be better than simply copying the media files in the library as done here - https://github.com/LibreTime/libretime/blob/e584f859feb98412f9bff3e9b588e7079d987bb7/utils/airtime-backup.py#L44-L48 because it would waste disk cycles and time rewriting data.

In an ideal world the settings page would have some kind of backup system integration so that the owner/admin could verify their backups are occurring regularly. If they are left running in the background unmonitored then it could be a disaster if they ever need them.

So backups of the code would be nice but that can essentially be handled by git for the most part, until people start hacking their code on their own which I don't think we can really easily support. On the other-hand we can easily backup their code directories since the size is rather small compared to the media files.

The database would also be essential to backup. Backing it up in its raw form and/or as SQL statements is simple just via pg_dump as done here in the existing backup utility - https://github.com/LibreTime/libretime/blob/e584f859feb98412f9bff3e9b588e7079d987bb7/utils/airtime-backup.py#L35-L39

So I think the primary goal for us should be to make it easy for people to setup their own backup solution. We could develop utilities that integrate with various 3rd party services or even just utilize common protocols such as rsync over ssh or even FTP. The primary goal should be that every user running LibreTime is made aware and given the opportunity to backup.

I can imagine a robust way to do this would be to code another python service that manages the backups and can be tracked via the status page and setup via the web-based install. This is also significantly more work than writing up concise documentation on how to manually create off-site backups etc.

Another idea I had was the idea of maintaining a hot-spare via backup. Sort of mirroring LibreTime into a system so that if your main machine goes down you can easily spin up a new one from the archives. It would also handle the restoration aspect of building a backup system. If someone were to offer 3rd party services for LibreTime off-site backup, monitoring and restoration might be useful. It would be great if this was tied into the core and easily duplicated by any stations out there that wanted to easily deploy LibreTime.

[paddatrapper]

I think another always-running python service is overkill. I would recommend file-based local backups scheduled through cron/systemd timer that the admin can then backup off-site according to their backup policy.

I tend to use other solutions for hot-swap and backup (zfs-send, rsync, etc), so I think building the data transfer into LT would limit the flexibility of tying LT into existing infrastructure. The UNIX philosophy of doing one thing and doing it well comes to mind... I do think adding documentation pointing to various backup solutions and supporting optional integration is a good idea, but by default do backup to a local directory that the admin can use as they will.

[zklosko]

I agree with @paddatrapper. I think a cron + rsync option would be much easier. I don't think it should be automated with the installer but I'd be okay with adding a tutorial to our docs section so others can manually configure it.

I guess bonus points if we add the cron + rsync option to Libretime settings, so it can be configured using the interface.

[zklosko]

Looking in the /srv/airtime directory on our Libretime installation, I noticed files were organized into folders by numbers. What do the numbers represent? Track types?

$ ls /srv/airtime
'11' '12' '2' '5' '7' '8'

Update: my latest guess is user IDs

[zklosko]

@hairmare I don't think an extra partition on the same system is going to provide adequate protection for our users. Doing so only protects them against a software error. Libretime is incredibly good for alpha software; users more likely with encounter hardware errors (especially drive errors). Backing up should always be to a separate system, or separate drive at the very least.

That being said, here's a quick-and-dirty backup script using rsync:

# Replace /backup with backup folder location!
# Used for backing up the media library
sudo rsync -aE --delete /srv/airtime/stor/ /backup

# Used for backing up the database
sudo -u postgres pg_dumpall | gzip -c > libretime-db-backup.gz
sudo mv libretime-db-backup.gz /backup

# Write date/time of backup to file
date >> /backup/datelog.txt

We can add a quick reference to CRON in the docs.

0 0 1 * * root /home/libretime/libretime/backup.sh

[paddatrapper]

date >> /backup/datelog.txt

Assuming that /backup/ is only root writeable, this will fail. I'd recommend putting everything into a script and running that as root/sudo instead. Other than that small nitpick, it looks good

[zklosko]

At the moment, the script to back up Libretime is perfectly functional; the same can't be said for a restore script. In order for a restore script to work, we would need files to be copied back and to have the databases and services set up as part of the script. Simply importing the database causes an error.

[zklosko]

@hairmare We currently have a backup script and instructions for using it as part of the main repo. Is there anything else we need to complete this issue?