Flynn: Backup system

[titanous]

Currently we have a basic backup command that pulls a logical backup of a postgres db using the flynn CLI, but we should have a more robust system.

Some things to think about:

• WAL logs allow point-in-time-recovery, which would be nice (see: WAL-E).
• Not all environments have a blobstore like S3 available, support for some of NFS/CIFS/FTP/SCP, etc. would be useful.
• Scheduling of backups.
• Encryption is necessary for storing data on external systems.
• The system should be generic enough to handle other data appliances.

[KernCheh]

:+1: Would love to see the scheduled backups feature :)

[philiplb]

May I throw in Amazon S3 as backup storage? This would be exactly what I would love to use here. :)

[onnimonni]

I'm developing a solution for this which would just encrypt the results of $ flynn cluster backup and throw them to s3. If the spec later sounds okay I will create PR later into flynn. This doesn't implement everything but it would be a start.

Implementation

The backup process is called backy and it will implement the same style envs as blobstore:

$ flynn -a backy env
BACKEND_S3MAIN=backend=s3 region=eu-west-1 bucket=flynn-backup access_key_id=XXXXXX secret_access_key=YYYYYYYYY
DEFAULT_BACKEND=s3main
BACKUP_SCHEDULE='* */8 * * *'
BACKUP_ENCRYPTION=type=symmetric symmetric=ZZZZZZZZZZZ

BACKUP_SCHEDULE will use cron time format.
BACKUP_ENCRYPTION is one of none, symmetric, public-key. And it needs to include either symmetric or private_key parameter
BACKUP_ADMINS is the list of extra gpg public keys which can open the backup when public-key option is used.

I would recommend us to use same permissions as in https://flynn.io/docs/production#amazon-s3 but without "s3:DeleteObject" permission. This way the process can only read and write backups but not delete them. This helps if the cluster gets hacked so backups won't have to be compromised.

We can add versioning and expiration into the requirements so that backups will get deleted automatically eventually. User needs to set these details in the bucket.

Only s3 implements this append-only permission so I wouldn't recommend Google or Azure for now but they will get implemented too.

I will use gpg from https://github.com/maxwellhealth/go-gpg for encryption. Symmetric key (password) will be the default option but list of sysadmin public gpg keys can be used too.

I hoping to use the storage functions from blobstore:

import (
       ...
    "github.com/flynn/flynn/blobstore/backend"
)

[titanous]

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.