Mailcow-dockerized: Improve the backup script (incremental etc)

[klausenbusk]

The current backup script create a full backup on every run, which isn't very space effective.

We should consider providing a backup script which provide some sort of incremental backup.

FWIW, I have created the following script which use Borg. Borg dedups everything, so it is very effective.

#!/bin/bash
set -o nounset -o errexit
REPO="$PWD/borg"
COMPRESSION="zlib"

if [ ! -d "${REPO}" ]; then
    borg init --encryption=none "${REPO}"
fi

backup () {
    VOLUME="$(docker volume ls -qf name="${1}")"
    MOUNTPOINT="$(docker volume inspect --format '{{ .Mountpoint }}' "${VOLUME}")"
    borg create --info --list --filter=AME --stats --compression "${COMPRESSION}" "${REPO}::${1}-{now}" "${MOUNTPOINT}"
}

backup "vmail-vol-1"

docker exec $(docker ps -qf name=redis-mailcow) redis-cli save
backup "redis-vol-1"
backup "rspamd-vol-1"
backup "postfix-vol-1"
backup "mysql-vol-1"

# Do I need to backup this?
backup "crypt-vol-1"
backup "dkim-vol-1"

[catcun]

Borg is an awesome backup solution. What bothers me is consistency. Maildir consistency is mentioned in #1031, but I wonder about the other parts like mysql.
Perhaps the best thing would be a volume snapshot before backing up, I don't know. So far, my servers are on VMs and I am backing them up at the hypervisor level

[klausenbusk]

but I wonder about the other parts like mysql.

MySQL should be able to recover from that.

Perhaps the best thing would be a volume snapshot before backing up

That is the ideal solution, but if you use some sort of cloud provider that isn't really a option, you would need LVM/ZFS or something similar and changing the filesystem is not trivial.

[andryyy]

I doubt MySQL can always and easily recover from that, it may require further steps. I have been in such scenarios too often. Dovecot doesn’t care that much, a broken index is always self healing.

External software shouldn’t be required by a simple default backup script.

[klausenbusk]

FWIT, I use the following script for the time being to take backup

#!/bin/bash
set -o nounset -o errexit

script='export BORG_PASSPHRASE="<secret>"
REPO="user@server:borg/mail"
COMPRESSION="zlib"
while true; do
    borg create --info --list --filter=AME --stats --compression "\${COMPRESSION}" "\${REPO}::mailcow-{now}" /data
    sleep 3600
done'
echo -e "$script"

# I should probably just use a Dockerfile for this part.
docker rm -f borg || true
# Assume we are running from $HOME and copy .ssh to the image.
docker run --name borg -v "${PWD}/.ssh:/foo:ro" alpine sh -c "apk add --no-cache borgbackup openssh && cp -r /foo \$HOME/.ssh && echo -e \"$script\">/backup.sh"
docker commit borg borg
docker rm borg

docker run -d --name borg --restart always -t -i $(sudo docker volume ls --format="-v {{.Name}}:/data/{{.Name}}:ro" | grep -- "^-v mailcow") borg sh /backup.sh

First it create a borg image, then a container with all the mailcow volumes attached and run borg from that container hourly (pushing to a remote server). I have only used it for < 24 hours at the moment, but it seems to work, every backup add ~ 2-5MB. I'm not sure about the consistency and need to test recovery, but besides that it seems to work :)

[protree]

@klausenbusk Can you share your experience please? Did you test recovery?

[klausenbusk]

@klausenbusk Can you share your experience please? Did you test recovery?

It has been running now for over 1 month without any issue, I would expect that recovery just work, but I haven't tested it (yet).

Anything in particular you want answer to?

[protree]

Thank you. I am mostly concerned about recovery, which you didn‘t test so I might test it for myself. Thanks anyway :)

Am 15.04.2018 um 11:16 schrieb Kristian Klausen [e-mail]:

@klausenbusk Can you share your experience please? Did you test recovery?

It has been running now for over 1 month without any issue, I would expect that recovery just work, but I haven't tested it (yet).

Anything in particular you want answer to?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[andrestaffe]

@protree Did you already tested the recovery?
Thanks in advance

[protree]

No, I decided to use the script provided by default. My installation is rather small and I have enough space for backups so I decided to keep it simple :-)

Am 29.07.2018 um 13:53 schrieb axtion [e-mail]:

@protree Did you already tested the recovery?
Thanks in advance

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[thomas126]

Looks really interesting mate. Can you tell some more about your backup script? Is it capable of just backing up vmail every hour without consistency problems? Isn't that the part which take up the most disk space?

[BlamKiwi]

I modified the existing script to use uncompressed archives. That way borg can still apply its own deduplication and zstd compression.

[ntimo]

I use a Restic container that backs up vmail, crypt, rspamd and postfix. And then using the mailcow script I backup mysql and redis this way I can dedplicate and have incremental hourly backups of my mails but can still restore the database if needed :)

the Restic docs can be found at https://restic.readthedocs.io/en/stable/ . And Restic also encrypts the data before uploading it to your backup storage so, this way your backups re also secure.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michacassola]

@ntimo I am trying to figure out a good way to backup my mailcow server also.
Can you provide more details please or point me in the direction of a tutorial? Thank you very much in advance!

[abzsol]

I use a Restic container that backs up vmail, crypt, rspamd and postfix. And then using the mailcow script I backup mysql and redis this way I can dedplicate and have incremental hourly backups of my mails but can still restore the database if needed :)

the Restic docs can be found at https://restic.readthedocs.io/en/stable/ . And Restic also encrypts the data before uploading it to your backup storage so, this way your backups re also secure.
@ntimo
Can you provide more info?
Thanks

[catcun]

I am using autorestic https://cupcakearmy.github.io/autorestic/
I backup separately /opt/mailcow-dockerized and /var/lib/docker/volumes and
instruct autorestic to stop/start the containers before/after backing up
the volumes. It has been working flawlessly but I only backup once at night
for disaster recovery so stopping the containers is not a problem. Anyway,
I see that the backups are so quick that stopping containers on working
hours may not be a problem

On Thu, Oct 1, 2020 at 11:44 AM abzsol [e-mail] wrote:

I use a Restic container that backs up vmail, crypt, rspamd and postfix.
And then using the mailcow script I backup mysql and redis this way I can
dedplicate and have incremental hourly backups of my mails but can still
restore the database if needed :)

the Restic docs can be found at https://restic.readthedocs.io/en/stable/
. And Restic also encrypts the data before uploading it to your backup
storage so, this way your backups re also secure.

Can you provide more info?
Thanks

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mailcow/mailcow-dockerized/issues/1032#issuecomment-702260877,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGTKIRT25UTO3QNSC4FBGSDSISWVZANCNFSM4EQL4UZA
.

[shiz0]

@michacassola
@abzsol
I use borg to create incremental hourly encrypted backups of my cow. I've created a docker image for that:
https://github.com/shiz0/mailcowdockerized_borgbackup_unofficial
Feel free to take a look.
It needs versioning of components and a proper readme, but I am already using it in production, so I'd say it works. :-)

[abzsol]

@michacassola
@abzsol
I use borg to create incremental hourly encrypted backups of my cow. I've created a docker image for that:
https://github.com/shiz0/mailcowdockerized_borgbackup_unofficial
Feel free to take a look.
It needs versioning of components and a proper readme, but I am already using it in production, so I'd say it works. :-)

@shiz0 this is the first option i tried, but i'm missing how borg works. created a trial accoung on borgbase with 10GB to test but i kept getting error on connecting. I will retry.

Thanks

[andryyy]

If you like you can add it to the official repo. A restore desc would be cool. I also still need to check your selinux stuff. :( sorry

[shiz0]

Yes, looking forward to do that! I will move it once i worked it over, especially the versioning should be done I think, instead of using latest on all the source stuff. Also I wanted to automate image rebuilds (build and push action maybe?) but did not have time for either so far :( . There should be a proper restore howto in the readme at minimum, maybe even automated, at least partly.