Nextcloud-snap: Migration between two servers

[QuentinBellus]

It could be interesting to have a way to migrate nextcloud (snap) from one server to another.
This can be needed in different cases:

• hardware failure (especially storage)
• moving from cloud to a dedicated server
• migrating to a server with more cpu/memory/storage

The official migration procedure is not that complicated, but still requires to dump/restore the database, as well as doing a rsync of logs, user data and configuration files.

Assuming the old server has ssh access to the new one, it should be possible to create a dedicated command that would:

1. Enable the maintenance mode
2. Connect to the new server
3. Install the same snap of nextcloud
4. Import the database dump
5. Perform the various rsyncs required (data, config, logs)

[r4co0n]

This seems to be the same request as has been discussed here: #413. I am personally also interested in getting this to work with easy commands, but we will need maintenance mode for this, which is currently acting up, see #486.

[QuentinBellus]

Indeed it seems to be pretty much the same, I see that doing a rsync of the whole nextcloud snap folder should do the trick, it might be even easier than expected.

Looks like the maintenance mode is only missing a restart of php-fpm, probably not a big issue

[r4co0n]

See also #450, it is a better idea to migrate the database via mysql dump and restore.
And #185, which seems to be the original issue for this, where @kyrofa lists what files we need to backup, it would be wasteful to copy over much more than we need.

[kyrofa]

Yeah, dupe of #185. I agree this would be great to have. I even have a good start on it if someone wants to collaborate on it, let me push it up.

[kyrofa]

I suggest we move discussion over there, though.

[stone212]

@r4co0n You seemed to disagree with the earlier point that "Indeed it seems to be pretty much the same, I see that doing a rsync of the whole nextcloud snap folder should do the trick, it might be even easier than expected."

Is that because that method won't work or because it requires you to copy more than is strictly necessary?

Also, what folder is the "nextcloud snap folder"? Is that /etc/snap/nextcloud/current?

[r4co0n]

@stone212, the nextcloud snap folder is /var/snap/nextcloud/current (you had /etc/...), which is a symlink pointing to /var/snap/nextcloud/<someversion>/, the current version of the snap in use. See also this and what @kyrofa wrote here on what to backup - Note that there is no recommendation there to backup all of current.

My objections against using rsync stem from the fact that almost any DBS comes with dump/restore commands, which avoid having to do arcane magic in some corner-cases, e.g. switching architectures as mentioned here. As the documentation of MySQL recommends using this to backup databases, it seems the way to go.

Unfortunately, I think we are missing something like snap run nextcloud.mysqlrestore, snap run nextcloud.mysqldump is implemented. Probably simple dump/restore commands for the snap are the preferable option here, as there might be further house-keeping to do after restore, e.g. to make sure the files in db and on disk are the same, rebuild some caches and whatnot.

After all, working on or a reaction on #185 seems the way to go.

[stone212]

@r4co0n I don't see the connection with the first link but the second link says you want to back up:

    MySQL database
    MySQL root.ini (assuming you're dumping everything in the above step), in $SNAP_DATA/mysql
    SSL setup (private and public keys, in $SNAP_DATA/certs)
    Nextcloud config (in $SNAP_DATA/nextcloud/config)
    Nextcloud apps (in $SNAP_DATA/nextcloud/extra-apps)
    Data, optional (in $SNAP_COMMON/nextcloud)

But that means we're going in circles. Because the two challenges/questions that I am asking about are:

1. How do you access the mySQL database of a snap?

2. What are the $SNAP_DATA and $SNAP_COMMON directories. (The command "echo $SNAP_DATA" from the server command line shows no value)

I don't need this to be automatic. I can write a script easily. But I need to know what should go into the script.

The other link you said had this useful information:

There is a mysqldump command as well as a mysql-client command in the snap which should come in handy.

But it's not so useful that I can use it. What does it mean to be a "command... in the snap"? The only way I know to access mySQL is the normal way but the snap is self-contained right? There's some secret key I'm missing and once I know if I think all of this will make sense and I will post info about the script I write for others to use.

[stone212]

@r4co0n Okay now I see that your first link answers my later question. Now the question is: how do I get items from inside the snap exported to the server so I can copy them to a backup server. But at least you significantly helped me take a few steps.

[r4co0n]

Well, after all, 'everything is a file'. So you can grab the relevant files from the directories described above as you would with any other files.

In order to not do the backup while Nextcloud is working on your data, you should put it in maintenance mode first, with:

snap run nextcloud.occ maintenance:mode --on

Unfortunately, this is currently not working, see bug #486.

That's a "command in the snap", here occ. There is also:

snap run nextcloud.mysqldump
snap run nextcloud.mysql-client

which you can use to dump and restore your database, respectively. This is what they do.

All of this could be wrapped up in two new commands:

snap run nextcloud.backup
snap run nextcloud.restore

If you plan on writing a script for this, please share it, there's a good chance it can be adapted to be used for the commands. Bonus points if you keep this second use-case in mind while writing the script :)

[stone212]

@r4co0n

If you plan on writing a script for this, please share it,

Yes and I would be happy to. But I need to know what needs to be backed up first. It is taking longer to get a straight answer on what to back up then I have to devote to this. I'll come back in another month when I have time.

[r4co0n]

Okay, so you want some more spoon-feeding. (scnr)

You quoted:

@kyrofa wrote:

MySQL database
MySQL root.ini (assuming you're dumping everything in the above step), in $SNAP_DATA/mysql
SSL setup (private and public keys, in $SNAP_DATA/certs)
Nextcloud config (in $SNAP_DATA/nextcloud/config)
Nextcloud apps (in $SNAP_DATA/nextcloud/extra-apps)
Data, optional (in $SNAP_COMMON/nextcloud)

I linked to https://docs.snapcraft.io/reference/env, which says:

SNAP_DATA
Directory for system data of a snap.

This directory is owned and writable by root and is meant to be used by background applications (daemons, services). Unlike SNAP_COMMON this directory is backed up and restored across snap refresh and snap revert operations.

Typical value /var/snap/hello-world/27

Note that this is what is linked to from /var/snap/hello-world/current/ as I previously explained, except that our snap is called nextcloud, not hello-world.

# ls -l /var/snap/nextcloud/current
lrwxrwxrwx 1 root root 4 May  8 22:40 /var/snap/nextcloud/current -> 6916

and

SNAP_COMMON

Directory for system data that is common across revisions of a snap.

This directory is owned and writable by root and is meant to be used by background applications (daemons, services). Unlike SNAP_DATA this directory does is not backed up and restored across snap refresh and revert operations.

Typical value: /var/snap/hello-world/common

I explained how to backup the database:

There is also:

snap run nextcloud.mysqldump
snap run nextcloud.mysql-client

which you can use to dump and restore your database, respectively.

So what exactly are you missing? I am really trying to help you, but expect you to do your work, and that includes reading and getting to know the stuff you use, yourself.

And yes, this support-work also consumes my time.

[stone212]

@r4co0n You misunderstood what I said but I understand why. I was not expecting you to write this (but I thank you!) I was just staying the obvious, that I would have to compile this information. I knew that I would not be able to do it for several weeks because I have my own dev projects I am working on so I probably sounded pretty needy.

Anyway thank you. When I have time to look at what you read and convert it unto usable data I will do so. Unfortunately it will be many weeks or months. So I desperately hope that my NextCloud server stays alive. :)

[r4co0n]

@stone212, if you stop the snap, tar up /var/snap/nextcloud/common and what /var/snap/nextcloud/current points to, you should be able to restore by throwing away the directories and restoring from the tar and setting an appropriate symlink. I haven't tested this, as I am backing up the whole machine, but I see no reason why this shouldn't work.

@kyrofa might have further insight if this will work _just like this_.

[stone212]

@r4co0n I've finally had time to read your post from 20 days ago. That wasn't spoon feeding. It wasn't even helpful. What you did was repeat the incomplete information that caused me (and other people) to ask what needs to be backed up in the first place.

What is it that I am missing? Why is it do difficult for NextCloud to create a list of the items that need to be backed up? I can write a little script if I know what to include in the script.

Also this sentence is not clear:

you should be able to restore by throwing away the directories and restoring from the tar and setting an appropriate symlink.

I'm not saying you are wrong. Just that it is not clear what you are saying.

Is there a list of what needs to be backed up? If not then could NextCloud create one for issues like this and #615?

[mikolodz]

@r4co0n @kyrofa

if you stop the snap, tar up /var/snap/nextcloud/common and what /var/snap/nextcloud/current points to, you should be able to restore by throwing away the directories and restoring from the tar and setting an appropriate symlink. I haven't tested this, as I am backing up the whole machine, but I see no reason why this shouldn't work.

I don't see it working if you do it on a system with other nextcloud revision installed (or no revision installed). Snap knows exactly where it needs to point to with a 'current' symlink. I actually did throw all backed up folders (common & 5132) to the new destination on the second server (/var/snap/nextcloud/) and it's not even trying to work. Firstly, I did create a folder 'nextcloud' inside snap (fresh ubuntu server installation), because I hoped snap will figure it out and find imported folders, but it didn't. Then I tried to install same NextCloud revision (5132) with snap, but it is not possible as far as this version isn't even listed with 'snap info nextcloud' - so I dropped the idea. The third attempt was to create a symlink to the 5132 folder with the name 'current', but it also doesn't work. 'current' link is created by Snap while the app is being brought up and modifying it afterwards makes no sense. When the app is down, there is no 'current' symlink to be edited.
If I knew where snap manager configuration files are stored, maybe it would be possible to manually modify the revision and location of the nextcloud installation. As for now it is a straight mess and it lacks documentation.

Do you have any idea how can I force snap to recognize the existing NextCloud folder, so it tried to start it up?

[joshmorel]

@mikolodz There is extra metadata stored in /var/lib/snapd/state.json (with the revision id as a key) as well as likely other data in /var/lib/snapd directory related to the app's revision. So you can't just copy over a specific revision and symlink to current to it.

From my experiments so far, just copying over the entire /var/snap/nextcloud directory will work, so long as the revision number is the same.

To get to the same revision, the best option is to upgrade the existing server to the latest snap revision available through the snap store (I've described an alternate if you want to keep the same revision below). Once the upgrade is verified to be working on the source server the migration can be attempted.

Put the source in maintenance mode sudo snap run nextcloud.occ maintenance --on while on the target install what is assumed to be the same revision (say 7658) with sudo snap install nextcloud and put in in maintenance mode as well. Start copying over the files from /var/snap/nextcloud. You will want to omit or just delete any non-current revisions to limit file size. rsync or tar/scp/untar - just make sure with rsync that all existing files are deleted on the target. If the hostname is the same you're practically done (given the scenario this is likely the case). If not you will need to set trusted_hosts with sudo snap run nextcloud.occ config:system:set trusted_domains 1 --value=mynewdomain.com and also create new certs. The only other thing are those config variables set via sudo snap set nextcloud for example php.memory-limit. You'll have to reset any non-defaults. I haven't yet figured out if there is an export/import option for these config options which are stored in /var/lib/snapd/state.json. Take out of maintenance mode and it should work - sudo snap run nextcloud.occ maintenance --off. These steps have worked to me through an experimental run only -- so of course care should be taken for production.

If you don't want to upgrade the source server first you can either use a different channel or build the snap from source. For latest minor revision within a major version use channel, for example sudo snap install nextcloud --channel=12/stable. For a specific minor revision, you need to build. First install snapcraft with sudo snap install snapcraft --classic (classic confinement since snapcraft uses home directory) . Then download the release from GitHub, say v12.0.5 with wget https://github.com/nextcloud/nextcloud-snap/archive/12.0.5snap3.tar.gz. Untar and change directory into the repo directory. Run snapcraft and go get a coffee (or beer or tea or whatever) while the snap builds... Then run sudo snap install nextcloud_12.0.5snap3_amd64.snap --dangerous (dangerous required as not signed) and you should be good to follow similar steps as above.

[mikolodz]

@joshmorel
Josh, your post was very helpful and relevant.
I considered upgrading my source installation to bump up a revision number, but as far as I don't know any doable, rational way of restoring backup with details I can't go with such a risky option. If there's no official way of doing these things, how can I trust "snap refresh && nextcloud.occ upgrade"...

I will follow your build instructions and hopefully it will get the things done.

[ASmith-]

The 'run' command used in conjunction with nextcloud.mysqldump and nextcloud.mysql-client appears to have been depreciated, anyone following the syntax repeatedly posted and not re-edited above will result in a syntax error.

Multiple posted commands above need to be edited/updated to
$ sudo nextcloud.occ maintenance:mode --on
$ sudo nextcloud.mysqldump > mysqldump-nextcloud.txt
$ sudo nextcloud.mysql-client nextcloud < mysqldump-nextcloud.txt <-- correct useage

However the mysql-client 'restore' function is still not functioning for me properly whatsoever and there are just not enough breadcrumbs albeit even with the wrong syntax to figure out what the correct usage is.

$ sudo nextcloud.occ maintenance:mode --on
$ sudo nextcloud.mysqldump > nextcloud-sqldump.txt

Restore Command ??

$ sudo nextcloud.mysql-client --defaults-file=/home/name/nextcloud-sqldump.txt
mysql [ERROR] unknown variable 'defaults-file=/home/name/nextcloud-sqldump.txt

Same error with $ sudo nextcloud.mysql-client --defaults-file=nextcloud-sqldump.txt

In Feb 14 (edited) kyrofa wrote
https://github.com/nextcloud/nextcloud-snap/issues/450
"Once you're to that point, you should be able to import your database on the new snap with nextcloud.mysql-client < database.bak. That will require mysql to have been setup already, so make sure the new snap is installed and to the point of prompting for an admin account before doing that. That will likely need to be run as root".

I however found for that command to work, a database needs to be named on the command line, correcting that advise to this

$ sudo nextcloud.mysql-client nextcloud < nextcloud-sqldump.txt

[kyrofa]

Please take a look at #688 for a feature supporting this, it could use some testing.