Nextcloud-snap: Question: How does snap revert handle server-side encryption?

[the-sane]

Consider the following example:

I'm on Nextcloud version A using the snap. I upgrade to version B and then deploy server-side encryption. The files are encrypted and the keyfiles are stored by the server in their appropriate locations (locations listed here: https://github.com/syseleven/nextcloud-tools/blob/master/server-side-encryption.md#file-types). Then, I revert back to version A using the snap revert command.

Assume that server-side encryption as a feature exists and hasn't changed significantly between versions A and B.

Questions:

• Does the reversion attempt to decrypt everything beforehand? I assume no?
• What happens to those keyfiles? Are they wiped with the reversion?
• Does anything in the database or configuration files change during the reversion? Will the server forget that server-side encryption was enabled and expect to see unencrypted data again?

[the-sane]

I believe I have answered the second question while continuing to research this. It appears encryption keys are stored directly in the $SNAP_DATA directory. I'm assuming a snap revert doesn't touch this directory, which would mean the keys are left alone.

As a bonus, this also means the nextcloud.export script grabs the encryption keys while backing up the data folder. Once I test this myself and confirm the details, I'll probably create a request to expand the export script's functionality so it can back up encryption keys separately from all the data. I'll let you know which specific directories would need backing up.

[kyrofa]

Excellent questions:

• Does the reversion attempt to decrypt everything beforehand? I assume no?

It does not. Reversion happens without the snap being aware, and the raw data (which is encrypted at this point) is shared between all revisions, even after a revert. This is done simply because the data is too large to hold multiple copies.

• What happens to those keyfiles? Are they wiped with the reversion?

No, as you noticed they are stored within $SNAP_DATA, which is specific to the revision you're using. Reverting does not remove it, it just stops using that particular directory.

• Does anything in the database or configuration files change during the reversion? Will the server forget that server-side encryption was enabled and expect to see unencrypted data again?

Yes indeed. There are two important directories here: $SNAP_DATA and $SNAP_COMMON. $SNAP_DATA is versioned, that is, specific to a given revision. When the snap updates, a new $SNAP_DATA is created based on the old revision's data, which is an important part of the rollback strategy. $SNAP_COMMON, on the other hand, is shared across all revisions, and copies are never made. As a result, messing up $SNAP_DATA can be recovered with a revert, whereas messing up $SNAP_COMMON will require restoring from a backup. This might be helpful in learning more about these (and other) snap-writable areas.

Anyway, the Nextcloud snap stores as much as it reasonably can within $SNAP_DATA so that reverts are as meaningful as possible. This includes the database. Did you upgrade across two major versions and end up with Nextcloud saying "sorry, we don't support that"? Revert. Did an upgrade fail to run database migrations? Revert. The side effect is that, if you upgrade to a new revision and enable encryption, that effects both the database in $SNAP_DATA as well as the raw data in $SNAP_COMMON. Reverting will roll back to using the old database before encryption was enabled, but sadly the raw data is still encrypted. I'm not sure there's a good way to handle that situation.

Note: This is assuming one is using the default Nextcloud snap config. It's possible to store the data in a place other than $SNAP_COMMON, but unless one specifically places it within $SNAP_DATA what is said here still applies.

[the-sane]

Thank you very much for the info! It sounds like to handle the database issues, the reversion would have to be aware of the encryption and then go muck about in the database post-reversion. It would need to implement some fully automated version of the tools this guy has recently created:
https://github.com/syseleven/nextcloud-tools/

That sounds like (maybe just a little bit?) too much to ask, and it's probably just better to stick some cautions somewhere. Since the user has to the snap's nextcloud.occ tool to enable encryption, is it possible to throw up a warning? Something along the lines of:

WARNING: Due to the way snap stores versioned databases, it is not possible to use the snap revert tool to return to a version before encryption was enabled, or before major changes were made to the way Nextcloud handles encryption. Attempting to do so will result in DATA LOSS! The files will still be encrypted, but the database will have reverted to a state where it is unaware of this fact and, therefore, unable to decrypt them. All encrypted data must be fully decrypted prior to such a reversion.

It is recommended that the nextcloud.export tool be used to make a backup of the current database before proceeding!

I'm not sure where to put that once the bugs are fixed and encryption can be fully enabled through the web interface. Nextcloud's encryption is only partially documented, especially given that it's a bit buggy and requires some workarounds currently, so I was considering gathering all the info I've learned for you to put into a wiki article. Maybe that will suffice?

[kyrofa]

It sounds like to handle the database issues, the reversion would have to be aware of the encryption and then go muck about in the database post-reversion.

Right. The challenge there is determining why the reversion is happening. Is it because something is broken? Are we confident we can successfully decrypt the data using a broken revision? As a general rule, that would be a bad idea.

Since the user has to the snap's nextcloud.occ tool to enable encryption, is it possible to throw up a warning?

nextcloud.occ is just a shell wrapper that call's Nextcloud's occ tool. We might be able to hack a warning in there, but honestly the limitations here are no different from the limitations of any other type of Nextcloud install, are they? Perhaps such a warning should go upstream.

[the-sane]

A normal Nextcloud install doesn't have the snap revert capability, and downgrades are specifically not supported. Any admin attempting a downgrade on a traditional install would be doing it as a very manual process and, therefore, I think we can assume they would be aware of the encryption and handle it accordingly.

The main reason I suggest the warning for the snap is because the snap revert tool is so easy to use and commonly recommended as a solution to upgrade problems. With the encryption situation above, someone who isn't thinking can enter one command and quickly lose all their data.

[the-sane]

I don't know if it's possible to catch a snap revert being issued on nextcloud, since it's a snap tool and not a nextcloud tool, but if that is possible, here's how to check whether encryption is enabled or not:
nextcloud.occ encryption:status | awk 'NR==1{print $3}'
It returns true if encryption is enabled and false if it is not.

Based on everything I'm learning about Nextcloud's encryption, I might even go so far as to say that the snap revert functionality while encryption is enabled should be an unsupported operation. I'm discovering that too many things can go wrong. _It will almost certainly result in data loss, even when reverting between two versions where encryption was enabled in both cases._

Edit: As for why it will result in data loss: The database stores information on which files are encrypted and some signature information for them. If any pieces of this data is missing from the database, then the files can't be decrypted. So consider a scenario similar to what I described in the OP, but both versions A and B had encryption enabled. Whatever files were newly added or changed while version B was installed will not be decryptable after the reversion to A.

[the-sane]

I apologize for flooding this issue with information. I'm in the process of writing up everything I've learned in an appropriately simplified list specifically for snap users, so you can add it to the wiki if you choose. And questions keep coming up! :)

Even in a normal non-encrypted Nextcloud, the database contains the file cache that references all the stored files. So reverting to a previous database after new files had been added to the instance would cause data loss, wouldn't it? The only difference being that in an unencrypted instance, you can run nextcloud.occ files:scan to recover the files that the database had forgotten about.

I have no idea if running files:scan can recover files in the situation directly above where both versions A and B had encryption enabled would work. It certainly wouldn't work in the scenario from my OP.

[the-sane]

So files:scan cannot be used to recover files lost from the database after encryption is enabled. It will break things further.
https://github.com/nextcloud/server/issues/16419#issuecomment-514606377

yahesh:

We also found that you should not use ./occ files:scan when operating an encrypted Nextcloud instance (for example to correct the database after moving files on the file level). Files are added to the oc_filecache table with encrypted equal to 0 even if the files are encrypted.

nickvergessen:

Uff, yeah files:scan should be really used with care. I would argue to even disable it when encryption is on.

[the-sane]

I think the questions here have been sufficiently answered now, thank you! Closing this issue.