Rocket.chat: RC Schema Migration Failure: Database locked at version: 136 (then 147)

[taputeranga]

Description:

Upgrading RC from 0.72.0 to 1.3.1 results in failure when starting RC, specifically I get the following (extracted from log). Re-starting RC resulted in the migration progressing to version 147. Re-starting RC thereafter results in no further change. Log is accessible in gist, see below.

Steps to reproduce:

Not sure if reproducible, but this is what I did:

1. Migrate Mongodb database from WiredTiger to MMapV1 by restoring back-up into fresh db set to MMapV1 storage engine.
2. Downloaded latest RC from https://releases.rocket.chat/latest/download
3. Started new version of RC

Expected behavior:

RC schema Migration should complete without error.

Actual behavior:

RC stops with error in /var/log/messages (edited for brevity; full log available as gist - see below for link)

Your database migration failed
Cannot destructure property value of 'undefined' or 'null'.
This Rocket.Chat version: 1.3.1
Database locked at version: 136
Database target version: 149

Commit: 63bb25e9e9849e9e992f2520cfeea18a53fa774c
Date: Thu Aug 8 21:02:16 2019 -0300
Branch: HEAD
Tag: 1.3.1

Server Setup Information:

• Version of Rocket.Chat Server: 1.3.1
• Operating System: Centos 7
• Deployment Method: Manual Installation
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version: 8.11.3
• MongoDB Version: 3.6.13

Additional context

Before upgrade, the mongo db was migrated from WiredTiger to mmapv1, as mmapv1 is what the doc's recommend. This seemed to go without a hitch.

Relevant logs:

/var/log/messages for RC here: https://gist.github.com/taputeranga/a053780f7ae95c25596b4be1c706c1f5

[reetp]

For as long as I can remember I have always been told that to save yourself a lot of heartache and pain you should do upgrades one a version at a time due to schema changes in mongo.

So 0.72 -> 0.73 -> 0.74 -> 1.0.x -> 1.1.x -> 1.2.x -> 1.3.x

Jumping versions is likely to cause issues.

You are upgrading a fast moving application through 6 versions in one go. Your version was released last December and is several versions out of date.

Kind of like expecting it all to work if say you tried upgrading from Windows 98/ME direct to Windows 10. YMMV.

If you read the docs you will also find that you need also need Oplogs enabled.

https://rocket.chat/docs/installation/updating/to-1-0/#oplog-required

I would agree that there should be some proper documentation on how upgrade with some warnings but I don't think you are going to get this particular issue fixed.

[saadismail]

I ran into the same issue going from 0.74.3 -> 1.3.2. I will try upgrading one major version at a time and get back here how it goes. Thank you for the help.

[reetp]

I'll see if I can't get this documented somehow.

Note I've personally been upgrading my main system one version at a time since around v0.20 and never had an issue.....

[saadismail]

The update went successfully when I upgraded one major version at a time. Thank you so much @reetp for the workaround.

[crazyquesadilla]

I saw the same problem trying to upgrade from 0.65.2 to 1.3.0 in a test environment. I was able to make it work by trying to upgrade from 0.65.2 to 1.0.0, and then from 1.0.0 to 1.3.2.

[Exordian]

this migration seems to break: https://github.com/RocketChat/Rocket.Chat/blob/develop/server/startup/migrations/v148.js#L7

since there is no setting FEDERATION_Domain yet

[winterstefan]

Same issue for me as @Exordian mentioned:

• v147 removes a few FEDERATION_* settings (including FEDERATION_Domain)
• v148 tries to fetch the FEDERATION_Domain setting

That results in a locked v147 version and the error Cannot destructure property 'value' of 'undefined' or 'null'.

I also must humbly admit, that I'm trying to upgrade to 1.3.2 from a rather old version. But having v148 trying to access settings which are removed in v147 seems wrong anyways. At least I don't see a release-by-release upgrade in this particular case.

Nevertheless, I will try the mentioned _workaround_ (upgrading to 1.0.0 in the first step).

Could anybody shed some light on this?

[SmashTester]

Attempts to manually enter the deleted field:

db.getCollection('rocketchat_settings').insert({
"_id" : "FEDERATION_Domain",
"type": 'string',
"i18nLabel": 'FEDERATION_Domain',
"i18nDescription": 'FEDERATION_Domain_Description',
"alert": 'FEDERATION_Domain_Alert',
"disableReset": true,
});

[skudi]

Thanks @SmashTester after running your query migrations ended succesfully.
I've got almost the same situatiion as @taputeranga: migration from 0.74.0 to 1.3.2 in one step.

[winterstefan]

_jfyi_ The current PR #15285 will fix the issue. For any upgrade before release of this PR, see workaround(s) above.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alansikora]

Closing as it seems to be fixed.