Rocket.chat: migration failed upgrading from 0.65.2 to 0.66.1

Created on 5 Jul 2018 · 31Comments · Source: RocketChat/Rocket.Chat

Description:

Tried to upgrade a 0.65.2 installation this morning but did not succeed. Error in log:

Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m+#033[39m#033[31m-----------------------------------------------------------------------------------------------------------------#033
[39m#033[31m+#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m                                                                                                                 #033[31m|#03
3[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                              ERROR! SERVER STOPPED                                              #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                                                                                                 #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                         Your database migration failed:                                         #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m  cannot use the part (settings of settings.preferences.groupByType) to traverse the element ({settings: null})  #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                                                                                                 #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                        Please make sure you are running the latest version and try again.                       #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                 If the problem persists, please contact support.                                #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                                                                                                 #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                         This Rocket.Chat version: 0.66.1                                        #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                         Database locked at version: 124                                         #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m                                           Database target version: 129                                          #033
[39m#033[31m|#033[39m
Jul  5 09:51:05 rocketchat-extern RocketChat[5392]: #033[31m|#033[39m#033[31m

Steps to reproduce:

Upgrade from 0.65.2 to 0.66.1
Error

Expected behavior:

Update works.

Actual behavior:

Update fails.

Server Setup Information:

Version of Rocket.Chat Server: 0.65.2 / 0.66.1
Operating System: Linux
Deployment Method: tar
Number of Running Instances: 1
DB Replicaset Oplog: 1
NodeJS Version: 8.11.3
MongoDB Version: 3.6

Additional context

This happens only on one of my rocketchat machines. Others succeeded running the update.

Source

himpierre

Most helpful comment

For any of those that skipped 125 run this query:

db.users.update({
    'settings.preferences.groupByType': { $exists: true }
}, {
    $rename: {
        'settings.preferences.groupByType': 'settings.preferences.sidebarGroupByType'
    }
}, {
    multi: true
});

https://forums.rocket.chat/t/snap-migration-failed-for-0-66-1/1505 also detailed here

But to summarize for those that haven't read above... this should do the trick:

sudo rocketchat-server.mongo
use parties
db.migrations.update({_id: 'control'}, {$set: {locked: false, version:125}})
# Lets run the failed migration
RocketChat.models.Users.update({
    'settings.preferences.groupByType': { $exists: true }
}, {
    $rename: {
        'settings.preferences.groupByType': 'settings.preferences.sidebarGroupByType'
    }
}, {
    multi: true
});

Then exit the mongo shell and restart Rocket.Chat
sudo systemctl restart snap.rocketchat-server.rocketchat-server

geekgonecrazy on 6 Jul 2018

👍3

All 31 comments

Same here.

Some users in mongodb don't have settings :
db.users.count( { "settings" : { $exists: false } } )

I guess that we have to update those users in db, still digging...

magicbelette on 5 Jul 2018

edit: please don't run the command bellow

db.users.updateMany( 
{ "settings" : 
    { $exists: false } 
},  
{ $addToSet: 
    { "settings" : 
        { "preferences" : 
            { 
                "newRoomNotification" : "door", 
                "newMessageNotification" : "chime", 
                "useEmojis" : true, 
                "convertAsciiEmoji" : true, 
                "saveMobileBandwidth" : true, 
                "collapseMediaByDefault" : false, 
                "autoImageLoad" : true, 
                "emailNotificationMode" : "mentions", 
                "unreadAlert" : true, 
                "desktopNotificationDuration" : 0,  
                "viewMode" : 0, 
                "hideUsernames" : false, 
                "hideRoles" : true, 
                "hideAvatars" : false, 
                "hideFlexTab" : false,  
                "highlights" : [ ], 
                "sendOnEnter" : "normal", 
                "roomsListExhibitionMode" : "unread", 
                "sidebarSortby" : "activity", 
                "sidebarViewMode" : "medium", 
                "sidebarHideAvatar" : false, 
                "sidebarShowUnread" : true, 
                "sidebarShowFavorites" : true, 
                "groupByType" : true
            } 
        } 
    } 
} )

magicbelette on 5 Jul 2018

Uahh. Should i issue that command?

himpierre on 5 Jul 2018

You can modify preferences to match yours. It will add settings to users who don't have but you must test it on a non-production environnement first.

Cheers

magicbelette on 5 Jul 2018

@magicbelette same problem here.

`{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Migrating from version 119 -> 129","time":{"$date":1530799649196},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 120","time":{"$date":1530799649199},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 121","time":{"$date":1530799649217},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 122","time":{"$date":1530799649237},"level":"info"}
Fixing ChatSubscription u._id_1_name_1_t_1_code_1
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 123","time":{"$date":1530799649262},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 124","time":{"$date":1530799649267},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 125","time":{"$date":1530799649277},"level":"info"}

+-----------------------------------------------------------------------------------------------------------------+
| |
| ERROR! SERVER STOPPED |
| |
| Your database migration failed: |
| cannot use the part (settings of settings.preferences.groupByType) to traverse the element ({settings: null}) |
| |
| Please make sure you are running the latest version and try again. |
| If the problem persists, please contact support. |
| |
| This Rocket.Chat version: 0.66.1 |
| Database locked at version: 119 |
| Database target version: 129 |
| |
| Commit: fb5257f618b22638c6b2ac4c678f76809f5a7d7e |
| Date: Wed Jul 4 15:05:11 2018 -0300 |
| Branch: HEAD |
| Tag: 0.66.1 |
| |
+-----------------------------------------------------------------------------------------------------------------+`

andreas-bulling on 5 Jul 2018

ok, I made some progress here. Now the error changed:

`{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Migrating from version 126 -> 129","time":{"$date":1530803855394},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 127","time":{"$date":1530803855396},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 128","time":{"$date":1530803855404},"level":"info"}
{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 129","time":{"$date":1530803855422},"level":"info"}

andreas-bulling on 5 Jul 2018

Same problem here:

{"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Migrating from version 124 -> 129","time":{"$date":1530822356940},"level":"info"}
Jul  5 22:25:56 chat rocketchat-server.rocketchat-server[30864]: {"line":"160","file":"rocketchat_migrations.js","message":"Migrations: Running up() on version 125","time":{"$date":1530822356943},"level":"info"}
--------
cannot use the part (settings of settings.preferences.groupByType) to traverse the element ({settings: null})
---------
Trying to fix that problem using:
db.migrations.update({_id: 'control'},{$set:{locked:false,version:125}})
without succes.

Would it be a solution this?
db.users.updateMany( 
{ "settings" : 
    { $exists: false } 
},  
{ $addToSet: 
    { "settings" : 
        { "preferences" : 
            { 
                "groupByType" : true
            } 
        } 
    } 
} )

robotergarten on 5 Jul 2018

This worked for me but then I ran into another problem (see #11362)

andreas-bulling on 5 Jul 2018

@robotergarten I had same error and was able to resolve it by running the following commands
sudo /snap/rocketchat-server/current/bin/mongo
use parties
db.migrations.update({_id: 'control'}, {$set: {locked: false,version:129}})

realgt on 6 Jul 2018

Same problem here, temporarily fixed by bumping the db version to 129 (Thanks @realgt ) but a proper fix is really needed.

NickStallman on 6 Jul 2018

Taking a look at this. Working through it with a few in #ubuntu-snap on open.rocket.chat

PLEASE for your own sake do not skip migration. By setting migration number ahead, you will certainly encounter issues with your install going forward.

geekgonecrazy on 6 Jul 2018

Is there a way to set it back?

kevincolten on 6 Jul 2018

👍1

you can set back via: db.migrations.update({_id: 'control'}, {$set: {locked: false,version:119}})

Right now I suspect doing as suggested by @robotergarten will do the trick

Basically stop Rocket.Chat - sudo systemctl stop snap.rocketchat-server.rocketchat-server

Then fix the issue causing the migration to fail:

sudo rocketchat-server.mongo
use parties
db.users.updateMany( 
{ "settings" : 
    { $exists: false } 
},  
{ $set: 
    { "settings" : 
        { "preferences" : 
            { 
                "groupByType" : true
            } 
        } 
    } 
} )

db.migrations.update({_id: 'control'}, {$set: {locked: false}})

Then exit mongo shell and run: sudo systemctl start snap.rocketchat-server.rocketchat-server

This should fix it

geekgonecrazy on 6 Jul 2018

@robotergarten suggestion as you stated above did not work for me. It was locked at 124 and it still gave me:

cannot use the part (settings of settings.preferences.groupByType) to traverse the element ({settings: [ { preferences: { groupByType: true } } ]})

I had to set my migration forward to 129 again temporarily until a something really works again.

dmotors on 6 Jul 2018

Same here, it did not work for me so I bumped it to 125 to get it working until a working solution is found.

thompsonchief on 6 Jul 2018

Skipping to 125 is the safest because only skipping the broken migration.

sudo rocketchat-server.mongo
use parties
db.migrations.update({_id: 'control'}, {$set: {locked: false, version:125}})

Exit shell then run: sudo systemctl restart snap.rocketchat-server

geekgonecrazy on 6 Jul 2018

Here is the process that seems to have fixed it for me. Thanks to @geekgonecrazy!

I shutdown Rocket.Chat.

In MongoDB I set my version back using:

'db.migrations.update({_id: 'control'}, {$set: {locked: false,version:124}})' - This was my original value.

Then ran...

`use parties
db.users.updateMany( { "settings" : { $exists: false } }, { $set: { "settings" : { "preferences" : { "groupByType" : true} } } } )

db.migrations.update({_id: 'control'}, {$set: {locked: false}})`

Started up Rocket.Chat server, no errors on startup, and it is up and running.

While it is working well, is there a way to check the schema to verify it has the needed changes?

JoeatMJ on 6 Jul 2018

For any of those that skipped 125 run this query:

db.users.update({
    'settings.preferences.groupByType': { $exists: true }
}, {
    $rename: {
        'settings.preferences.groupByType': 'settings.preferences.sidebarGroupByType'
    }
}, {
    multi: true
});

https://forums.rocket.chat/t/snap-migration-failed-for-0-66-1/1505 also detailed here

But to summarize for those that haven't read above... this should do the trick:

sudo rocketchat-server.mongo
use parties
db.migrations.update({_id: 'control'}, {$set: {locked: false, version:125}})
# Lets run the failed migration
RocketChat.models.Users.update({
    'settings.preferences.groupByType': { $exists: true }
}, {
    $rename: {
        'settings.preferences.groupByType': 'settings.preferences.sidebarGroupByType'
    }
}, {
    multi: true
});

Then exit the mongo shell and restart Rocket.Chat
sudo systemctl restart snap.rocketchat-server.rocketchat-server

geekgonecrazy on 6 Jul 2018

👍3

@geekgonecrazy Is it safe to do the RocketChat.models.Users.update... portion after the rocketchat server restart?

thepowerprocess on 6 Jul 2018

You'll want to do a restart of Rocket.Chat after for safe measure, but yes it should be.

geekgonecrazy on 6 Jul 2018

When I switched to 125, I did see 126, 127, 128, and 129 execute successfully.

When I went back to apply the missed 125 update, I am getting a "ReferenceError: RocketChat is not defined" message.

What is missing?

JoeatMJ on 6 Jul 2018

Oops updated comment. Typed command we would run not the command you would run. Sorry about that.

Just swap RocketChat.models.Users with db.users

geekgonecrazy on 6 Jul 2018

Thanks for the update. I ran this:

db.users.update({ 'settings.preferences.groupByType': { $exists: true }}, {$rename: { 'settings.preferences.groupByType': 'settings.preferences.sidebarGroupByType' }}, { multi: true}):

I get the result "errmsg" : "cannot use the part (settings of settings.preferences.groupByType) to traverse the element ({settings.preferences.groupByType' : true } } ]})"

Let me know what I am missing.

JoeatMJ on 6 Jul 2018

I found it. db.Users. The query executed successfully.

JoeatMJ on 6 Jul 2018

👍1

That got me closer, but now hitting #11362

|                                                                      |
|                         ERROR! SERVER STOPPED                        |
|                                                                      |
|                    Your database migration failed:                   |
|       Cannot read property 'desktopNotifications' of undefined       |
|                                                                      |
|  Please make sure you are running the latest version and try again.  |
|           If the problem persists, please contact support.           |
|                                                                      |
|                   This Rocket.Chat version: 0.66.1                   |
|                    Database locked at version: 128                   |
|                     Database target version: 129                     |
|                                                                      |
|           Commit: fb5257f618b22638c6b2ac4c678f76809f5a7d7e           |
|                  Date: Wed Jul 4 15:05:11 2018 -0300                 |
|                             Branch: HEAD                             |
|                              Tag: 0.66.1                             |
|                                                                      |
+----------------------------------------------------------------------+

kevincolten on 6 Jul 2018

@kevincolten have you ran the command from @magicbelette which includes a $addToSet command?

unfortunate that command it wrong.. it should not be a $addToSet (which creates an array) but a $set instead (that creates an object). I have edited his comment to at least hide the command so people don't run it.

so in your case, you'll need to run the additional command, which will revert all arrays to objects again:

db.users.find({ 'settings.0': {$exists:true} }, { settings: 1 }).forEach(function(user) {
    db.users.update({ _id: user._id }, { $set: { settings: user.settings[0] } });
})

sampaiodiego on 6 Jul 2018

👍1

db.Users.find({ 'settings.0': {$exists:true} }, { settings: 1 }).forEach(function(user) {     
  db.users.update({ _id: user._id }, { $set: { settings: user.settings[0] } });
})

Seems to have done the trick! thanks @sampaiodiego

kevincolten on 6 Jul 2018

🎉1

Sorry for the huge mistake ! And thank you @sampaiodiego for your quick answer

magicbelette on 6 Jul 2018

no worries @magicbelette .. you did it with good intention.. 🤗

sampaiodiego on 6 Jul 2018

I've encountered this issue as well - Rocket.Chat suddenly stopped working just before our deadline
Big thanks to everyone! Skipping migration to 125 and applying your fixes worked.

mpodshivalin on 6 Jul 2018

🎉2

Thanks guys. I've finally taken our installation offline and with 0.66.2 all I needed to do was revert the db version back to 124 and restart RocketChat.
0.66.2 is available via Snap.

All migrations completed successfully now without any additional tinkering with the database.

NickStallman on 10 Jul 2018

🎉2

Was this page helpful?

0 / 5 - 0 ratings