Rocket.chat: Database broken with 0.49

[admin-crowd-iugo]

Your Rocket.Chat version: (make sure you are running the latest)
0.49

We upgraded last night and this morning we had tons of missing/lost messages. Tried to revert to 0.48.2 and rocket chat failed to come up saying database was locked. tried clearing lock manually by setting variable from mongo directly but it kept locking and not coming up. we are running mongo 3.2 with nodejs 4.7 and 7 node instances of rocket chat.

the commands i used were :
use rocketchat
db.migrations.find().pretty()
db.migrations.update({}, { $set: { version: 79, locked: false } })

[engelgabriel]

Can you give us some more info?
How many users?
Home many chat process?
Is the MongoDB running on the same machine?
Is the MongoDB running in ReplicaSet mode?
Can you backup your data and try with the version 0.49.1?

[johnlund]

@engelgabriel, in reference to this issue: https://github.com/RocketChat/Rocket.Chat/issues/5542, my database is still locked and my chat site is still down. I'm using Rocketchat through Ubuntu Snap. 1) 49.1 doesn't appear to have hit the Snap server yet (no updates available using refresh), and 2) it's not easy for me to manually access my DB and unlock it on a Snap - will it automatically unlock once the next update goes through? Thanks.

[engelgabriel]

1) I'll verify why.
2) Yes.

[engelgabriel]

@geekgonecrazy can you verify why 0.49.1 doesn't appear to have hit the Snap server yet?

[engelgabriel]

https://travis-ci.org/RocketChat/Rocket.Chat/builds/191813269 completed just fine..

[engelgabriel]

Hummm...

@rodrigok have you changed the password back?

[geekgonecrazy]

@engelgabriel I'll get it sorted back out. Manually pushing 0.49.1 now

[engelgabriel]

@geekgonecrazy do we know why did they stop?

[geekgonecrazy]

@johnlund Sorry about that. 0.49.1 Revision 386 is now available. It should automatically roll out, but if you want it sooner try: snap refresh rocketchat-server

@engelgabriel the macroon/token stored by snapcraft expired. The code we have there is supposed to be refreshing it. Either its not working, or someone signed out all other users via the ubuntu console. At either rate i'm dropping a fresh token to get us back up to auto deploying. Digging into the launchpad path, as that will alleviate that problem going forward.

[engelgabriel]

I think @rodrigok signed out all other users, because he tried changing the password to test the electron build. He changed the password back to the original value, but chances are this cased signed out all other users.

[johnlund]

Working now. Thanks @engelgabriel, and the rest of the Rocketchat team!

[Sing-Li]

@johnlund ... awesome! Glad you're back in action.

For our information -- did you just do a snap refresh and it started working again, or did you have to do any other special procedures (such as accessing mongodb manually) ? Thanks for the info.

[Nemra1]

This somthing happened with manual setup in centos6 when update my db is locked and icant get back online again but i was have backup and back to older version

[admin-crowd-iugo]

How many users? Approx 200+
Home many chat process? 7
Is the MongoDB running on the same machine? Yes
Is the MongoDB running in ReplicaSet mode? Yes
Can you backup your data and try with the version 0.49.1? We will likely wait until a couple of builds have passed before upgrading again.

[graywolf336]

v0.49.2 is going to be released later today and it includes several bug fixes. So, if you're still on 0.49 then I recommend upgrading to that version when it comes out.

[johnlund]

@Sing-Li I didn't do anything. I checked my server about an hour after the Snap push and Snap had automatically pulled 49.1 and restarted the server. Everything was working.

[engelgabriel]

We lock the database when any migration fails, and it will remain locked until a new release is installed with a possible new fix or after 5 minutes to give time for a DBA to try to fix the problem manually.

Migrations = {
    _list: [DefaultMigration],
    options: {
        // false disables logging
        log: true,
        // null or a function
        logger: null,
        // enable/disable info log "already at latest."
        logIfLatest: true,
        // lock will be valid for this amount of minutes
        lockExpiration: 5,
        // retry interval in seconds
        retryInterval: 10,
        // max number of attempts to retry unlock
        maxAttempts: 30,
        // migrations collection name
        collectionName: "migrations"
            // collectionName: "rocketchat_migrations"
    },
    config: function(opts) {
        this.options = _.extend({}, this.options, opts);
    },
}