Rocket.chat: Can't find migration version 98 / Old version was running after upgrade / snap keeps downgrading

[amenk]

Rocket.Chat Version: 0.57.0
Running Instances: 1
DB Replicaset OpLog: ?
Node Version: 4.8.1 - x64

The chat does not start anymore.

Logs:

Jul 05 03:00:34 chat2 snap[1001]: {"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 98","time":{"$date":1499216434691},"level":"info"}

But later:

Jul 05 05:28:27 chat2 snap[26856]: Exception in callback of async function: Error: [Can't find migration version 98]

[amenk]

Turned out, an old version was running.

I was able to solve it with

$ sudo snap refresh rocketchat-server --revision=801
2017-07-05T08:57:12+02:00 INFO cannot auto connect rocketchat-server:network-bind to     core:network-bind: (plug auto-connection), existing connection state "rocketchat-server:network-bind     core:network-bind" in the way
rocketchat-server 0.57.0 from 'rocketchat' refreshed

Will leave it open a bit, in case others have the same problem or if this is a problem with the snap upgrate procedure.

[amenk]

Happend again ...

[amenk]

to refresh the snap to the new version - the problem just happend again, without us doing anything

[benbourner]

Same here, exactly as described by you, both fix and recurrence.

[benbourner]

Error: [Can't find migration version 98]
  at Object.Migrations._findIndexByVersion (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:405:8)
  at Object.Migrations._migrateTo (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:272:22)
  at Object.Migrations.migrateTo (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:201:20)
  at meteorInstall.server.startup.migrations.xrun.js (/snap/rocketchat-server/671/programs/server/app/app.js:4808:24)
  at fileEvaluate (packages/modules-runtime.js:197:9)
  at require (packages/modules-runtime.js:120:16)
  at /snap/rocketchat-server/671/programs/server/app/app.js:9754:1
  at /snap/rocketchat-server/671/programs/server/boot.js:303:34
  at Function._.each._.forEach (/snap/rocketchat-server/671/node_modules/underscore/underscore.js:153:9)
  at /snap/rocketchat-server/671/programs/server/boot.js:128:5
  at /snap/rocketchat-server/671/programs/server/boot.js:352:5
  at Function.run (/snap/rocketchat-server/671/programs/server/profile.js:510:12)
  at /snap/rocketchat-server/671/programs/server/boot.js:351:11

[amenk]

My guess: Snap tried to upgrade, runs into problems, reverts, rocket.chat runs in to problems then ...

[benbourner]

Sounds likely. Do you know how to turn off automatic upgrades?

[amenk]

Workaround: sudo systemctl disable snapd.refresh.service

[bentimms]

I think that might be sudo systemctl disable snapd.refresh.timer on Ubuntu 16.04
EDIT: and also sudo systemctl stop snapd.refresh.timer ?

[amenk]

@bentimms does not seem to help either -- just downgraded again automatically

[thachpn165]

I've faced this problem too:

Exception in callback of async function: Error: [Can't find migration version 98]
  at Object.Migrations._findIndexByVersion (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:405:8)
  at Object.Migrations._migrateTo (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:272:22)
  at Object.Migrations.migrateTo (/snap/rocketchat-server/671/programs/server/packages/rocketchat_migrations.js:201:20)
  at meteorInstall.server.startup.migrations.xrun.js (/snap/rocketchat-server/671/programs/server/app/app.js:4808:24)
  at fileEvaluate (packages/modules-runtime.js:197:9)
  at require (packages/modules-runtime.js:120:16)
  at /snap/rocketchat-server/671/programs/server/app/app.js:9754:1
  at /snap/rocketchat-server/671/programs/server/boot.js:303:34
  at Function._.each._.forEach (/snap/rocketchat-server/671/node_modules/underscore/underscore.js:153:9)
  at /snap/rocketchat-server/671/programs/server/boot.js:128:5
  at /snap/rocketchat-server/671/programs/server/boot.js:352:5
  at Function.run (/snap/rocketchat-server/671/programs/server/profile.js:510:12)
  at /snap/rocketchat-server/671/programs/server/boot.js:351:11

Just fixed with @amenk solution but it happened again.

Waiting for another path update.

[bentimms]

For belt and braces, perhaps this too: sudo systemctl stop snapd.refresh.timer

[gregs007]

Seeing this too. So far I've:

sudo systemctl disable snapd.refresh.service
sudo systemctl disable snapd.refresh.timer
sudo systemctl stop snapd.refresh.timer

But the bigger question is why is this happening? Is this a problem with the newest snap (801?)

[gregs007]

FYI the above does not work. Snap is still refreshing and pulling down the server every few hours. Anyone know how to make it stay at a version? Is there a website I can block? This keeps taking down our production chat.

[geekgonecrazy]

Sorry for the trouble guys.

Basically.. 0.57.0 came out we released the snap. Revision 801. We then had a few people encounter issues with avatars. So we halted the roll out.

Unfortunately... we didn't stop it in time. Anyone that had already upgraded to 0.57.0 during the window it was deployed.. was then rolled back to 0.56.0 when we stopped the roll out.

Since its listed as the newest snap available... it keeps updating your snap regardless of what revision you install.

I've gone ahead and released it back so revision 801 is the latest. We have an avatar fix in the works and will be able to roll it out in a very short amount of time to fix anyones avatars that broke.

Sorry again for the troubles.

[trungducnet]

When I run this command the second time, after fixing it and got back after a few hours:

root@manage:~# sudo snap refresh rocketchat-server --revision=801

And this is an error:

error: cannot perform the following tasks:
- Copy snap "rocketchat-server" data (cannot copy "/var/snap/rocketchat-server/801" to "/var/snap/rocketchat-server/801": failed to copy all: "cp: cannot stat ‘/var/snap/rocketchat-server/801’: No such file or directory" (1))

Who can solve this?

[gregs007]

@trungducnet When that happened to me, I tried reverting to a previous snap first, then back again and it worked. But it did delete my caddy file and I had to recreate it. Snap 812 is out now, so maybe just the standard sudo snap refresh rocketchat-server will do the trick.

[geekgonecrazy]

@trungducnet yes i'd try a standard refresh. if you still have problems please feel free to join #ubuntu-snap and we can try and debug in a bit more real time.

[geekgonecrazy]

The crashing reported here should be resolved. Going ahead and closing.