Cht-core: Broken migrations should prevent API from starting

[alxndrsn]

Currently my migrations are failing with:

Migration "extract-user-settings" failed with: {"message":"Cannot create lowercase username org.couchdb.user:xxx, user already exists.","stack":"Error: Cannot create lowercase username org.couchdb.user:xxx, user already exists.\n    at /Users/user/workspaces/medic/api/migrations/extract-user-settings.js:28:25\n    at Request._callback (/Users/user/workspaces/medic/api/node_modules/nano/lib/nano.js:217:16)\n    at Request.self.callback (/Users/user/workspaces/medic/api/node_modules/request/request.js:200:22)\n    at emitTwo (events.js:100:13)\n    at Request.emit (events.js:185:7)\n    at Request.<anonymous> (/Users/user/workspaces/medic/api/node_modules/request/request.js:1067:10)\n    at emitOne (events.js:95:20)\n    at Request.emit (events.js:182:7)\n    at IncomingMessage.<anonymous> (/Users/user/workspaces/medic/api/node_modules/request/request.js:988:12)\n    at emitNone (events.js:85:20)"}

However, API continues to start. I am now missing things created in other migrations, such as the translations docs.

[estellecomment]

Can't track down the conversation about this, but another possibility is to block on some migrations, the disruptive important ones, and not on lighter less important ones.
But just blocking for all sgtm too.

[garethbowen]

The downside of this is slow migrations will keep the app offline for ages, but it's still the right thing to do.

[estellecomment]

You can also practice a migration ahead of time on a copy of the prod db. Then you can run it in one go on the prod system.
The error above in extract-user-settings is likely to happen, and needs a human decision, so figuring it out ahead is helpful.

[mandric]

Related https://github.com/medic/medic-api/issues/52

[garethbowen]

I'm promoting this to the current milestone. We're about to do a bunch of 0.4 to 2.x upgrades so migration failures are likely and we need to know about them.

[alxndrsn]

The error above in extract-user-settings is likely to happen, and needs a human decision, so figuring it out ahead is helpful.

@estellecomment what is the human decision that would be needed on failure?

[estellecomment]

@alxndrsn You need humans when you are lowercasing usernames and get a conflict because the lowercase version already exist. You probably need to rename one user to something else, or delete one of the two, depends on the setting.

[alxndrsn]

depends on the setting

have we found clashes in multiple projects and resolved them in different ways?

[estellecomment]

I only worked on CA2.
I don't see a unique reason across projects for why the username clash it would happen.
Why is it happening in the project you're upgrading?

[alxndrsn]

Why is it happening in the project you're upgrading?

No idea!