Docker-gitlab: Two factor auth has error

[atkrad]

Hi,

I set GITLAB_SECRETS_OTP_KEY_BASE=generated-token thereafter open /profile/two_factor_auth then Gitlab throw error 500

production.log:

Started GET "/profile/two_factor_auth" for 192.168.11.197 at 2016-08-24 10:52:24 +0430
Processing by Profiles::TwoFactorAuthsController#show as HTML
Completed 500 Internal Server Error in 57ms (ActiveRecord: 3.7ms)
OpenSSL::Cipher::CipherError (bad decrypt):
app/controllers/profiles/two_factor_auths_controller.rb:5:in `show'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'

Could you help me, please.

[adamlc]

Same problem here, although I can't even login!

OpenSSL::Cipher::CipherError (bad decrypt):
  app/controllers/sessions_controller.rb:108:in `valid_otp_attempt?'
  app/controllers/concerns/authenticates_with_two_factor.rb:46:in `authenticate_with_two_factor_via_otp'
  app/controllers/concerns/authenticates_with_two_factor.rb:35:in `authenticate_with_two_factor'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

[solidnerd]

Hey,
this could come from several ways. Did you change your GITLAB_SCRETS_OTP_KEY_BASE two times ? It should be set only once. Did you set GITLAB_SECRETS_SECRET_KEY_BASE as well. I tested it on some of my hosts and there i didn't get this issue. I need some more informations about it. Do you use Two-Factor auth before 8.11.0

[dominikschulz]

If you have access to the DB you can disable 2FA until this issue is fixed:
UPDATE users SET otp_required_for_login = false;

[adamlc]

@SolidNerd I used TFA before 8.11.0. Before I had only GITLAB_SECRETS_DB_KEY_BASE, so for the upgrade I also added a GITLAB_SECRETS_SECRET_KEY_BASE and GITLAB_SECRETS_OTP_KEY_BASE.

[atkrad]

@adamlc me too

[dominikschulz]

Opened an Issue with GitLab, as I'm not sure if this is related to docker-gitlab or GitLab itself: https://gitlab.com/gitlab-org/gitlab-ce/issues/21348

[adamlc]

@dominikschulz the query didn't seem to work for me. Its still erroring on /profile/two_factor_auth :(

[dominikschulz]

The query will only let you login w/o 2FA. I can't access /profile/two_factor_auth either.

[adamlc]

Unfortunately it won't even let me login, even after running the query. It keeps redirecting to /profile/two_factor_auth

[adamlc]

I'm thinking I could set encrypted_otp_secret, encrypted_otp_secret_iv & encrypted_otp_secret_salt to null and see if it helps.

[solidnerd]

Could you check if you have an .secret in your container. It should be under the following path /home/git/gitlab

So if not. Try to restore to 8.10-7 and check if it's there. When it's exist there should be the correct values for otp.

[adamlc]

Nothing there @SolidNerd

root@d421122371b5:/home/git/gitlab# cat .secret
cat: .secret: No such file or directory

[dominikschulz]

No .secret here either. We use the docker-compose.yml from this repo which mounts a data volume to /home/git/data. This means that all the other files are not persisted upon upgrade (since we remove the container to start the new image). So far I wasn't aware that there is anything outside ENV/DB/datafolder to be persisted.

[atkrad]

@dominikschulz I have .secret in /home/git/data/ path

[dominikschulz]

@atkrad Indeed, there is a /home/git/data/.secret. I did only look in /home/git/gitlab/.secret.

[smcgivern]

In GitLab 8.11, we made some changes to secret names and locations to reflect what they're actually used for and make managing them easier. There's a description of this for Omnibus here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update/README.md#migrating-legacy-secrets

For source installs, this is what happens:

1. Previously, we used the value from .secret for otp_key_base and secret_key_base. While secret_key_base can be rotated fairly harmlessly (it will expire password reset tokens, etc.), changing the value of otp_key_base will prevent 2FA from working for all users: https://gitlab.com/gitlab-org/gitlab-ce/issues/3963
2. On upgrade, move the value of .secret into config/secrets.yml under both keys, so that this is clearer, and delete the .secret file.
3. If there is no .secret file and there are no matching values in config/secrets.yml, we generate new secrets - again, if there's existing 2FA data encrypted in the DB, this will also have the effect of stopping 2FA working for everyone.

So my first suggestion would be to see if you have a backup of your old .secret file, and provide that value in config/secrets.yml. If /home/git/data/.secret has the right value, that should work too.

[smcgivern]

@SolidNerd I see in the README this advises people:

Note: Since GitLab 8.11.0 you need to provide the GITLAB_SECRETS_SECRET_KEY_BASE and GITLAB_SECRETS_OTP_KEY_BASE parameter while starting the image.

I'm sorry if the above https://github.com/sameersbn/docker-gitlab/issues/859#issuecomment-241986947 wasn't clear from the CE repo changes 😞 . Will the old value still be around anywhere, by default?

[adamlc]

Taking the value from /home/git/data/.secret and setting that as GITLAB_SECRETS_OTP_KEY_BASE worked for me.

In theory everyone should have that in their Docker data volume :)

[solidnerd]

@smcgivern I will update the README asap.

@adamlc Set this value for GITLAB_SECRETS_SECRET_KEY_BASE as well.

[adamlc]

Will do @SolidNerd, thanks @smcgivern for the info :)

[smcgivern]

@SolidNerd @adamlc _technically_ that doesn't matter too much. Like I said, secret_key_base is OK to rotate - otp_key_base, though, is really important and should be kept very safe! However, for now that's safe advice.

[adamlc]

Awesome, I'll leave it as a different value in that case :)

[solidnerd]

@smcgivern If you want you could help us to improve our documentation. I would really appreciate it :green_heart: . Feel free to create a PR.

[dominikschulz]

Using the value from /home/git/data/.secret for GITLAB_SECRETS_SECRET_KEY_BASE and GITLAB_SECRETS_OTP_KEY_BASE works like a charm.

Thanks for the awesome real-time support!

[atkrad]

Thanks @smcgivern @SolidNerd

[smcgivern]

@SolidNerd will do, this was kind of a mess before so the upgrade can be tricky. Sorry for the confusion in what these actually do.

@dominikschulz @atkrad @adamlc no problem, glad it was that easy to resolve!

[martingabelmann]

In future something like this (comparing the value of GITLAB_SECRETS_OTP_KEY_BASE and the one stored in the container) somewhere in the assets could also prevent from been confused, right?

[smcgivern]

@martingabelmann yes, absolutely. I don't think I'll have time to work on that, unfortunately, but we have tried to make source and Omnibus upgrades to 8.11 as painless as possible for this, so it would be great if it could be the same for this version :tada:

[smcgivern]

Doc changes: https://github.com/sameersbn/docker-gitlab/pull/860

[wayearn]

@solidnerd Why should set GITLAB_SECRETS_OTP_KEY_BASE and GITLAB_SECRETS_SECRET_KEY_BASE the same?

[smcgivern]

@kanlidy if you are upgrading from an earlier version, they will have implicitly had the same values. If you change one of them, that will invalidate some encrypted secrets.

If you're starting with a new installation, they don't need to be the same!