Mattermost-server: OAuth2 Failure with Gitlab — Bad Token Type

[kkirsche]

Summary

Gitlab 9.2.1 responds to SSO OAuth request and a 500 Error — Bad Token Type is displayed to the user

Steps to reproduce

1. Login to Mattermost 3.9.0
2. Login to Gitlab 9.2.1
3. Navigate in Gitlab to Admin Area > Applications and click create new application
4. Input in create new application within Gitlab two callback URLS, https:///login/gitlab/complete and https:///signup/gitlab/complete
5. Save Application
6. Navigate in Mattermost to the system console
7. Navigate to Authentication > Gitlab
8. Enable authentication with Gitlab
9. Enter the Application ID and Application Secret
10. Enter the API endpoints https://<gitlab url>/api/v3/user, https://<gitlab url>/oauth/authorize, https://<gitlab url>/oauth/token
11. Click save configuration in Gitlab

Easiest way to then test the issue

1. As the admin user, navigate in Mattermost to Account Settings > Security
2. Click Sign-In Method and click Switch to using Gitlab SSO
3. Enter Gitlab password SSO and click Switch Account to Gitlab SSO
4. Click Authorize
5. Receive error JSON {"id":"api.user.authorize_oauth_user.bad_token.app_error","message":"Bad token type","detailed_error":"","request_id":"d7n8de14qprozff7nrgs4reasr","status_code":500}

Expected behavior

SSO would receive a valid token and create the user or change the authentication method

Observed behavior

Received:

{"id":"api.user.authorize_oauth_user.bad_token.app_error","message":"Bad token type","detailed_error":"","request_id":"wadpeoo9bpn5ip69w5xc7qf7ic","status_code":500}

After getting the authorize Gitlab page

Possible fixes

Around
https://github.com/mattermost/platform/blob/5f9a28ed6f45dd02a0aa2b4220717c29567a7ba8/app/oauth.go#L603-L605

[esethna]

Hi @kkirsche, thanks for the report,

Can you please check your GitLab application settings and make sure that only the "api" box is selected? If other options are selected there is a known issue with backwards compatibility that may be causing the error you are seeing.

[kkirsche]

That was incorrectly setup, but changing this to API only did not fix the bug. I also deleted the application and re-added it in Gitlab using just the API to ensure that it wasn't incorrectly storing or caching it and this did not fix the issue either.

[kkirsche]

Oh, and here's the page where it's configured

[esethna]

Thanks for the information @kkirsche, we have been able to reproduce the issue internally and are currently looking into what might be the cause. I will circle back later this week with an update.

[thundo]

After updating to GitLab 9.2.1 (omnibus) this morning I can confirm that I'm unable to login at all on Mattermost via GitLab SSO.

[alextsam]

Same here. Clients that were already logged in are still working normally, new logins give the bad token result.

[esethna]

@kkirsche @thundo @alextsam @GhostLyrics @znz @elritsch

Thanks for your participation on this thread, highly appreciated.

We are working on narrowing down the issue, if you are open to it sharing your GitLab logs for any errors occurring around the time someone attempts to login via GitLab SSO, it would be very helpful.

Also, is there anything non-standard about your omnibus setups that may be leading to this issue?

[kkirsche]

There is nothing unique that I'm aware of (e.g. not using anything like LDAP). are the logs you need from the GUI in the admin area or somewhere on server? Just need to know where the logs you may need are so I can get them

[esethna]

Thanks @kkirsche, the logs are found at /var/log/gitlab/mattermost/mattermost.log

[thundo]

@esethna
The output is pretty much the same as the error returned to the authenticating user

web error

{"id":"api.user.authorize_oauth_user.bad_token.app_error","message":"Bad token type","detailed_error":"","request_id":"jajngxfcff8uxfcb3b6wfys83r","status_code":500}

Log error

[2017/05/26 23:26:58 CEST] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=jajngxfcff8uxfcb3b6wfys83r uid= ip=*** Bad token type [details: token_type=, response_body=]

Btw my /var/log/gitlab/mattermost/mattermost.log hasnt been written since updating to 9.2.1 this morning. However I have my running mattermost log (error shown above included) at /var/log/gitlab/mattermost/mattermost.logmattermost.log. Is this to be expected?

[unaig]

I have the same "EROR" at /var/log/gitlab/mattermost/mattermost.logmattermost.log:

[EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=3xwqt9a3ctftxdz5g7jork5pna uid= ip=xxxxxxxxxxx Bad token type [details: token_type=, response_body=]

[alextsam]

Logs:

2017-05-27_18:42:21.07657 [2017/05/27 18:42:21 UTC] [INFO] Server is listening on 127.0.0.1:8065
2017-05-27_18:42:48.38287 [2017/05/27 18:42:48 UTC] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=<reducted> uid= ip=<reducted>, <reducted> Bad token type [details: token_type=, response_body=]

[CollenK]

GitLab support here. We've had two reports of the same issue. In both cases commenting out mattermost['service_site_url'] = 'https://mattermost.example.com:443' in the gitlab.rb file resolved the issue.

Setting the service_site_url is required when enabling email batching.

/cc @lfbrock @it33

[faradaytrs]

@CollenK this string was commented in my config, but i still have the same problem

[faradaytrs]

Solved when i removed port 443 from my mattermost_external_url config
Was: "https://mattermost.example.com:443"
Now: "https://mattermost.example.com"
It solved the problem

[thundo]

@CollenK commenting it out, re-enabled GitLab SSO on our instance.

Thanks for the workaround, waiting for a fix

[elritsch]

Yet another workaround (inspired by @CollenK's and @faradaytrs's comments):

Changing
mattermost['service_site_url'] = 'https://mattermost.example.com:443'
to
mattermost['service_site_url'] = 'https://mattermost.example.com'
(i.e. removing the port number 443) in the gitlab.rb file fixes the problem on our instance.

Thanks a lot guys for your great support!

[szantnerb]

Unfortunately none of the above mentioned workarounds seem to work for my installation.
On GitLab 9.2.2, Mattermost 3.9.0 (Omnibus install, upgraded last night from 9.0 if I'm not mistaken)

[esethna]

@bszantner can you confirm that you've tried:

1. Removing the port number from the mattermost['service_site_url'] in the gitlab.rb
2. If that doesn't work, commenting out the `mattermost['service_site_url'] line

Is there anything custom about your setup? Can you please provide your gitlab and mattermost logs so we may debug further?

[esethna]

Thanks everyone for helping to debug this issue, it's highly appreciated.

[CollenK]

@bszantner Did you run sudo gitlab-ctl reconfigure after making the changes? If not, please do that for the new settings to apply.

[alextsam]

I would like to report that removing the port number from the mattermost['service_site_url'] did the trick for me too!

[GhostLyrics]

I would like to report that removing the port number from the mattermost['service_site_url'] did the trick for me too!

Same here.

[matfiz]

The workaround have also worked for me, thanks. Do you have any idea if it actually breaks any functionality in Mattermost? If not, maybe it is sufficient to update the docs here https://docs.gitlab.com/omnibus/gitlab-mattermost/ ?

[szantnerb]

@esethna

1. interestingly the port was not there in the first place (https:// but no :443 at the end of the mattermost['service_site_url'] setting)
2. also tried commenting out mattermost['service_site_url']

Not sure if this is of any importance: the gitlab / mattermost is behind an nginx reverse proxy (another vm) which terminates the SSL/TLS (redirects http to https); the gitlab authenticates from ldap.

@CollenK indeed, I do the reconfigure after editing /etc/gitlab/gitlab.rb

/var/log/gitlab/gitlab-rails/production.log

Started POST "/oauth/authorize" for [client ip redacted] at 2017-05-30 10:00:18 +0200
Processing by Oauth::AuthorizationsController#create as HTML
  Parameters: {"utf8"=>"â..", "authenticity_token"=>"[redacted]", "client_id"=>"[redacted]", "redirect_uri"=>"https://mattermost.example.com/signup/gitlab/complete", "state"=>"[redacted]", "response_type"=>"code", "scope"=>"api", "nonce"=>""}
Redirected to https://mattermost.example.com/signup/gitlab/complete?code=[redacted]&state=[redacted]
Completed 302 Found in 28ms (ActiveRecord: 4.2ms)
Started POST "/oauth/token" for [gitlab local ip] at 2017-05-30 10:00:18 +0200
Processing by Doorkeeper::TokensController#create as JSON
  Parameters: {"client_id"=>"[redacted]", "client_secret"=>"[FILTERED]", "code"=>"[FILTERED]", "grant_type"=>"authorization_code", "redirect_uri"=>"http://mattermost.example.com/signup/gitlab/complete"}
Completed 401 Unauthorized in 6ms

note: not sure why redirect_uri is not https here

/var/log/gitlab/mattermost/current

/signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=wtsyryu3eirfxqzih4ns6uwrfe uid= ip=[redacted] Bad token type [details: token_type=, response_body=]

[esethna]

@bszantner I think your custom set up may be the source of these issues. This error is known to happen if there is a mismatch of http/https in the Mattermost user api/Auth/Token endpoints. Can you verify that they are correctly set to http/https?

[esethna]

@matfiz it should not break any Mattermost functionality. We have an MR open to update the docs, thanks for your help.

[szantnerb]

@esethna I checked again to be sure, and even in the generated /var/opt/gitlab/mattermost/config.json file: the auth, token and userapi endpoint gitlab urls all use https://.
The only strange thing in the /var/log/gitlab/gitlab-rails/production.log log is still the "redirect_uri"=>"http://mattermost.example.com/signup/gitlab/complete"
In the Gitlab application I have both callback urls use https://

https://mattermost.example.com/signup/gitlab/complete
https://mattermost.example.com/login/gitlab/complete

Anything else I could check?

[romantomchak]

Hi @esethna and @bszantner,
I have had the same issue. I'm also using Gitlab and Mattermost behind an nginx reverse proxy and I have had the same log output as in @bszantner and problem was in Mattermost proxy configuration.

By default Gitlab provide proxy header X-Forwarded-Proto with valuehttp but in this case mattermost is not correctly detects a http schema and generates url with http. I have changed this value to https and it's solved my issue.

My current configuration is:

mattermost_nginx['proxy_set_headers'] = {
   "X-Forwarded-Proto" => "https",
   "X-Forwarded-Ssl" => "on"
}

But I have another issue. I cannot login via Gitlab from MacOS desktop application.
From the application I choose "Sign in with Gitlab" and I successfully redirects to the Gitlab login page but when I submit login form I'm always receiving error message "Invalid login or password".

But these credentials are correct and login works from browser.
In my /var/log/gitlab/gitlab-rails/production.log file I have this:

Started POST "/users/sign_in" for <my IP>t 2017-06-02 23:45:55 +0200
Processing by SessionsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"lu8TPA1i3lDAgZ8Y0+cg+CP7L5qNo1NFsDauyi7oIGzt5ZFuOG2m4MzpEDfxQCEnsdvCqiODh/jRloEY1iQkAw==", "user"=>{"login"=>"roman", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 401 Unauthorized in 244ms (ActiveRecord: 17.7ms)
Processing by SessionsController#new as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"lu8TPA1i3lDAgZ8Y0+cg+CP7L5qNo1NFsDauyi7oIGzt5ZFuOG2m4MzpEDfxQCEnsdvCqiODh/jRloEY1iQkAw==", "user"=>{"login"=>"roman", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 200 OK in 117ms (Views: 16.2ms | ActiveRecord: 2.2ms)

[szantnerb]

@romantomchak thank you for the tip, my nginx reverse proxy was indeed missing the X-Forwarded-Proto header config. Manually setting it to https in the mattermost vhost solved this issue for me.

@esethna I also tried to re-enable (uncomment) the mattermost['service_site_url'], and it causes no problem now with my custom setup.

[esethna]

Hi @romantomchak, thanks for your resolution!

Regarding your issue with GitLab login on the desktop application can you please help to submit an issue to the desktop repo with repro steps and any additional info that might help debug?

[marc0s]

I can confirm the issue of the wrong X-Forwarded-Proto header. Setting it to https made it work.

[lindy65]

Thanks @romantomchak and @marc0s

It seems this issue is solved for now so I'll close this off... Feel free to reopen the issue if you still experience problems.

[clauded]

I have the problem with v9.3.4 : my Gitlab/Mattermost is behind an HAproxy server with SSL termination. The HAproxy is configured with "reqadd X-Forwarded-Proto:\ https" and all http requests are redirected to https. I also tried to add X-Forwarded-Proto https to the nginx configuration of Gitlab/Mattermost but still have the token error.

frontend www_frontend
    bind *:80
    # private key and server cert in same crt file
    bind *:443 ssl crt /etc/ssl/private/mysites.com.pem
    # HAProxy informs web server that there was a TLS Termination Proxy
    reqadd X-Forwarded-Proto:\ https
    # force-switch to HTTPS
    redirect scheme https if !{ ssl_fc }
    # Distinguish between secure and insecure requestsa (used in next two lines)
    acl secure dst_port eq 443
    # Mark all cookies as secure if sent over SSL
    rsprep ^Set-Cookie:\ (.*) Set-Cookie:\ \1;\ Secure if secure
    # Add the HSTS header with a 1 year max-age
    rspadd Strict-Transport-Security:\ max-age=31536000 if secure
    # gitlab
    acl host_gitlab hdr(host) -i gitlab.mysite.com
    use_backend gitlab if host_gitlab
    # gitlab mattermost
    acl host_mattermost hdr(host) -i mattermost.mysite.com
    use_backend mattermost if host_mattermost
backend gitlab
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server gitlab gitlab.local:80 check
backend mattermost
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server mattermost gitlab.local:8065 check

[elritsch]

@clauded, could it be that you are running into this issue: Mattermost and Gitlab behind SSL-terminating reverse proxy: token request fails because of local ip and HTTPS port usage

[clauded]

@elritsch, Thanks for the pointer. Here's how I finally resolved it :

mattermost_external_url 'http://mattermost.mysite.com'

mattermost['service_site_url'] = "https://mattermost.mysite.com"

mattermost['gitlab_auth_endpoint'] = "https://gitlab.mysite.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://gitlab.mysite.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://gitlab.mysite.com/api/v3/user"

# disable non-gitlab auth
mattermost['email_enable_sign_up_with_email'] = false
mattermost['email_enable_sign_in_with_email'] = false
mattermost['email_enable_sign_in_with_username'] = false
mattermost['email_send_email_notifications'] = true
mattermost['email_require_email_verification'] = false

Also, I had to adjust the Mattermost application settings in Gitlab to use https.

[james1293]

Based on faradaytrs's comment ...

> Solved when i removed port 443 from my mattermost_external_url config

... Does this mean it's not currently possible to use mattermost like so, with a port as part of the external url?

mattermost_external_url 'http://my.company.com:1234'

I had Mattermost set up like this (with a port as part of the URL) from April 21 to mid-May, and it was working. I recently tried logging into Mattermost after not using it for a while, and got the same error that others are getting, presumably because of an update.

EDIT: I got my config working again. I commented out everything mattermost related in the gitlab.rb config file except mattermost_external_url 'http://code.tsv.textron.com:1338' and it works!