Zero-to-jupyterhub-k8s: GitLab OAuth require api scope in 0.9.0-beta.3, but didn't before

[conet]

I was running the version 0.9-484ce8e for a while now without problems and I thought it's time for an update, when I ran 0.9.0-beta.3 everything works properly for logged in users even the login works if it's not oauh, I have one instance where I have gitlab ouath defined but that fails on 0.9.0-beta.3 even though it works without issues on 0.9-484ce8e.
Here is the error log:

[I 2020-01-23 06:53:35.572 JupyterHub log:174] 200 GET /hub/login (@79.114.17.155) 28.34ms
[I 2020-01-23 06:53:37.399 JupyterHub oauth2:100] OAuth redirect: 'https://jhub-host/hub/oauth_callback'
[I 2020-01-23 06:53:37.400 JupyterHub log:174] 302 GET /hub/oauth_login?next= -> https://auth-host/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Fjhub-host%2Fhub%2Foauth_callback&client_id=[secret]&state=[secret]&scope=read_user (@79.114.17.155) 1.98ms
[E 2020-01-23 06:53:38.007 JupyterHub web:1788] Uncaught exception GET /hub/oauth_callback?code=[secret]&state=[secret] (79.114.17.155)
    HTTPServerRequest(protocol='https', host='jhub-host', method='GET', uri='/hub/oauth_callback?code=[secret]&state=[secret]', version='HTTP/1.1', remote_ip='79.114.17.155')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/oauthenticator/oauth2.py", line 207, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 699, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.6/dist-packages/oauthenticator/gitlab.py", line 130, in authenticate
        self.gitlab_version = await self._get_gitlab_version(access_token)
      File "/usr/local/lib/python3.6/dist-packages/oauthenticator/gitlab.py", line 182, in _get_gitlab_version
        resp = await AsyncHTTPClient().fetch(req, raise_error=True)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden

[E 2020-01-23 06:53:38.015 JupyterHub log:166] {
      "Cookie": "_ga=[secret]; _xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Language": "en-US,en;q=0.9,ro;q=0.8,hu;q=0.7",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://jhub-host/hub/login",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "Sec-Fetch-User": "?1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "X-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "jhub-host",
      "X-Forwarded-For": "79.114.17.155,10.36.135.2",
      "X-Real-Ip": "79.114.17.155",
      "X-Request-Id": "6b24da3985ea79be4487d10d9ad5889a",
      "Host": "jhub-host",
      "Connection": "close"
    }
[E 2020-01-23 06:53:38.016 JupyterHub log:174] 500 GET /hub/oauth_callback?code=[secret]&state=[secret] (@79.114.17.155) 371.69ms

Seems like the internal callback fails with a HTTP 403: Forbidden any ideas why?

[conet]

It seems like this issue https://github.com/jupyterhub/oauthenticator/issues/322#issuecomment-567813243 let me try the proposed fix.

[conet]

I see that the problem appeared in this commit https://github.com/jupyterhub/oauthenticator/commit/712617b5c23b7cc5ae63adaaf851e0d9ff1466f3

[consideRatio]

@vindvaki, I think this relates to the PR we spent time on here: https://github.com/jupyterhub/oauthenticator/pull/283

Is it perhaps so that this issue relates to permission to check gitlabs API on the version endpoint?

[conet]

Sounds like the read_user scope is not enough to do the version check, looking at gitlab scopes I don't know which extra scope to enable to allow the call to the version api, api would probably work but I'm reluctant to do that because:

Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry.

I don't see why auth would require such extended permissions.

[consideRatio]

Yes it seems like the access_token received cause the request to /api/v4/version respond with HTTP 403: Forbidden. Another response could have been 401 Unauthorized if the identity of the person trying to access this was unknown, but now it is simply forbidden.

@conet, could you try visiting your gitlab instance /api/v4 after signing in as a non-admin user on gitlab to see if you get a proper response?

[conet]

The /api/v4/version properly returns the version when accessed by a regular (non-admin) user.

[consideRatio]

@vindvaki it seems that https://github.com/jupyterhub/oauthenticator/pull/283 escalated the required scope for the requested access_token in order to function properly. Now I wonder, what is the minimal scope needed currently. Do you know what the minimal need would be? What do you think of the security implications?

I note in https://gitlab.com/gitlab-org/gitlab/blob/76015b67667f51bfdd289d9a1754c3ca48ffc549/spec/models/clusters/applications/jupyter_spec.rb#L101, that GitLab now requests a more permissive scope.

[manics]

@consideRatio Do you know if there's a list of all available GitLab OAuth scopes? I can't find anything in the GitLab docs.

[conet]

I got this from gitlab's UI:
Scopes:
api
Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry.
read_user
Grants read-only access to the authenticated user's profile through the /user API endpoint, which includes username, public email, and full name. Also grants access to read-only API endpoints under /users.
read_repository
Grants read-only access to repositories on private projects using Git-over-HTTP or the Repository Files API.
write_repository
Grants read-write access to repositories on private projects using Git-over-HTTP (not using the API).
sudo
Grants permission to perform API actions as any user in the system, when authenticated as an admin user.
openid
Grants permission to authenticate with GitLab using OpenID Connect. Also gives read-only access to the user's profile and group memberships.
profile
Grants read-only access to the user's profile data using OpenID Connect.
email
Grants read-only access to the user's primary email address using OpenID Connect.

[manics]

https://docs.gitlab.com/ee/api/v3_to_v4.html

Since GitLab 9.0, API V4 is the preferred version to be used.

Do you know when v4 was first supported? If it's old enough we could always assume v4 and avoid the version check?

[conet]

@manics isn't that something else handled by this? https://github.com/jupyterhub/oauthenticator/blob/master/oauthenticator/gitlab.py#L51
The new check looks for >=12.4 which is something else.

[manics]

@conet Sorry, you're right!

[consideRatio]

The check for >=12.4 was added as a the logic depends on the GitLab version currently.

@conet, if you can manipulate the scopes easily, can you verify that adding read_user,openid,profile,email isn't enough either?

I typically ended up requesting these scopes as part of authenticating with services, even though it isn't well documented.

Further, I know that if the /versions endpoint is OK, it still would need to access /groups/<group_id>/members/<user_id> which is not part of the /users endpoint, but perhaps that is included with the openid scope after all.

[conet]

read_user,openid,profile,email is not enough only api,read_user,openid,profile,email works.

[consideRatio]

I would very much like to know if we are able to access the /groups endpoint using an access token as received by requesting read_user,openid,profile,email. I hoped to get myself one using the GitLab UI under my user settings -> access tokens, but i cannot select openid,profile,email from there.

[conet]

openid,profile,email only appear in OAuth context because those are standard OAuth scopes and do not refer to accessing gitlab API. I have no way of checking the call to groups using the OAuth token since I don't have access to the token. But if were to guess if /version is not accessible then neither will /groups be. I'm not exactly sure what the fix was for but could it be written like this: only do the new feature when the proper scope api is available?

[consideRatio]

@conet excellent summary. Yes I agree, we should support authentication without forcing need for api scope. I've made this issue about /version endpoint of gitlab's API, but it is simply the tip of the iceberg.

[conet]

I just noticed this PR https://github.com/pusher/oauth2_proxy/pull/231 of a similar project, the part that got my attention was:

... group memberships can be retrieved without requiring the whole api scope. Instead it only requires the openid, profile and email scopes (which were already used by default).

isn't group membership what you were trying to obtain with the new calls?

[manics]

@conet Good find! It looks like oauth2_proxy uses the /oauth/userinfo endpoint which includes the lists of groups the user belongs to https://docs.gitlab.com/ee/integration/openid_connect_provider.html

[consideRatio]

isn't group membership what you were trying to obtain with the new calls?

Will you receive group membership information by requesting the groups scope in the OAuth2 request to GitLab as an OAuth2 provider, and can this be used instead of manually interacting with the gitlab REST API?

I think the answer is "no/maybe" for two reasons:

1. I'm not sure, but I don't think a registered GitLab OAuth2 application will respond with information about group membership when passed the groups scope. I think it will ignore that request. But I don't know.
2. You can be a member of a group in different ways, like a reporter, maintainer, owner etc, and I wonder if this is used to determine access.

[vindvaki]

/oauth/userinfo is a great find, but unfortunately it only includes group names (as opposed to the different access levels), and does not cover project membership at all

[vindvaki]

We just merged https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25211, allowing read_user to access /version. This will be included in GitLab 12.9, and live on GitLab.com some time in the next couple of weeks

[consideRatio]

This is largely an issue about the escalation of permissions required to work with the GitLabOAuthenticator. Due to this, I'm closing it here, and opening https://github.com/jupyterhub/oauthenticator/issues/372 with a summary of what we have concluded.

While documenting that issue, I learned that there is now a read_api scope that we can request, which is far less problematic than the api. That should be sufficient for us!

For this repo, I think we need to update the Authenticator documentation to match what we learn and document in the oauthenticator repo, but I'm closing this issue and making a note of that in my issue triage report.