Zero-to-jupyterhub-k8s: 400: Bad Request - OAuth state missing from cookies (Google auth)

[anton-khodak]

Hi,

I want to use Google authentication with JupyterHub. The problem is that cookie "oauthenticator" that get set up by JupyterHub disappears once Google redirects from oauth page back to Jupyter. Thus I see

403 : Forbidden 
You are not signed in to your jupyter.cellgeni.sanger.ac.uk account.

and on refresh

400 : Bad Request
OAuth state missing from cookies

I noticed that among other cookies "oauthenticator" one is the only one set with HttpOnly: true. This is the only reason I can think of to be invalidated since the expiration date for the cookie is one day ahead.

This is the cookie state after the server sets up cookies :

Once Google authenticated a user and returns them back to JupyterHub, the oauthenticator cookie just disappears. Other cookies stay.

I looked into https://github.com/jupyterhub/jupyterhub/issues/2044 and https://github.com/jupyterhub/jupyterhub/issues/1519, didn't help to understand how to fix this problem.

Hope someone can give a clue how to handle it.

Contents of config.yaml

proxy:
  secretToken: <token>

auth:
  type: google
  google:
    clientId: "<client-id>"
    clientSecret: "<secret>"
    callbackUrl: "https://jupyter.cellgeni.sanger.ac.uk/hub/oauth_callback"
    hostedDomain: "jupyter.cellgeni.sanger.ac.uk"
    loginService: "JupyterHub at Wellcome Trust Sanger Institute"

cull:
  timeout: 129600

singleuser:
  defaultUrl: "/lab"
  storage:
    capacity: 30Gi
  memory:
    limit: 20G
    guarantee: 16G
  cpu:
    limit: 4
    guarantee: 2
  image:
    name: quay.io/cellgeni/cellgeni-jupyter
    tag: v0.2.8
  lifecycleHooks:
    postStart:
      exec:
        command: ["bash", "/poststart.sh"]

[divatemangesh]

I am getting 500 Internal server error when autheticating with GITLAB-CE seems like authentication-state cookies expiration issue

cleared browser cookies and cache and any saved data and in New Tab opened https://localhost
security warning came like site is not safe I clicked on proceed safely
In chrome ctrl+shift+C>> application >> cookies I can see following entries
Sign in with gitlab page

Name:- xsrf
Value:- 2|d6b491b6|bf36d80df77faf9466583355e204eb77|1550066103
Expires:- 2020-03-15T13:55:03.000Z

Name:- username-localhost-8888
Value:-"2|1:0|10:1551431956|23:username-localhost-8888|44:MmU4ZDlmY2JkMzRkNDNjNWJhOTJiNmQ4NzNmYjA4NDA=|d5174b5df574432e334f4733f11943c12caf3cf3e05c4b47c7c962937c2a8960"
Expires:- 2020-03-31T09:19:16.000Z
-------------------------------- Once Clicked on Sign in with gitlab button ----------------------------

additional one entry came and disappered in less than half second and I landed to GitLab-CE login page but I screen recorded it to see the disappred entry

The one which disappeared was

Name :- oauthenticator-state
Value:- "2|1:0|10:1552127982|20:oauthenticator-state|120:Z... was not able to see comple value as it was partialy visiable in recording
_expires :- 2019-03-10T1 ..._

-------------------Once landed on gitlab-CE login-----------------------------------

Name:- _gitlab_session
Value:- ba46390addc98a11c22e4cd4dd064fdb
Expires:- 2019-03-09T12:39:42.873Z

---------------------loged in to gitlab with username-----------------------------
_gitlab_session
6dc3812c2eb2a0593ed8cb6c4f4f12b6
1969-12-31T23:59:59.000Z

--------------------Once Clicked on authosise in gitlab------------
Name:- _xsrf
Value :- 2|d6b491b6|bf36d80df77faf9466583355e204eb77|1550066103
Expireds:- 2020-03-15T13:55:03.000Z

Name:- username-localhost-8888
Value:- "2|1:0|10:1551431956|23:username-localhost-8888|44:MmU4ZDlmY2JkMzRkNDNjNWJhOTJiNmQ4NzNmYjA4NDA=|d5174b5df574432e334f4733f11943c12caf3cf3e05c4b47c7c962937c2a8960"
Expires:- 2020-03-31T09:19:16.000Z

GOT 500 INTERNAL SERVER ERROR

If issue is due authentication state cookies expiring before creating then how to solve it

[abdidarmawan007]

this is epic fail bug never fix lol

[consideRatio]

Hmmm, cookies are valid in various domains. I recently learned about what HTTP=true cookies imply and that should be fine still. It makes the cookie inaccessible from javascript but is still passed when making HTTP requests from the browser/client when GET/POST-ing etc to a webserver.

I'm not confident about what goes on, but cookies are things on the browser being passed to the webserver when making requests etc, and depending on the domain, different cookies are sent. So if facebook.com stores a cookie with HTTP=true on the browser by returning a "Set-Cookie" response header, it will be sent back to the webserver when the browser makes the next request back to facebook.com, but only facebook.com

So, if you browse what cookies are available while on google.com, those relates to google.com, and google.com will certainly store various auth related cookies, but jupyterhub wants to store a separate one i assume.

Hmmm, questions:

• what does it say in the hub logs when making the getting the 403 error?
• what version of the helm chart are you using?
• does upgrading to --version=0.9-8463734 resolve the issues? (Includes #1185 and #1171)
  UPDATE: No, the issue is resolved in --version=0.9-9c580a1 though I think.

Info:

• There is a bug fixed with google auth by @joshbode in #1185, I'm not at all confident this is the same issue though, but could perhaps be figured out by looking in the hub pod's logs.
• I fixed some bugs in #1171 (that impact gitlab @divatemangesh, but not googles authenticator I think @anton-khodak)

[nscozzaro]

I'm also experiencing this issue:

[consideRatio]

Is there someone having a functional google authenticator up and running?

[nscozzaro]

Not sure if it's related, but while trying to debug this I ran helm upgrade ... again. My hub then started going into a crash loop, so to debug I ran kubectl logs hub-8576fdcd66-v599d (that's the name of my hub pod), and I see the following traceback:

>>> kubectl logs hub-8576fdcd66-v599d
Loading /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
[I 2019-08-17 20:04:39.343 JupyterHub app:1673] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-0.8.0
[I 2019-08-17 20:04:39.343 JupyterHub app:1673] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2019-08-17 20:04:39.344 JupyterHub app:1016] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[I 2019-08-17 20:04:39.362 JupyterHub dbutil:125] Upgrading sqlite:///jupyterhub.sqlite
[I 2019-08-17 20:04:39.363 JupyterHub dbutil:105] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2019-08-17-200439
[I 2019-08-17 20:04:39.972 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
[I 2019-08-17 20:04:39.973 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
FAILED: Can't locate revision identified by '4dc2d5a8c53c'
[E 2019-08-17 20:04:39.977 alembic.util.messaging messaging:60] Can't locate revision identified by '4dc2d5a8c53c'
[E 2019-08-17 20:04:40.092 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1955, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1680, in initialize
        self.init_db()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1073, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/dbutil.py", line 130, in upgrade_if_needed
        upgrade(db_url)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/dbutil.py", line 89, in upgrade
        ['alembic', '-c', alembic_ini, 'upgrade', revision]
      File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpvq9ylmnn/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 255.

[nscozzaro]

Is there someone having a functional google authenticator up and running?

@consideRatio, I now have a functional Google authenticator running.

What worked was to use the latest development release of the helm chart (--version 0.9-470ec04) and to remove "hostedDomain" setting for Google auth in the config.yaml

And, to solve the error I was getting above I added the following to config.yaml

hub:
  db:
    upgrade: true

which I tried because I saw a similar issue here: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1244

[moki298]

Is there someone having a functional google authenticator up and running?

Yes I do

[pbadenski]

Is there plans to get this out in stable release version?

[consideRatio]

I dont have a clear idea about when this issue occur or why yet :/

[kyleatmousera]

I had this same error "400: Bad Request - OAuth state missing from cookies" when using (Google auth)
I found that removing "hostedDomain" setting for Google auth in the config.yaml worked for helm chart version 0.8.2.

[csudanthi]

had to remove hostedDomain as well also helm chart version 0.8.2.

[consideRatio]

I'll go ahead and close this issue for now, please open a new issue if this fails on 0.9.0+.