Zero-to-jupyterhub-k8s: Give notebook containers elevated privileges

[mrocklin]

Hi,

I would like to use a FUSE file system in my containers. To do this locally it seems that I need to pass the /dev/fuse device to the docker container and run it with elevated permissions. Locally I do the following:

docker run -it --device /dev/fuse --cap-add SYS_ADMIN --privileged daskdev/pangeo-notebook

Is there a way for me to specify these flags in my config.yaml file?

[yuvipanda]

Try the following (using a v0.6.0- chart)

hub:
  extraConfig: |
    from kubernetes import client
    def modify_pod_hook(spawner, pod):
        pod.spec.containers[0].security_context = client.V1SecurityContext(
            privileged=True,
            capabilities=client.V1Capabilities(
                add=['SYS_ADMIN']
            )
        )
        return pod
    c.KubeSpawner.modify_pod_hook = modify_pod_hook

singleuser:
  uid: 0
  gid: 0
  storage:
    extraVolumes:
      - name: fuse
        hostPath:
          - path: /dev/fuse
    extraVolumeMounts:
      - name: fuse
        mountPath: /dev/fuse

[mrocklin]

OK, I've given this a shot and I'm now getting 500 errors. What is the right way to diagnose this?

[yuvipanda]

Where are you getting 500 from? Logs would be useful!

On Fri, Jan 5, 2018 at 12:12 PM, Matthew Rocklin [e-mail]
wrote:

OK, I've given this a shot and I'm now getting 500 errors. What is the
right way to diagnose this?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/379#issuecomment-355653838,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23jn0ygUoOZCaShWvqCwpBJxD4WNIks5tHoIzgaJpZM4RUtye
.

--
Yuvi Panda T
http://yuvi.in/blog

[mrocklin]

Directly after trying to log in.

500 : Internal Server Error
Failed to start your server. Please contact admin.

You can try restarting your server from the home page.

Logs

mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs hub-594749b96d-jj6v6 --namespace pangeo | tail -100
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}



[E 2018-01-05 20:16:31.077 JupyterHub log:114] {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Host": "35.188.222.190",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-For": "10.128.0.38,10.20.5.12",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Connection": "close",
      "X-Forwarded-Host": "35.188.222.190",
      "Accept": "*/*",
      "X-Forwarded-Port": "80,80",
      "X-Real-Ip": "10.128.0.38",
      "X-Original-Uri": "/hub/user/myuser/api/terminals?1515183390974"
    }
[E 2018-01-05 20:16:31.077 JupyterHub log:122] 500 GET /hub/user/myuser/api/terminals?1515183390974 ([email protected]) 6.12ms
[I 2018-01-05 20:16:35.390 JupyterHub log:122] 302 GET /user/myuser/api/contents/?content=1&1515183395350 → /hub/user/myuser/api/contents/?content=1&1515183395350 (@10.128.0.38) 0.93ms
[E 2018-01-05 20:16:35.462 JupyterHub base:711] Preventing implicit spawn for myuser because last spawn failed: (400)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}


[E 2018-01-05 20:16:35.463 JupyterHub web:1590] Uncaught exception GET /hub/user/myuser/api/contents/?content=1&1515183395350 (10.128.0.38)
    HTTPServerRequest(protocol='http', host='35.188.222.190', method='GET', uri='/hub/user/myuser/api/contents/?content=1&1515183395350', version='HTTP/1.1', remote_ip='10.128.0.38', headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'Cookie': 'jupyter-hub-token="2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'Host': '35.188.222.190', 'Accept-Language': 'en-US,en;q=0.9', 'X-Forwarded-Proto': 'http,http', 'X-Scheme': 'http', 'Referer': 'http://35.188.222.190/user/myuser/lab?redirects=1', 'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-For': '10.128.0.38,10.20.5.12', 'X-Xsrftoken': '2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'Connection': 'close', 'X-Forwarded-Host': '35.188.222.190', 'Accept': '*/*', 'X-Forwarded-Port': '80,80', 'X-Real-Ip': '10.128.0.38', 'X-Original-Uri': '/hub/user/myuser/api/contents/?content=1&1515183395350'})
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 713, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 747, in get
        yield self.spawn_single_user(current_user)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 439, in spawn
        raise e
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 378, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.5/dist-packages/kubespawner/spawner.py", line 917, in start
        pod
      File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
        return self.__get_result()
      File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
        raise self._exception
      File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubespawner/spawner.py", line 888, in asynchronize
        return method(*args, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6561, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6651, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 335, in call_api
        _preload_content, _request_timeout)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 148, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 393, in request
        body=body)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/rest.py", line 287, in POST
        body=body)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/rest.py", line 240, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (400)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}



[E 2018-01-05 20:16:35.471 JupyterHub log:114] {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Host": "35.188.222.190",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-For": "10.128.0.38,10.20.5.12",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Connection": "close",
      "X-Forwarded-Host": "35.188.222.190",
      "Accept": "*/*",
      "X-Forwarded-Port": "80,80",
      "X-Real-Ip": "10.128.0.38",
      "X-Original-Uri": "/hub/user/myuser/api/contents/?content=1&1515183395350"
    }
[E 2018-01-05 20:16:35.471 JupyterHub log:122] 500 GET /hub/user/myuser/api/contents/?content=1&1515183395350 ([email protected]) 10.68ms
mrocklin@carbon:~/workspace/pangeo/gce$ 

[yuvipanda]

@mrocklin looks like the error is in:

    extraVolumes:
      - name: fuse
        hostPath:
          - path: /dev/fuse

Specifically, the '-' before the last line. Can you replace with:

    extraVolumes:
      - name: fuse
        hostPath:
          path: /dev/fuse

and try?

[mrocklin]

Still failing

mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' 'gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git' ']'
+ echo 'EXTRA_PIP_PACKAGES environment variable found.  Installing.'
+ /opt/conda/bin/pip install gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git
no environment.yml
EXTRA_PIP_PACKAGES environment variable found.  Installing.
Collecting git+https://github.com/pydata/xarray.git
  Cloning https://github.com/pydata/xarray.git to /tmp/pip-528fg1jr-build
Collecting git+https://github.com/alimanfoo/zarr.git
  Cloning https://github.com/alimanfoo/zarr.git to /tmp/pip-bi40oti8-build
Requirement already satisfied: gcsfs in /opt/conda/lib/python3.6/site-packages
Requirement already satisfied: numpy>=1.11 in /opt/conda/lib/python3.6/site-packages (from xarray==0.10.0+dev24.g5a28b89)
Requirement already satisfied: pandas>=0.18.0 in /opt/conda/lib/python3.6/site-packages (from xarray==0.10.0+dev24.g5a28b89)
Collecting asciitree (from zarr==2.2.0a2.dev176)
Collecting fasteners (from zarr==2.2.0a2.dev176)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting numcodecs>=0.5.2 (from zarr==2.2.0a2.dev176)
Requirement already satisfied: oauth2client>=1.5.2 in /opt/conda/lib/python3.6/site-packages (from gcsfs)
Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from gcsfs)
Requirement already satisfied: python-dateutil>=2 in /opt/conda/lib/python3.6/site-packages (from pandas>=0.18.0->xarray==0.10.0+dev24.g5a28b89)
Requirement already satisfied: pytz>=2011k in /opt/conda/lib/python3.6/site-packages (from pandas>=0.18.0->xarray==0.10.0+dev24.g5a28b89)
Collecting monotonic>=0.1 (from fasteners->zarr==2.2.0a2.dev176)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from fasteners->zarr==2.2.0a2.dev176)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: httplib2>=0.9.1 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: rsa>=3.1.4 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: idna<2.7,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Installing collected packages: xarray, asciitree, monotonic, fasteners, numcodecs, zarr
  Found existing installation: xarray 0.10.0
    Uninstalling xarray-0.10.0:
      Successfully uninstalled xarray-0.10.0
  Running setup.py install for xarray: started
mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
Error from server (NotFound): pods "jupyter-mrocklin" not found

[mrocklin]

logs from the hub

[E 2018-01-05 20:40:11.087 JupyterHub log:122] 500 GET /hub/user/myuser/api/terminals?1515184810974 ([email protected]) 6.30ms
[I 2018-01-05 20:40:11.382 JupyterHub log:122] 302 GET /user/myuser/api/contents/?content=1&1515184811351 → /hub/user/myuser/api/contents/?content=1&1515184811351 (@10.128.0.41) 0.71ms
[E 2018-01-05 20:40:11.439 JupyterHub base:711] Preventing implicit spawn for myuser because last spawn failed: Server at http://10.20.4.28:8888/user/myuser/ didn't respond in 30 seconds
[E 2018-01-05 20:40:11.439 JupyterHub web:1590] Uncaught exception GET /hub/user/myuser/api/contents/?content=1&1515184811351 (10.128.0.41)
    HTTPServerRequest(protocol='http', host='35.188.222.190', method='GET', uri='/hub/user/myuser/api/contents/?content=1&1515184811351', version='HTTP/1.1', remote_ip='10.128.0.41', headers={'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-Proto': 'http,http', 'X-Scheme': 'http', 'Host': '35.188.222.190', 'X-Real-Ip': '10.128.0.41', 'Connection': 'close', 'Accept': '*/*', 'Referer': 'http://35.188.222.190/user/myuser/lab?redirects=1', 'Cookie': 'jupyter-hub-token="2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'X-Forwarded-For': '10.128.0.41,10.20.5.12', 'X-Original-Uri': '/hub/user/myuser/api/contents/?content=1&1515184811351', 'Accept-Language': 'en-US,en;q=0.9', 'X-Forwarded-Host': '35.188.222.190', 'X-Forwarded-Port': '80,80', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'X-Xsrftoken': '2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400'})
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 713, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 720, in get
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), spawner._spawn_future)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 476, in spawn
        raise e
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 450, in spawn
        resp = yield server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 180, in wait_for_http_server
        timeout=timeout
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 135, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://10.20.4.28:8888/user/myuser/ didn't respond in 30 seconds

[E 2018-01-05 20:40:11.444 JupyterHub log:114] {
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Host": "35.188.222.190",
      "X-Real-Ip": "10.128.0.41",
      "Connection": "close",
      "Accept": "*/*",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "X-Forwarded-For": "10.128.0.41,10.20.5.12",
      "X-Original-Uri": "/hub/user/myuser/api/contents/?content=1&1515184811351",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Host": "35.188.222.190",
      "X-Forwarded-Port": "80,80",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400"
    }
[E 2018-01-05 20:40:11.445 JupyterHub log:122] 500 GET /hub/user/myuser/api/contents/?content=1&1515184811351 ([email protected]) 6.84ms

[yuvipanda]

We spent some more time and figured it all out :)

[yuvipanda]

It was the singleuser.cmd, which we default to jupyterhub-singleuser. It needed to be set to start.sh instead I believe.

[avinson]

If anyone comes across this issue in the future, I've created a jupyter image and a helm config fragment for mounting GCS storage via fuse here: https://github.com/noahhomes/r-mkl-notebook