Binderhub: Containers stuck at ContainerCreating

[ageorgou]

I have set up a BinderHub using a modified version of the instructions at https://zero-to-jupyterhub.readthedocs.io. I can load up the main page, but the process does not advance beyond the "Waiting" status, and the log there just shows "Waiting for build to start...", even after waiting more than an hour.

Looking at the pods, I see a container whose name starts with build-binder, which has been stuck at the ContainerCreating status. Strangely, it appears in the default namespace, even though everything else (BinderHub, JupyterHub) is under a different namespace (sheff-hub).

I can't access its log, but describe shows that mounting a volume has failed, because of a non-existent secret:

MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found

(full output attached at end)

Indeed, that secret does not exist in the default namespace, but it does in sheff-hub:

NAMESPACE     NAME                                  TYPE                                  DATA   AGE
default       default-token-2fq72                   kubernetes.io/service-account-token   3      4h
sheff-hub     binder-push-secret                    Opaque                                1      2h
sheff-hub     binder-secret                         Opaque                                2      2h
sheff-hub     binderhub-token-wkfzx                 kubernetes.io/service-account-token   3      2h
sheff-hub     default-token-zcdbw                   kubernetes.io/service-account-token   3      4h
sheff-hub     hub-secret                            Opaque                                2      2h
sheff-hub     hub-token-6dd7s                       kubernetes.io/service-account-token   3      2h
sheff-hub     sheff-hub-image-cleaner-token-kfmh5   kubernetes.io/service-account-token   3      2h

(table snipped to remove secrets from kube-system, kube-public namespaces)

This is on OS X, with

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:35:19Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

I also had to use the workaround in #809, because I was encountering the same error when trying to install binderhub, but am not getting the authentication error now.

Thanks in advice for any insight or advice!

---

This is the full output of describing the container:

$ kubectl describe pod build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Name:               build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-nodepool1-33022287-1/10.240.0.6
Start Time:         Mon, 18 Mar 2019 15:03:56 +0000
Labels:             component=binderhub-build
                    name=build-binder-2dexamples-2drequirements-55ab5c-a73ba1
Annotations:        binder-repo: https://github.com/binder-examples/requirements
Status:             Pending
IP:                 
Containers:
  builder:
    Container ID:  
    Image:         jupyter/repo2docker:0.8.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      jupyter-repo2docker
      --ref
      a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --image
      binder-2dexamples-2drequirements-55ab5c:a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
      --no-clean
      --no-run
      --json-logs
      --user-name
      jovyan
      --user-id
      1000
      --push
      https://github.com/binder-examples/requirements
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  0
    Requests:
      memory:  0
    Environment:
      KUBERNETES_PORT_443_TCP_ADDR:  sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
      KUBERNETES_PORT:               tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:       tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:       sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
    Mounts:
      /root/.docker from docker-push-secret (rw)
      /var/run/docker.sock from docker-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2fq72 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  docker-socket:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  docker-push-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-push-secret
    Optional:    false
  default-token-2fq72:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2fq72
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                   From                               Message
  ----     ------       ----                  ----                               -------
  Normal   Scheduled    83m                   default-scheduler                  Successfully assigned default/build-binder-2dexamples-2drequirements-55ab5c-a73ba1 to aks-nodepool1-33022287-1
  Warning  FailedMount  7m59s (x45 over 83m)  kubelet, aks-nodepool1-33022287-1  MountVolume.SetUp failed for volume "docker-push-secret" : secrets "binder-push-secret" not found
  Warning  FailedMount  118s (x36 over 81m)   kubelet, aks-nodepool1-33022287-1  Unable to mount volumes for pod "build-binder-2dexamples-2drequirements-55ab5c-a73ba1_default(0d3243e8-498f-11e9-8d40-fe8b4ef2ccd8)": timeout expired waiting for volumes to attach or mount for pod "default"/"build-binder-2dexamples-2drequirements-55ab5c-a73ba1". list of unmounted volumes=[docker-push-secret]. list of unattached volumes=[docker-socket docker-push-secret default-token-2fq72]

[consideRatio]

Hmmm, I dont know why the pod has ended up there _really_, but it is because its replicaset is there i assume, and the replicaset is there because its deployment is there, but why is the deployment located in the default namespace?

Did you install the deployment without using helm? helm will add the namespace field to the deployment's metadata.namespace field, but without it added it will install in default

I know a dirty fix that only cures this symptom but probably will just could be the first of many symptoms of something that needs fixing. You could edit the deployments namespace with kubectl edit.

Oh oh wait this pod is perhaps created by binderhub and isnt managed by a deployment/replicaset...

Hmmm then why did binderhub not start the pod in the correct namespace? Hmmm... How does binderhub know what namespace it should spawn things in? Probably by being deployed with helm and allowing its template to render {{ .Release.Namespace }} or something to an environment variable it reads. Then, again id wonder if you perhaps installed this without rendering the templates using helm.

A stream of thoughts written down without edit from a mobile, from a person sitting on a train, on a baggage shelf, in sweden, the world, in the milkyway, in our cluster of galaxies.

:heart:

[betatim]

Oh oh wait this pod is perhaps created by binderhub and isnt managed by a deployment/replicaset...

Correct, the build pods are launched by BinderHub itself. The namespace is passed in from the outside

https://github.com/jupyterhub/binderhub/blob/67d7d684c558e86c4dbfa8c8142e48b5747af703/binderhub/app.py#L292-L300

Which I think is set here so I'd kubectl describe deployment binder to see what you see.

[ageorgou]

@consideRatio Thanks for the quick reply and insight!
@betatim So the pod should be in the same namespace as the other pods/deployments, right? Here is the description of binder deployment:

$ kubectl describe deployment binder -n sheff-hub
Name:                   binder
Namespace:              sheff-hub
CreationTimestamp:      Mon, 18 Mar 2019 13:45:22 +0000
Labels:                 app=binder
                        component=binder
                        heritage=Tiller
                        name=binder
                        release=sheff-hub
Annotations:            deployment.kubernetes.io/revision: 4
Selector:               app=binder,component=binder,heritage=Tiller,name=binder,release=sheff-hub
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:
  Labels:           app=binder
                    component=binder
                    heritage=Tiller
                    name=binder
                    release=sheff-hub
  Annotations:      checksum/config-map: 026f886f0165bb14a780e16d3f209f1e1cfed2be1cffed1f112cc136eb33eff9
                    checksum/secret: 33529552e727cf341b2b6cb0cd31999bed31f7be35bdbf9bf79ac7ad9af325d3
  Service Account:  binderhub
  Containers:
   binder:
    Image:      jupyterhub/k8s-binderhub:cddf018
    Port:       8585/TCP
    Host Port:  0/TCP
    Requests:
      cpu:     200m
      memory:  512Mi
    Liveness:  http-get http://:binder/about delay=10s timeout=10s period=5s #success=1 #failure=3
    Environment:
      BUILD_NAMESPACE:        (v1:metadata.namespace)
      JUPYTERHUB_API_TOKEN:  <set to the key 'binder.hub-token' in secret 'binder-secret'>  Optional: false
    Mounts:
      /etc/binderhub/config/ from config (rw)
      /etc/binderhub/secret/ from secret-config (rw)
      /root/.docker from docker-secret (ro)
  Volumes:
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      binder-config
    Optional:  false
   secret-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-secret
    Optional:    false
   docker-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  binder-push-secret
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   binder-5cb597bdd4 (1/1 replicas created)
Events:          <none>

Sorry, I can't figure out how to see what v1:metadata refers to. There is a binder configmap but I don't see a relevant entry in there. Or is the value literally "(v1:metadata.namespace)"?

[consideRatio]

@ageorgou it probably isnt literally "(v1:metadata.namespace)" but a reference to whatever value of the deployments "metadata.namespace" field is.

I wonder what it sais if you write:

# get the technical representation of what the describe command summarizes
kubectl get deploy -o yaml binder

# get the same stuff, but for the specific pod created by the deployment rather than the actual deployment
kubectl get pod -o yaml -l component=binder

[betatim]

You can also "ssh into the pod" with kubectl exec -it binder-full-pod-name-here -n sheff-hub /bin/sh and then look at the value of the env variable with echo $BUILD_NAMESPACE.

[ageorgou]

Output of the first check:

$  kubectl get deploy -o yaml binder -n sheff-hub
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
  creationTimestamp: "2019-03-18T13:45:22Z"
  generation: 4
  labels:
    app: binder
    component: binder
    heritage: Tiller
    name: binder
    release: sheff-hub
  name: binder
  namespace: sheff-hub
  resourceVersion: "14552"
  selfLink: /apis/extensions/v1beta1/namespaces/sheff-hub/deployments/binder
  uid: 138b9f7a-4984-11e9-8d40-fe8b4ef2ccd8
spec:
  progressDeadlineSeconds: 2147483647
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: binder
      component: binder
      heritage: Tiller
      name: binder
      release: sheff-hub
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config-map: 026f886f0165bb14a780e16d3f209f1e1cfed2be1cffed1f112cc136eb33eff9
        checksum/secret: 33529552e727cf341b2b6cb0cd31999bed31f7be35bdbf9bf79ac7ad9af325d3
      creationTimestamp: null
      labels:
        app: binder
        component: binder
        heritage: Tiller
        name: binder
        release: sheff-hub
    spec:
      containers:
      - env:
        - name: BUILD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: JUPYTERHUB_API_TOKEN
          valueFrom:
            secretKeyRef:
              key: binder.hub-token
              name: binder-secret
        image: jupyterhub/k8s-binderhub:cddf018
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /about
            port: binder
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 10
        name: binder
        ports:
        - containerPort: 8585
          name: binder
          protocol: TCP
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/binderhub/config/
          name: config
        - mountPath: /etc/binderhub/secret/
          name: secret-config
        - mountPath: /root/.docker
          name: docker-secret
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: binderhub
      serviceAccountName: binderhub
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: binder-config
        name: config
      - name: secret-config
        secret:
          defaultMode: 420
          secretName: binder-secret
      - name: docker-secret
        secret:
          defaultMode: 420
          secretName: binder-push-secret
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2019-03-18T13:47:53Z"
    lastUpdateTime: "2019-03-18T13:47:53Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 4
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

and the second:

$  kubectl get pod -o yaml -l component=binder -n sheff-hub
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      checksum/config-map: 026f886f0165bb14a780e16d3f209f1e1cfed2be1cffed1f112cc136eb33eff9
      checksum/secret: 33529552e727cf341b2b6cb0cd31999bed31f7be35bdbf9bf79ac7ad9af325d3
    creationTimestamp: "2019-03-18T14:23:39Z"
    generateName: binder-5cb597bdd4-
    labels:
      app: binder
      component: binder
      heritage: Tiller
      name: binder
      pod-template-hash: "1761536880"
      release: sheff-hub
    name: binder-5cb597bdd4-75gm8
    namespace: sheff-hub
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: binder-5cb597bdd4
      uid: 6ce5fccf-4989-11e9-8d40-fe8b4ef2ccd8
    resourceVersion: "14539"
    selfLink: /api/v1/namespaces/sheff-hub/pods/binder-5cb597bdd4-75gm8
    uid: 6cecbec7-4989-11e9-8d40-fe8b4ef2ccd8
  spec:
    containers:
    - env:
      - name: BUILD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      - name: JUPYTERHUB_API_TOKEN
        valueFrom:
          secretKeyRef:
            key: binder.hub-token
            name: binder-secret
      - name: KUBERNETES_PORT_443_TCP_ADDR
        value: sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
      - name: KUBERNETES_PORT
        value: tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      - name: KUBERNETES_PORT_443_TCP
        value: tcp://sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io:443
      - name: KUBERNETES_SERVICE_HOST
        value: sheffhubcl-shefftesthub-eb9e00-2228b749.hcp.westeurope.azmk8s.io
      image: jupyterhub/k8s-binderhub:cddf018
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /about
          port: binder
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 10
      name: binder
      ports:
      - containerPort: 8585
        name: binder
        protocol: TCP
      resources:
        requests:
          cpu: 200m
          memory: 512Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /etc/binderhub/config/
        name: config
      - mountPath: /etc/binderhub/secret/
        name: secret-config
      - mountPath: /root/.docker
        name: docker-secret
        readOnly: true
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: binderhub-token-wkfzx
        readOnly: true
    dnsPolicy: ClusterFirst
    nodeName: aks-nodepool1-33022287-0
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: binderhub
    serviceAccountName: binderhub
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - configMap:
        defaultMode: 420
        name: binder-config
      name: config
    - name: secret-config
      secret:
        defaultMode: 420
        secretName: binder-secret
    - name: docker-secret
      secret:
        defaultMode: 420
        secretName: binder-push-secret
    - name: binderhub-token-wkfzx
      secret:
        defaultMode: 420
        secretName: binderhub-token-wkfzx
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2019-03-18T14:23:39Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2019-03-18T14:23:44Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2019-03-18T14:23:39Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://2bea23261588c1a4a2743f3a579d743aaeef985d05e013a3f12580e57c431d9b
      image: jupyterhub/k8s-binderhub:cddf018
      imageID: docker-pullable://jupyterhub/k8s-binderhub@sha256:da3807eabee71d2e0518017894aa665a124205d46e3083937a4e37f3e1ed48d0
      lastState: {}
      name: binder
      ready: true
      restartCount: 0
      state:
        running:
          startedAt: "2019-03-18T14:23:43Z"
    hostIP: 10.240.0.4
    phase: Running
    podIP: 10.244.2.9
    qosClass: Burstable
    startTime: "2019-03-18T14:23:39Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[ageorgou]

The variable seems to be set correctly "in the pod":

# echo $BUILD_NAMESPACE
sheff-hub

[ageorgou]

Trying this again from scratch, it seems to work (the containers are created in the expected namespace). I was following the same instructions, and the only difference I can see is in the description of the binder deployment. I now have

Annotations:            deployment.kubernetes.io/revision: 2

instead of

Annotations:            deployment.kubernetes.io/revision: 4

(it's possible that I updated the deployment more times when I first tried this)

I guess I must have made a mistake somewhere along the way the first time, so this can be closed as far as I'm concerned!

[alexmorley]

Nice one @ageorgou . Are there any tricks for debugging that would be helpful for others? Otherwise I'll stick with the advice "can you try starting again from scratch" next time this happens :sweat_smile:

[ageorgou]

@alexmorley No ideas, unfortunately. The only thing that I noticed is that the build- containers were running in the wrong namespace, as above, and I got that by running get and describe on things that looked relevant... The logs were not particularly helpful as nothing was breaking there.

[betatim]

Thanks for chasing this done. I'll close this here now.

We can move this discussion to http://discourse.jupyter.org/ (where most of the people who hang out here also hang out, as well as more people) if you want to keep exchanging debugging tips. We are trying to streamline the different discussion places and want to use the issues on GitHub repos for technical discussions on how to change the contents of the repo. More general discussions should go to discourse so that they are easier to find, better indexed by google and generally more accessible than being hidden in the bowls of a GitHub repository ;)

Thanks!

[jtpio]

I wanted to share here what seems to be related to this issue (haven't found a related thread on discourse).

Testing with a fresh release of the BinderHub chart (version 0.2.0-5ca42ec and following the guide) on a Kubernetes cluster with rbac enabled.

The release is deployed in the binderhub namespace, but binder wants to list the pods in the default namespace, even though BUILD_NAMESPACE is correctly set to binderhub (the namespace where the chart is released):

$ kubectl exec -it binder-d998c657c-zmdf8 -- env | grep BUILD_NAMESPACE
BUILD_NAMESPACE=binderhub

root@binder-d998c657c-zmdf8:/# tr '\0' '\n' < /proc/1/environ | grep BUILD_NAMESPACE
BUILD_NAMESPACE=binderhub

The logs for the binder pod:

[E 190710 10:48:12 app:638] Failed to cleanup build pods
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/binderhub/app.py", line 630, in watch_build_pods
        lambda: Build.cleanup_builds(
      File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.6/site-packages/binderhub/app.py", line 633, in <lambda>
        self.build_max_age,
      File "/usr/local/lib/python3.6/site-packages/binderhub/build.py", line 91, in cleanup_builds
        label_selector='component=binderhub-build',
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod
        (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
        _return_http_data_only, collection_formats, _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request
        headers=headers)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET
        query_params=query_params)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (403)
    Reason: Forbidden
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 10 Jul 2019 10:48:12 GMT', 'Content-Length': '286'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:binderhub:binderhub\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Looking at the code, it looks like build_namespace should be correctly passed:

https://github.com/jupyterhub/binderhub/blob/01b1c59b9e7dc81250c1ed579c492ec2fd6baaf6/binderhub/app.py#L630-L634

The issue mentioned above seems to correspond to the step that comes after the binderhub serviceaccount has the correct rights on the default namespace.

[rochaporto]

I'm seeing this issue as well with the latest binder and deploying using the helm chart.

Is this followed up?

[betatim]

I don't think anyone has worked on this. Sounds like it is a problem with Python namespacing/class vs instance variables.

[ivan-gomes]

@jtpio @rochaporto did either of you end up finding a solution to this? I am running into this issue working with HEAD of master even now.

[jtpio]

@ivan-gomes I haven't tested since.

But if I remember correctly restarting some of the pods manually helped.

[ivan-gomes]

Thanks. For anyone stumbling here from Google, in my case the appearance of this error was a red herring. binderhub_config.py was throwing an error because of some custom extraConfig and the code that uses env variable $BUILD_NAMESPACE to set BinderHub.build_namespace was presumably never being reached.

[jtpio]

@ivan-gomes Just tried today a brand new BinderHub deployment on a new k8s cluster (on OVH), and facing the exact same issue as mentioned in https://github.com/jupyterhub/binderhub/issues/810#issuecomment-510020306.

Setting:

config:
  BinderHub:
    auth_enabled: false

helps get past the following error in /binderhub_config.py:

Loading /etc/binderhub/config/values.yaml
Loading /etc/binderhub/secret/values.yaml
[BinderHub] ERROR | Exception while loading config file /binderhub_config.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/application.py", line 563, in _load_config_files
    config = loader.load_config()
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/loader.py", line 457, in load_config
    self._read_file_as_dict()
  File "/usr/local/lib/python3.7/site-packages/traitlets/config/loader.py", line 489, in _read_file_as_dict
    py3compat.execfile(conf_filename, namespace)
  File "/usr/local/lib/python3.7/site-packages/ipython_genutils/py3compat.py", line 198, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "/binderhub_config.py", line 87, in <module>
    hub_url = urlparse(c.BinderHub.hub_url)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 367, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 123, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 107, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/usr/local/lib/python3.7/urllib/parse.py", line 107, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'LazyConfigValue' object has no attribute 'decode'

Setting:

extraConfig:
  01-custom: |
      c.BinderHub.build_namespace = "binderhub"

helps prevent Binder trying to list pods in the default namespace.

However permissions still seem to be off:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:binderhub:binderhub\" cannot list resource \"pods\" in API group \"\" in the namespace \"binderhub\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Although the binderhub Role and RoleBinding look good and correspond to the default from the chart:

https://github.com/jupyterhub/binderhub/blob/58a0b72021d17264519438f6e06f452021617a35/helm-chart/binderhub/templates/rbac.yaml#L2-L34

[jtpio]

Some more investigation.

Just tried the exact same steps, a fresh BinderHub on a brand new k8s cluster on GKE (using the default settings).

This time it works flawlessly, everything was up and running in a matter of minutes.

The main differences between the two setups are:

• the Kubernetes version: 1.18.1 on OVH (recommended setting), and 1.14.10-gke.36 on GKE (default).
• the cloud provider: OVH / GKE

Could it be an issue with the latest 1.18 version?

[jtpio]

Just tried a new setup on OVH (new cluster and new BinderHub), but this time choosing 1.15 (the lowest they offer at the moment) as the Kubernetes version.

And it's working out of the box without having to follow the steps mentioned in https://github.com/jupyterhub/binderhub/issues/810#issuecomment-651089134.

We should probably open a new issue to keep track of this. It looks like it is related to some rbac changes in the latest versions?

[xubofei1983]

seeing same issue in EKS 1.16

[alpha-nome]

Same issue on AKS 1.20.2.

Any clue on how to fix this? Thanks