Zero-to-jupyterhub-k8s: proxy-public service stalled/pending

Created on 9 Apr 2018 · 20Comments · Source: jupyterhub/zero-to-jupyterhub-k8s

Seems like everything except the proxy is coming up fine - the proxy however just sits in a strange 'pending' state ... the hub pod appears to be running though .. not seeing any obvious errors in the dashboard... but on the worker node where the proxy is running, I see this in /var/log/messages:
```Apr 9 13:28:08 worker2 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/5beef49b-3c1b-11e8-9520-005056861e50/volumes/kubernetes.io~glusterfs/pvc-5bd3424a-3c1b-11e8-836d-005056861358.
Apr 9 13:28:08 worker2 systemd: Starting Kubernetes transient mount for /var/lib/kubelet/pods/5beef49b-3c1b-11e8-9520-005056861e50/volumes/kubernetes.io~glusterfs/pvc-5bd3424a-3c1b-11e8-836d-005056861358.
Apr 9 13:28:09 worker2 kernel: fuse init (API version 7.22)
Apr 9 13:28:09 worker2 systemd: Mounting FUSE Control File System...
Apr 9 13:28:09 worker2 systemd: Mounted FUSE Control File System.
Apr 9 13:28:09 worker2 kubelet: I0409 13:28:09.110993 14956 glusterfs.go:358] successfully mounted /var/lib/kubelet/pods/5beef49b-3c1b-11e8-9520-005056861e50/volumes/kubernetes.io~glusterfs/pvc-5bd3424a-3c1b-11e8-836d-005056861358
Apr 9 13:28:09 worker2 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Apr 9 13:28:09 worker2 NetworkManager[804]: [1523294889.4578] device (vetha55b5b00): link connected
Apr 9 13:28:09 worker2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Apr 9 13:28:09 worker2 NetworkManager[804]: [1523294889.4586] manager: (vetha55b5b00): new Veth device (/org/freedesktop/NetworkManager/Devices/8)
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered blocking state
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered disabled state
Apr 9 13:28:09 worker2 kernel: device vetha55b5b00 entered promiscuous mode
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered blocking state
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered forwarding state
Apr 9 13:28:09 worker2 kernel: IPv6: eth0: IPv6 duplicate address fe80::68bb:96ff:fe02:e821 detected!
Apr 9 13:28:14 worker2 dockerd: time="2018-04-09T13:28:14.462959354-04:00" level=error msg="Handler for GET /v1.27/images/jupyterhub/configurable-http-proxy:3.0.0/json returned error: No such image: jupyterhub/configurable-http-proxy:3.0.0"
Apr 9 13:28:18 worker2 dockerd: time="2018-04-09T13:28:18.696369148-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container e0e89f962727bf0be831255a365afa7be88aecd877678164937641e9c1bb2a43"
Apr 9 13:28:19 worker2 dockerd: time="2018-04-09T13:28:19.483472017-04:00" level=error msg="Handler for GET /v1.27/images/jupyterhub/k8s-hub:fad0f64/json returned error: No such image: jupyterhub/k8s-hub:fad0f64"
Apr 9 13:28:39 worker2 dockerd: time="2018-04-09T13:28:39.857479960-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container 0e7b60a13967633f20f21c2d706ae1455e32b7d0947bb0659890fd981a26ce53"
Apr 9 13:28:53 worker2 dockerd: time="2018-04-09T13:28:53.857246385-04:00" level=error msg="Handler for POST /v1.27/containers/aeb40444924add79272d644edc58066c901a53c9f1de810bc5fa12915d748f73/stop returned error: Container aeb40444924add79272d644edc58066c901a53c9f1de810bc5fa12915d748f73 is already stopped"

not sure if that's anything to do with it ... anyway, not sure how to debug this proxy... any ideas folks?

## Contents of `config.yaml`
```yaml
proxy:
    secretToken: "{{ _jhub_secret_token }}"
singleuser:
  image:
    name: jupyter/scipy-notebook
    tag: latest
  storage:
    dynamic:
      storageClass: gluster-sc
    capacity: 1Gi
  memory:
    limit: 1Gi
    guarantee: 1Gi
  cpu:
    limit: .5
    guarantee: .5

Source

miramar-labs

Most helpful comment

For future readers: I solved the problem by using MetalLB.

angrymeir on 11 Sep 2019

👍2

All 20 comments

Heya!

Can you provide output of:

kubectl --namespace=<namespace> describe pod <proxy-pod-name>
kubectl --namespace=<namespace> describe svc

Thanks!

On Mon, Apr 9, 2018 at 10:43 AM, Aaron Cody notifications@github.com
wrote:

Seems like everything except the proxy is coming up fine - the proxy
however just sits in a strange 'pending' state ... the hub pod appears to
be running though .. not seeing any obvious errors in the dashboard... but
on the worker node where the proxy is running, I see this in
/var/log/messages:

Apr 9 13:28:08 worker2 systemd: Starting Kubernetes transient mount for /var/lib/kubelet/pods/5beef49b-3c1b-11e8-9520-005056861e50/volumes/kubernetes.io~glusterfs/pvc-5bd3424a-3c1b-11e8-836d-005056861358.
Apr 9 13:28:09 worker2 kernel: fuse init (API version 7.22)
Apr 9 13:28:09 worker2 systemd: Mounting FUSE Control File System...
Apr 9 13:28:09 worker2 systemd: Mounted FUSE Control File System.
Apr 9 13:28:09 worker2 kubelet: I0409 13:28:09.110993 14956 glusterfs.go:358] successfully mounted /var/lib/kubelet/pods/5beef49b-3c1b-11e8-9520-005056861e50/volumes/kubernetes.io~glusterfs/pvc-5bd3424a-3c1b-11e8-836d-005056861358
Apr 9 13:28:09 worker2 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Apr 9 13:28:09 worker2 NetworkManager[804]: [1523294889.4578] device (vetha55b5b00): link connected
Apr 9 13:28:09 worker2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Apr 9 13:28:09 worker2 NetworkManager[804]: [1523294889.4586] manager: (vetha55b5b00): new Veth device (/org/freedesktop/NetworkManager/Devices/8)
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered blocking state
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered disabled state
Apr 9 13:28:09 worker2 kernel: device vetha55b5b00 entered promiscuous mode
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered blocking state
Apr 9 13:28:09 worker2 kernel: cni0: port 2(vetha55b5b00) entered forwarding state
Apr 9 13:28:09 worker2 kernel: IPv6: eth0: IPv6 duplicate address fe80::68bb:96ff:fe02:e821 detected!
Apr 9 13:28:14 worker2 dockerd: time="2018-04-09T13:28:14.462959354-04:00" level=error msg="Handler for GET /v1.27/images/jupyterhub/configurable-http-proxy:3.0.0/json returned error: No such image: jupyterhub/configurable-http-proxy:3.0.0"
Apr 9 13:28:18 worker2 dockerd: time="2018-04-09T13:28:18.696369148-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container e0e89f962727bf0be831255a365afa7be88aecd877678164937641e9c1bb2a43"
Apr 9 13:28:19 worker2 dockerd: time="2018-04-09T13:28:19.483472017-04:00" level=error msg="Handler for GET /v1.27/images/jupyterhub/k8s-hub:fad0f64/json returned error: No such image: jupyterhub/k8s-hub:fad0f64"
Apr 9 13:28:39 worker2 dockerd: time="2018-04-09T13:28:39.857479960-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container 0e7b60a13967633f20f21c2d706ae1455e32b7d0947bb0659890fd981a26ce53"
Apr 9 13:28:53 worker2 dockerd: time="2018-04-09T13:28:53.857246385-04:00" level=error msg="Handler for POST /v1.27/containers/aeb40444924add79272d644edc58066c901a53c9f1de810bc5fa12915d748f73/stop returned error: Container aeb40444924add79272d644edc58066c901a53c9f1de810bc5fa12915d748f73 is already stopped"```

not sure if that's anything to do with it ... anyway, not sure how to debug this proxy... any ideas folks?

Contents of config.yaml

proxy:
secretToken: "{{ _jhub_secret_token }}"
singleuser:
image:
name: jupyter/scipy-notebook
tag: latest
storage:
dynamic:
storageClass: gluster-sc
capacity: 1Gi
memory:
limit: 1Gi
guarantee: 1Gi
cpu:
limit: .5
guarantee: .5

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/641, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAB23tbDaefyYKTtTb42LmGimSpeVw7mks5tm53agaJpZM4TM8bK
.

--
Yuvi Panda T
http://yuvi.in/blog

yuvipanda on 9 Apr 2018

kubectl --namespace=jupyterhub describe pod proxy-54fbf87d4d-t9tgx Name: proxy-54fbf87d4d-t9tgx Namespace: jupyterhub Node: worker2/192.168.0.57 Start Time: Mon, 09 Apr 2018 13:28:03 -0400 Labels: component=proxy heritage=Tiller hub.jupyter.org/network-access-hub=true hub.jupyter.org/network-access-singleuser=true name=proxy pod-template-hash=1096943808 release=jupyterhub Annotations: checksum/hub-secret=3afa583a4382588e80f8b2ab9f095e5e2ea869960f2212191c080284be829458 checksum/proxy-secret=01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b Status: Running IP: 10.244.3.3 Controlled By: ReplicaSet/proxy-54fbf87d4d Containers: chp: Container ID: docker://e0e89f962727bf0be831255a365afa7be88aecd877678164937641e9c1bb2a43 Image: jupyterhub/configurable-http-proxy:3.0.0 Image ID: docker-pullable://jupyterhub/configurable-http-proxy@sha256:c36cf3cc1c99f59348a8d6f5f64752df3eb4d88df93a11bc4e00acf23dbecfba Ports: 8000/TCP, 8001/TCP Host Ports: 0/TCP, 0/TCP Command: configurable-http-proxy --ip=0.0.0.0 --api-ip=0.0.0.0 --api-port=8001 --default-target=http://$(HUB_SERVICE_HOST):$(HUB_SERVICE_PORT) --error-target=http://$(HUB_SERVICE_HOST):$(HUB_SERVICE_PORT)/hub/error --port=8000 State: Running Started: Mon, 09 Apr 2018 13:28:18 -0400 Ready: True Restart Count: 0 Requests: cpu: 200m memory: 512Mi Environment: CONFIGPROXY_AUTH_TOKEN: <set to the key 'proxy.token' in secret 'hub-secret'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-z2mwz (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: default-token-z2mwz: Type: Secret (a volume populated by a Secret) SecretName: default-token-z2mwz Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 21m default-scheduler Successfully assigned proxy-54fbf87d4d-t9tgx to worker2 Normal SuccessfulMountVolume 21m kubelet, worker2 MountVolume.SetUp succeeded for volume "default-token-z2mwz" Normal Pulling 21m kubelet, worker2 pulling image "jupyterhub/configurable-http-proxy:3.0.0" Normal Pulled 20m kubelet, worker2 Successfully pulled image "jupyterhub/configurable-http-proxy:3.0.0" Normal Created 20m kubelet, worker2 Created container Normal Started 20m kubelet, worker2 Started container

```kubectl --namespace=jupyterhub describe svc
Name: glusterfs-dynamic-hub-db-dir
Namespace: jupyterhub
Labels: gluster.kubernetes.io/provisioned-for-pvc=hub-db-dir
Annotations:
Selector:
Type: ClusterIP
IP: 10.98.232.70
Port: 1/TCP
TargetPort: 1/TCP
Endpoints: 192.168.0.31:1,192.168.0.32:1,192.168.0.33:1 + 3 more...
Session Affinity: None
Events:

Name: hub
Namespace: jupyterhub
Labels:
Annotations:
Selector: name=hub
Type: ClusterIP
IP: 10.96.107.41
Port: 8081/TCP
TargetPort: 8081/TCP
Endpoints: 10.244.3.4:8081
Session Affinity: None
Events:

Name: proxy-api
Namespace: jupyterhub
Labels:
Annotations:
Selector: component=proxy,name=proxy,release=jupyterhub
Type: ClusterIP
IP: 10.105.79.175
Port: 8001/TCP
TargetPort: 8001/TCP
Endpoints: 10.244.3.3:8001
Session Affinity: None
Events:

Name: proxy-public
Namespace: jupyterhub
Labels: chart=jupyterhub-v0.7-b2ff98e
component=proxy
heritage=Tiller
release=jupyterhub
Annotations:
Selector: component=proxy,release=jupyterhub
Type: LoadBalancer
IP: 10.96.68.255
Port: http 80/TCP
TargetPort: 8000/TCP
NodePort: http 30041/TCP
Endpoints: 10.244.3.3:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: ```

miramar-labs on 9 Apr 2018

kubectl --namespace=jupyterhub get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE glusterfs-dynamic-hub-db-dir ClusterIP 10.98.232.70 <none> 1/TCP 23m hub ClusterIP 10.96.107.41 <none> 8081/TCP 23m proxy-api ClusterIP 10.105.79.175 <none> 8001/TCP 23m proxy-public LoadBalancer 10.96.68.255 <pending> 80:30041/TCP 23m

miramar-labs on 9 Apr 2018

kubectl --namespace=jupyterhub get pods NAME READY STATUS RESTARTS AGE hub-6c47b9ff6c-dlgcj 1/1 Running 0 24m proxy-54fbf87d4d-t9tgx 1/1 Running 0 24m

miramar-labs on 9 Apr 2018

Ah, so from my understanding the problem is the LoadBalancer service
proxy-public never gets an external-ip. Is that correct?

How was your cluster set up? Does it have support for creating LoadBalancer
objects? Your cluster might not support creating LoadBalancer objects, in
which case you might need to set up your own Ingress Controller and use the
ingress support we have documented in
https://zero-to-jupyterhub.readthedocs.io/en/latest/advanced.html#ingress.

On Mon, Apr 9, 2018 at 10:53 AM, Aaron Cody notifications@github.com
wrote:

NAME READY STATUS RESTARTS AGE
hub-6c47b9ff6c-dlgcj 1/1 Running 0 24m
proxy-54fbf87d4d-t9tgx 1/1 Running 0 24m```

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/641#issuecomment-379837879,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23ir7bN3O2uS82plv-2wCnm7xO8_Cks5tm6AMgaJpZM4TM8bK
.

--
Yuvi Panda T
http://yuvi.in/blog

yuvipanda on 9 Apr 2018

yes that's probably it - my cluster is manually configured from scratch on ESXi ... do you have any examples of how to manually configure the ingress-nginx controller for jupyterhub?

miramar-labs on 9 Apr 2018

well I have the NGINX Ingress Controller running:
------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.12.0 Build: git-1df421a Repository: https://github.com/kubernetes/ingress-nginx -------------------------------------------------------------------------------

but still same issue .. must still be missing a bit.... can you provide any clues?
TIA

miramar-labs on 10 Apr 2018

my config:
proxy: secretToken: "{{ _jhub_secret_token }}" ingress: enabled: true hosts: - {{_jhub_ingress}} singleuser: image: name: jupyter/scipy-notebook tag: latest storage: dynamic: storageClass: gluster-sc capacity: 1Gi memory: limit: 1Gi guarantee: 1Gi cpu: limit: .5 guarantee: .5

for __jhub_ingress_ I just picked the public IP of one of my worker nodes ... I'm assuming this is where we specify the 'external IP' right?

miramar-labs on 10 Apr 2018

The {{ }} within a values file makes me suspicious. The value files cant use values like the templates. The template files consumes values provided to them.

consideRatio on 10 Apr 2018

that's just the ansible template I'm using ... the final yaml that gets passed to kubectl looks like this:

proxy:
    secretToken: "984d816b6e209c04bd1fb4a51cfaebcff92ef8f53ffff18a092f2b844ad8aa3b"
ingress:
    enabled: true
    hosts:
     - worker1.lan.aaroncody.com
singleuser:
  image:
    name: jupyter/scipy-notebook
    tag: latest
  storage:
    dynamic:
      storageClass: gluster-sc
    capacity: 1Gi
  memory:
    limit: 1Gi
    guarantee: 1Gi
  cpu:
    limit: .5
    guarantee: .5

miramar-labs on 10 Apr 2018

even with the proxy in pending state:

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
glusterfs-dynamic-hub-db-dir   ClusterIP      10.98.232.70    <none>        1/TCP          23m
hub                            ClusterIP      10.96.107.41    <none>        8081/TCP       23m
proxy-api                      ClusterIP      10.105.79.175   <none>        8001/TCP       23m
proxy-public                   LoadBalancer   10.96.68.255    <pending>     80:30041/TCP

I can still access jupyter via the 30041 port ... and it is working flawlessly ... carving out individual volumes for each user - very cool ... but I would love to know how to get the proxy working ... I'm not using a cloud provider - I built this cluster from scratch on a bunch of ESXi VMs (3 Masters, 8 workers, 1 VIP managed by keepalived and loadbalanced across masters by HAProxy) ... not sure how 'external IP's' are allocated in this scenario ? I guess usually that's done under the covers by Google/AWS cloud?

miramar-labs on 10 Apr 2018

Heya! If you're using an ingress object, then the IP will not be in
EXTERNAL-IP but be provided by other means. However, this is highly
dependent on the way your cluster is set up and how networking is
performed. I'm not sure we could help much :( Perhaps there are online
resources on setting up Kubernetes Ingresses on ESXi VMs?

On Mon, Apr 9, 2018 at 8:13 PM, Aaron Cody notifications@github.com wrote:

even with the proxy in pending state:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs-dynamic-hub-db-dir ClusterIP 10.98.232.70 1/TCP 23m
hub ClusterIP 10.96.107.41 8081/TCP 23m
proxy-api ClusterIP 10.105.79.175 8001/TCP 23m
proxy-public LoadBalancer 10.96.68.255 80:30041/TCP

I can still access jupyter via the 30041 port ... and it is working
flawlessly ... carving out individual volumes for each user - very cool ...
but I would love to know how to get the proxy working ... I'm not using a
cloud provider - I built this cluster from scratch on a bunch of ESXi VMs
(5 Masters, 8 workers, 1 VIP managed by keepalived and loadbalanced across
masters by HAProxy) ... not sure how 'external IP's' are allocated in this
scenario ? I guess usually that's done under the covers by Google/AWS cloud?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/641#issuecomment-379960282,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23jA2cjX7v6auXwPgfY2EbxPAoOecks5tnCNqgaJpZM4TM8bK
.

--
Yuvi Panda T
http://yuvi.in/blog

yuvipanda on 10 Apr 2018

yeah - at this point it's not your problem ... so I'm closing it - thanks for the responses and great work!

miramar-labs on 10 Apr 2018

@miramar-labs Have you tried:

proxy:
  service:
    type: ClusterIP

$ kubectl -n jupyterhub-public get svc
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hub            ClusterIP   10.233.59.158   <none>        8081/TCP   3d
proxy-api      ClusterIP   10.233.18.87    <none>        8001/TCP   3d
proxy-public   ClusterIP   10.233.16.185   <none>        80/TCP     3d

manics on 10 Apr 2018

👍1

Installing on a LAN for now. Anybody has the magic recipe to make the default chart work?

$ cat config.yaml 
proxy:
  secretToken: "4eeb11cfb36..."
  service:
    type: ClusterIP

$ helm upgrade test-jupyterhub jupyterhub/jupyterhub --version=v0.6 -f ./config.yaml
Error: UPGRADE FAILED: Service "proxy-public" is invalid: [spec.ports[0].nodePort: Forbidden: may not be used when type is 'ClusterIP', spec.ports[1].nodePort: Forbidden: may not be used when type is 'ClusterIP']

epoch1970 on 26 Jul 2018

@epoch1970 I also received this error after trying to do an updated from service type LoadBalancer to ClusterIP, I had to do a helm delete --purge then a helm install again and that fixed my issue