Consul-helm: Consul-helm 0.7.0 - failing -sync-catalog CrashLoopBackOff

Created on 8 Apr 2019 · 6Comments · Source: hashicorp/consul-helm

there seems to be a issue with sync catalog in this version. Installed it straight from consul-helm-0.7.0, did a helm install and everything looked good in the beginning. All the ports are open. per policies for this env. Need some help with this. Is it a bug? I noticed that the docker image it is pulling fails within consul-k8s.

# kubectl get pods -n orbit-workload && kubectl get svc -n orbit-workload
NAME                                                   READY   STATUS             RESTARTS   AGE
consulhelm-rd92d                           0/1     Running            0          2d22h
consulhelm-sync-catalog-77665b6d45-kz2md   0/1     CrashLoopBackOff   1130       2d22h
consulhelm-v94wd                           0/1     Running            0          2d22h
NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
consulhelm-dns   ClusterIP   10.233.37.55   <none>        53/TCP,53/UDP   2d22h

# kubectl describe pod/consulhelm-sync-catalog-77665b6d45-kz2md -n orbit-workload
Name:               consulhelm-sync-catalog-77665b6d45-kz2md
Namespace:          workload
Priority:           0
PriorityClassName:  <none>
Node:               cs-k8s-lab-k8s-node-nf-1/x.x.x.x
Start Time:         Fri, 05 Apr 2019 22:56:15 +0000
Labels:             app=consulhelm
                    chart=consulhelm-helm
                    component=sync-catalog
                    pod-template-hash=77665b6d45
                    release=orbit
Annotations:        consul.hashicorp.com/connect-inject: false
                    k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "cni0",
                          "ips": [
                              "x.x.x.x"
                          ],
                          "default": true,
                          "dns": {}
                      }]
Status:             Running
IP:                 x.x.x.x
Controlled By:      ReplicaSet/consulhelm-sync-catalog-77665b6d45
Containers:
  consul-sync-catalog:
    Container ID:  docker://72052fbbac77530f4022847e400cc691c7584b9b4975d97fe1581a1ab14b5772
    Image:         hashicorp/consul-k8s:0.7.0
    Image ID:      docker-pullable://hashicorp/consul-k8s@sha256:24910bfd437a36aa3c1665eddc76cb7aabdc827ab93677bf5d19924850e9a75e
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
      consul-k8s sync-catalog \
        -http-addr=${HOST_IP}:8500 \
        -k8s-default-sync=true \
        -consul-domain=orbk8s \
        -k8s-service-prefix="consul-tag" \
        -k8s-write-namespace=${NAMESPACE} \
        -node-port-sync-type=ExternalFirst \
        -consul-k8s-tag=k8s-tag

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 08 Apr 2019 20:44:25 +0000
      Finished:     Mon, 08 Apr 2019 20:45:35 +0000
    Ready:          False
    Restart Count:  1128
    Liveness:       http-get http://:8080/health/ready delay=30s timeout=5s period=5s #success=1 #failure=3
    Readiness:      http-get http://:8080/health/ready delay=10s timeout=5s period=5s #success=1 #failure=5
    Environment:
      HOST_IP:     (v1:status.hostIP)
      NAMESPACE:  orbit-workload (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from orbit-orbit-consulhelm-sync-catalog-token-5dhdx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  consulhelm-sync-catalog-token-5dhdx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  consulhelm-sync-catalog-token-5dhdx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                        From                               Message
  ----     ------     ----                       ----                               -------
  Warning  Unhealthy  9m32s (x14631 over 2d21h)  kubelet, cs-k8s-lab-k8s-node-nf-1  Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    4m28s (x13828 over 2d21h)  kubelet, cs-k8s-lab-k8s-node-nf-1  Back-off restarting failed container

bug waiting-on-response

Source

csayler

All 6 comments

Can anyone help with this?

csayler on 10 Apr 2019

👀1

Hi @csayler,

We've done some initial checking and it looks like the binary and docker image are okay (something we worry about after a release). This suggests that there might be an issue with a specific option that you're enabling that could be causing this. Would you be willing to share your helm chart values so we can try to repro the issue?

adilyse on 11 Apr 2019

👍1

your repo is cloned and tar copied to the artifactory, then we run below:

helm install --set glocal.enabled=false,server.enabled=true,server.replicas=1,server.bootstrapExpect=1,client.enabled=true,dns.enabled=true,ui.enabled=true,syncCatalog.enabled=true,syncCatalog.nodePortSyncType=ExternalFirst https://ourartifactory/artifactory/orbit-k8s-helm-local/orbit-consulhelm-0.7.0.tgz -n consulhelm070 --namespace workloads

csayler on 11 Apr 2019

@csayler does this still occur with the latest version (v0.9.0)?