Consul-helm: Consul Servers are not coming up with error "pod has unbound immediate PersistentVolumeClaims"

[subashd]

I have followed https://www.consul.io/docs/platform/k8s/run.html#installing-consul to install consul in my kubernetes cluster ( I have synced to master rather than to v0.1.0, but results are same with v0.1.0 as well). I am facing issues where consul servers are not coming up with error “Warning FailedScheduling default-scheduler pod has unbound immediate PersistentVolumeClaims ” and when I checked the pvc getting “persistentvolume-controller no persistent volumes available for this claim and no storage class is set”

I am running Kubernetes version 1.16.

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}   


helm list
NAME    REVISION        UPDATED                         STATUS          CHART           APP VERSION     NAMESPACE
consul  1               Wed Nov  6 11:16:59 2019        DEPLOYED        consul-0.12.0                   default



kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP                NODE            NOMINATED NODE   READINESS GATES
consul-consul-server-0   0/1     Pending   0          45s   <none>            <none>          <none>           <none>
consul-consul-server-1   0/1     Pending   0          45s   <none>            <none>          <none>           <none>
consul-consul-server-2   0/1     Pending   0          45s   <none>            <none>          <none>           <none>
consul-consul-sp895      0/1     Running   0          45s   192.168.216.157   consul-node-2   <none>           <none>   



kubectl describe pods consul-consul-server-0
Name:           consul-consul-server-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=consul
                chart=consul-helm
                component=server
                controller-revision-hash=consul-consul-server-7c8dfd5c4c
                hasDNS=true
                release=consul
                statefulset.kubernetes.io/pod-name=consul-consul-server-0
Annotations:    consul.hashicorp.com/connect-inject: false
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/consul-consul-server
Containers:
  consul:
    Image:       consul:1.6.1
    Ports:       8500/TCP, 8301/TCP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
    Command:
      /bin/sh
      -ec
      CONSUL_FULLNAME="consul-consul"

      exec /bin/consul agent \
        -advertise="${POD_IP}" \
        -bind=0.0.0.0 \
        -bootstrap-expect=3 \
        -client=0.0.0.0 \
        -config-dir=/consul/config \
        -datacenter=dc1 \
        -data-dir=/consul/data \
        -domain=consul \
        -hcl="connect { enabled = true }" \
        -ui \
        -retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
        -retry-join=${CONSUL_FULLNAME}-server-1.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
        -retry-join=${CONSUL_FULLNAME}-server-2.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
        -server

    Readiness:  exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
    Environment:
      POD_IP:      (v1:status.podIP)
      NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /consul/config from config (rw)
      /consul/data from data-default (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from consul-consul-server-token-jzwww (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  data-default:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-default-consul-consul-server-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      consul-consul-server-config
    Optional:  false
  consul-consul-server-token-jzwww:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  consul-consul-server-token-jzwww
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  pod has unbound immediate PersistentVolumeClaims


kubectl get pv
No resources found in default namespace.     


kubectl get pvc
NAME                                  STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-default-consul-consul-server-0   Pending                                                     6m22s
data-default-consul-consul-server-1   Pending                                                     6m22s
data-default-consul-consul-server-2   Pending                                                     6m22s


kubectl describe pvc data-default-consul-consul-server-0
Name:          data-default-consul-consul-server-0
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        app=consul
               chart=consul-helm
               component=server
               hasDNS=true
               release=consul
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    consul-consul-server-0
Events:
  Type    Reason         Age                  From                         Message
  ----    ------         ----                 ----                         -------
  Normal  FailedBinding  8s (x26 over 6m14s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

[ishustava]

Hey @subashd, thanks for submitting this issue.

It looks like you don't have a default storage class defined in your k8s cluster. For example, here is what I have in my minikube:

$ kubectl get storageclass
NAME                 PROVISIONER                AGE
standard (default)   k8s.io/minikube-hostpath   52d

To give a bit more details, the Helm chart sets storageClass to null by default which means that it will use cluster's default storage class (docs). If you don't have one defined in your cluster, it will fail because Kubernetes doesn't know which storage provisioner it should use to provision the volumes.

If you'd like to use a different storage class you've defined in your k8s cluster, you could set the storageClass value in the Helm chart to the name of the class.

If you're still running into problems, let us know where your Kubernetes cluster is running so we can better understand the storage provider you may be using.

Hope this helps!

[subashd]

@ishustava I am running on Kubernetes on Prem setup. I have created storage class and mentioned the name in the helm chart... still I am getting error pod has unbound immediate PersistentVolumeClaims

kubectl get storageclass
NAME                 PROVISIONER               AGE
standard (default)   kubernetes.io/host-path   27m  

[ishustava]

You might want to kubectl describe pvc data-default-consul-consul-server-0 to see what the error was this time. Note that if the error was Failed to create provisioner: Provisioning in volume plugin "kubernetes.io/host-path" is disabled, then you'd want to enable host-path provisioner for the kube-contoller-manager.

Also, I'm not sure if you saw it already, but there are a few past similar issues (#237 and #229) that may be useful in debugging and resolving this further.

[ishustava]

Hey @subashd, we haven't heard from you in a while, but I hope you were able to resolve this issue. I'm going to close this, but please feel free to comment here to re-open if you're still having problems.

Thanks!

[abhinav-nath]

Hey @ishustava, I have landed into this same issue. I have a bare metal Kubernetes machine where I don't have any default storageclass defined. How do I create it ? Or more exactly which type of storageclass should I create ?

I tried creating a simple local-storage but it turned out that StatefulSets need dynamic storageclasses and local-storage doesn't support dynamic.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Please let me know what can I do to make consul work ?

[lkysow]

Hey Abhinav,
I've tested this out myself, let me know if it works.

Create a storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-ssd
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

For each consul server, create a persistent volume. Set the nodeAffinity for each node you'll run a Consul server on:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: consul-server-0
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-ssd # <<<<<<<<< This must match the storage class.
  local:
    path: /mnt/data # <<<<<<<<<<<<<< wherever you want the data
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - ip-192-168-23-98.us-west-2.compute.internal  # <<<<< Your hostnames

Repeat ☝️ for each server, i.e. consul-server-1, consul-server-2.

ssh into each node and create the directory, e.g. /mnt/data.

Create a persistent volume claim following the naming convention defined here: https://www.consul.io/docs/platform/k8s/predefined-pvcs.html data-<kubernetes namespace>-<helm release name>-consul-server-<ordinal>.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-default-luke-consul-server-0
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-ssd

Repeat ☝️ for each PV.

Run the helm install, e.g. helm install luke ./consul-helm -n default.

Please let me know if that works for you and I'll update the docs.

[abhinav-nath]

Hey Luke,

Sorry for a late reply and thanks for responding back.

I actually resolved it by going through other similar issues and comments by your team.

I did exactly what you are suggesting. Here are my configs :

I created a local-storage

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Then created a persistent volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-consul-pv0
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  claimRef:
    namespace: kube-system
    name: data-kube-system-hashicorp-consul-server-0
  hostPath:
    path: "/mnt/data"

Changed below :

values.yaml
server.storageclass: local-storage

server-statefulset.yaml
readinessProbe.initialDelaySeconds: 60