Openebs: Pod has unbound immediate PersistentVolumeClaims

Created on 11 Dec 2020  路  4Comments  路  Source: openebs/openebs

Openebs version: 1.12.0

I attached new NVMe drives to servers. I want to use disks as Local PV Device.
Unfortunately, this cannot be done using the official documentation - https://docs.openebs.io/docs/next/uglocalpv-device.html#optional-block-device-tagging
To avoid making a mistake, I just copy-paste from the official documentation and it does not work

  1. Block Device Tagging
kubectl label bd -n openebs blockdevice-2007cc4fef1c97ef1e0307ecbe45b24f openebs.io/block-device-tag=mongo
blockdevice.openebs.io/blockdevice-2007cc4fef1c97ef1e0307ecbe45b24f labeled

kubectl label bd -n openebs blockdevice-9ba544a37c7bb5fbf6961e6ed4ef1c84 openebs.io/block-device-tag=mongo
blockdevice.openebs.io/blockdevice-9ba544a37c7bb5fbf6961e6ed4ef1c84 labeled



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl label bd -n openebs blockdevice-d3ac8b630a191cc692239285f391727b openebs.io/block-device-tag=mongo
blockdevice.openebs.io/blockdevice-d3ac8b630a191cc692239285f391727b labeled



md5-d3bd912b4f2a6049fdda21bdd35f2bce



k get bd --show-labels 
NAME                                           NODENAME   SIZE            CLAIMSTATE   STATUS     AGE    LABELS
blockdevice-05e1e9cacd465ded251cd6cc1a62e5d3   svc3       960197124096    Unclaimed    Inactive   8d     kubernetes.io/hostname=svc3,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-1051ce23b21a5ff64b12eddfeaacf928   svc3       960197124096    Unclaimed    Unknown    8d     kubernetes.io/hostname=svc3,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-11908aa88690ec75e781745767cf0a15   svc1       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc1,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-2007cc4fef1c97ef1e0307ecbe45b24f   svc1       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc1,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true,openebs.io/block-device-tag=mongo
blockdevice-22cb4ea3f466f2bc7c99cbbd93a53a34   svc2       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc2,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-7bbdfddf578ec96c8b70672167eeadc5   svc3       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc3,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-9ba544a37c7bb5fbf6961e6ed4ef1c84   svc2       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc2,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true,openebs.io/block-device-tag=mongo
blockdevice-a5c58eb7c9c2edcfb488ff8e0f4269fe   svc1       3840755982336   Claimed      Inactive   128d   kubernetes.io/hostname=svc1,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-ab16d8bc2a85902c096069163645b5ec   svc3       3840755982336   Claimed      Inactive   128d   kubernetes.io/hostname=svc3,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-d3ac8b630a191cc692239285f391727b   svc3       3840755982336   Unclaimed    Active     8d     kubernetes.io/hostname=svc3,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true,openebs.io/block-device-tag=mongo
blockdevice-d5eb71dd349a894b32248ae12b451223   svc1       960197124096    Unclaimed    Unknown    15d    kubernetes.io/hostname=svc1,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-e8e4977a8a096bc0c6296afe781da94b   svc2       960197124096    Unclaimed    Unknown    14d    kubernetes.io/hostname=svc2,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true
blockdevice-ececb3cfa6e14bc891f99650c44b225c   svc2       3840755982336   Claimed      Inactive   128d   kubernetes.io/hostname=svc2,ndm.io/blockdevice-type=blockdevice,ndm.io/managed=true



md5-0a15ddb85c438fd963d738620e091da2



apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-device
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: device
      - name: FSType
        value: xfs
      - name: BlockDeviceTag
        value: "mongo"
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl apply -f local-device-sc.yaml
storageclass.storage.k8s.io/local-device created



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl get sc local-device -o yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    cas.openebs.io/config: |
      - name: StorageType
        value: device
      - name: FSType
        value: xfs
      - name: BlockDeviceTag
        value: "mongo"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"cas.openebs.io/config":"- name: StorageType\n  value: device\n- name: FSType\n  value: xfs\n- name: BlockDeviceTag\n  value: \"mongo\"\n","openebs.io/cas-type":"local"},"name":"local-device"},"provisioner":"openebs.io/local","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    openebs.io/cas-type: local
  creationTimestamp: "2020-12-11T06:30:28Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cas.openebs.io/config: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:openebs.io/cas-type: {}
      f:provisioner: {}
      f:reclaimPolicy: {}
      f:volumeBindingMode: {}
    manager: kubectl
    operation: Update
    time: "2020-12-11T06:30:28Z"
  name: local-device
  resourceVersion: "178329459"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/local-device
  uid: bcc78dd5-e00d-4430-89b5-37821a1082e1
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer



md5-cdb23bc277db5154883f20c81302c494



kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-device-pvc
spec:
  storageClassName: openebs-device
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl apply -f local-device-pvc.yaml
persistentvolumeclaim/local-device-pvc created



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl get pvc local-device-pvc
NAME               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS     AGE
local-device-pvc   Pending                                      openebs-device   39s



md5-6ee7d90af06bb08b190f8338f6a01381



apiVersion: v1
kind: Pod
metadata:
  name: hello-local-device-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-device-pvc
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl apply -f local-device-pod.yaml
pod/hello-local-device-pod created



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl get pod hello-local-device-pod
NAME                     READY   STATUS    RESTARTS   AGE
hello-local-device-pod   0/1     Pending   0          90s



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl describe pod hello-local-device-pod

Name:         hello-local-device-pod
Namespace:    openebs
Priority:     0
Node:         <none>
Labels:       <none>
Annotations:  Status:  Pending
IP:           
IPs:          <none>
Containers:
  hello-container:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done
    Environment:  <none>
    Mounts:
      /mnt/store from local-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6g9gt (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  local-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  local-device-pvc
    ReadOnly:   false
  default-token-6g9gt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6g9gt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "hello-local-device-pod": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "hello-local-device-pod": pod has unbound immediate PersistentVolumeClaims



md5-d3bd912b4f2a6049fdda21bdd35f2bce



kubectl get pvc local-device-pvc
NAME               STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS     AGE
local-device-pvc   Pending                                      openebs-device   4m5s



md5-d3bd912b4f2a6049fdda21bdd35f2bce



k get bdc
NAME                                           BLOCKDEVICENAME                                PHASE     AGE
bdc-02279260-b355-4b9c-a62b-c65fe4c7f1a7       blockdevice-ab16d8bc2a85902c096069163645b5ec   Bound     128d
bdc-4778f7b5-6d73-4b94-bfec-5ba5e7261e89       blockdevice-a5c58eb7c9c2edcfb488ff8e0f4269fe   Bound     128d
bdc-7106842d-1793-4dfe-9fcb-6496cbe7d26c       blockdevice-ececb3cfa6e14bc891f99650c44b225c   Bound     128d
bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb                                                  Pending   15h



md5-d3bd912b4f2a6049fdda21bdd35f2bce



k describe bdc bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb
Name:         bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb
Namespace:    openebs          
Labels:       <none>
Annotations:  <none>   
API Version:  openebs.io/v1alpha1
Kind:         BlockDeviceClaim
Metadata:   
  Creation Timestamp:  2020-12-10T14:49:40Z
  Finalizers:     
    local.openebs.io/finalizer
  Generation:  2 
  Managed Fields:                             
    API Version:  openebs.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:                                                                                      
        f:phase:                                                                                     
    Manager:      ndo                                                                                
    Operation:    Update                                                                             
    Time:         2020-12-10T14:49:40Z
    API Version:  openebs.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:spec:
        .:
        f:blockDeviceNodeAttributes:
          .:
          f:hostName:
        f:deviceClaimDetails:
        f:deviceType:
        f:hostName:
        f:resources:
          .:
          f:requests:
            .:
            f:storage:
        f:selector:
          .:
          f:matchLabels:
            .:
            f:openebs.io/block-device-tag:
      f:status:
    Manager:         provisioner-localpv
    Operation:       Update
    Time:            2020-12-10T14:49:40Z
  Resource Version:  177333466
  Self Link:         /apis/openebs.io/v1alpha1/namespaces/openebs/blockdeviceclaims/bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb
  UID:               7ad7dfca-b23a-4899-9765-e8e8137ff46d
Spec:
  Block Device Node Attributes:
    Host Name:  ch1
  Device Claim Details:
  Device Type:  
  Host Name:    
  Resources:
    Requests:
      Storage:  5G
  Selector:
    Match Labels:
      openebs.io/block-device-tag:  nvme-local
Status:
  Phase:  Pending
Events:
  Type     Reason           Age                    From                       Message
  ----     ------           ----                   ----                       -------
  Warning  SelectionFailed  89s (x11553 over 15h)  blockdeviceclaim-operator  no blockdevices found
picture svua

Most helpful comment

I have solved for this issue.

Firstly, There is a mistake in the documentation
https://docs.openebs.io/docs/next/uglocalpv-device.html#create-a-persistentvolumeclaim

In local-device-sc.yaml
name: local-device

In local-device-pvc.yaml
storageClassName: openebs-device

Therefore the copy-paste method does not work.

Secondly, We need use nodeSelector for pods if we have disks only on the different nodes. For example:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: pvc1
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage
  nodeSelector:
    kubernetes.io/hostname: svc1

Thanks for the tip with finalizers!

picture svua on 15 Dec 2020
❤3

All 4 comments

@svua - can you share the following outputs as well: 

kubectl get pv

The bdc that is in pending state actually has the match label configured as openebs.io/block-device-tag: nvme-local, even though you have configured the tag as mongo. Probably a left out from the previous run?

kmova picture kmova on 11 Dec 2020

Yes, I ran it many times and nothing worked. I decided to do it through copy-paste.

There is nothing interesting there (I removed volumes from the output for applications that run in the cluster)

kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                                                                         STORAGECLASS       REASON   AGE
local-pv-25b15fa                           58Gi       RWO            Retain           Bound       core/prometheus-metrics-prometheus-operato-prometheus-db-prometheus-metrics-prometheus-operato-prometheus-0   local-disks                 148d
local-pv-29fbea2d                          20Gi       RWO            Retain           Available                                                                                                                 local-disks                 99d
local-pv-34116f6                           3967Mi     RWO            Retain           Bound       core/metrics-grafana                                                                                          local-disks                 148d
pvc-15e409d5-253e-4ffe-a8a2-3b4335bdde17   40Gi       RWO            Retain           Bound       logging/storage-loki-0

How can I delete a claim?

kubectl delete pod hello-local-device-pod
Error from server (NotFound): pods "hello-local-device-pod" not found

kubectl delete pvc local-device-pvc
Error from server (NotFound): persistentvolumeclaims "local-device-pvc" not found

kubectl delete sc local-device
Error from server (NotFound): storageclasses.storage.k8s.io "local-device" not found

k delete bdc bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb
blockdeviceclaim.openebs.io "bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb" deleted

Delete is stuck and nothing happens

k get bdc
NAME                                           BLOCKDEVICENAME                                PHASE     AGE
bdc-02279260-b355-4b9c-a62b-c65fe4c7f1a7       blockdevice-ab16d8bc2a85902c096069163645b5ec   Bound     128d
bdc-4778f7b5-6d73-4b94-bfec-5ba5e7261e89       blockdevice-a5c58eb7c9c2edcfb488ff8e0f4269fe   Bound     128d
bdc-7106842d-1793-4dfe-9fcb-6496cbe7d26c       blockdevice-ececb3cfa6e14bc891f99650c44b225c   Bound     128d
bdc-pvc-fd0cb523-11ff-42c7-9997-fb4530257ceb                                                  Pending   17h
svua picture svua on 11 Dec 2020

@svua - are you using the kubernetes static provisioner as well as openebs dynamic provisioner here. Would you like to clean up both types of volumes and start fresh or do you want to only clean up the openebs objects and start configuring them?

Also, would you prefer to continue here or via slack for clearing the setup? I can help you with cleaning up and getting it back to working.

To remove the BDCs - they depend on the finalizers. So we need to start with clearing them up.

kmova picture kmova on 11 Dec 2020

I have solved for this issue.

Firstly, There is a mistake in the documentation
https://docs.openebs.io/docs/next/uglocalpv-device.html#create-a-persistentvolumeclaim

In local-device-sc.yaml
name: local-device

In local-device-pvc.yaml
storageClassName: openebs-device

Therefore the copy-paste method does not work.

Secondly, We need use nodeSelector for pods if we have disks only on the different nodes. For example:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: pvc1
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage
  nodeSelector:
    kubernetes.io/hostname: svc1

Thanks for the tip with finalizers!

svua picture svua on 15 Dec 2020
❤3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ajeshbaby picture ajeshbaby  路  4Comments
kmova picture kmova  路  3Comments
prateekpandey14 picture prateekpandey14  路  6Comments
tboerger picture tboerger  路  5Comments
mhemrg picture mhemrg  路  4Comments