Pipelines: Pending with "Unschedulable: pod has unbound immediate PersistentVolumeClaims"

[kim-sardine]

Hi,

What steps did you take:

I ran sample pipleline volumeop_sequential.py on totally-fresh-new kubeflow on GKE that I'v setup about half an hour ago.

but pvc with VOLUME_MODE_RWO works just normally (this example)

What happened:

mypvc operation looks good

time="2020-03-10T11:46:50Z" level=info msg="Creating a docker executor"
time="2020-03-10T11:46:50Z" level=info msg="Executor (version: v2.3.0, build_date: 2019-05-20T22:10:54Z) initialized (pod: kubeflow/volumeop-sequential-w8gjk-3057470781) with template:\n{\"name\":\"mypvc\",\"inputs\":{},\"outputs\":{\"parameters\":[{\"name\":\"mypvc-manifest\",\"valueFrom\":{\"jsonPath\":\"{}\"}},{\"name\":\"mypvc-name\",\"valueFrom\":{\"jsonPath\":\"{.metadata.name}\"}},{\"name\":\"mypvc-size\",\"valueFrom\":{\"jsonPath\":\"{.status.capacity.storage}\"}}]},\"metadata\":{},\"resource\":{\"action\":\"create\",\"manifest\":\"apiVersion: v1\\nkind: PersistentVolumeClaim\\nmetadata:\\n  name: 'volumeop-sequential-w8gjk-newpvc'\\nspec:\\n  accessModes:\\n  - ReadWriteMany\\n  resources:\\n    requests:\\n      storage: 10Gi\\n\"}}"
time="2020-03-10T11:46:50Z" level=info msg="Loading manifest to /tmp/manifest.yaml"
time="2020-03-10T11:46:50Z" level=info msg="kubectl create -f /tmp/manifest.yaml -o json"
time="2020-03-10T11:46:52Z" level=info msg=kubeflow/PersistentVolumeClaim./volumeop-sequential-w8gjk-newpvc
time="2020-03-10T11:46:52Z" level=info msg="Saving resource output parameters"
time="2020-03-10T11:46:52Z" level=info msg="[kubectl get PersistentVolumeClaim./volumeop-sequential-w8gjk-newpvc -o jsonpath={} -n kubeflow]"
time="2020-03-10T11:46:52Z" level=info msg="Saved output parameter: mypvc-manifest, value: map[kind:PersistentVolumeClaim apiVersion:v1 metadata:map[creationTimestamp:2020-03-10T11:46:52Z annotations:map[volume.beta.kubernetes.io/storage-provisioner:kubernetes.io/gce-pd] finalizers:[kubernetes.io/pvc-protection] name:volumeop-sequential-w8gjk-newpvc namespace:kubeflow selfLink:/api/v1/namespaces/kubeflow/persistentvolumeclaims/volumeop-sequential-w8gjk-newpvc uid:d587243a-62c4-11ea-bcc5-42010aa40053 resourceVersion:31502] spec:map[resources:map[requests:map[storage:10Gi]] storageClassName:standard volumeMode:Filesystem accessModes:[ReadWriteMany]] status:map[phase:Pending]]"
time="2020-03-10T11:46:52Z" level=info msg="[kubectl get PersistentVolumeClaim./volumeop-sequential-w8gjk-newpvc -o jsonpath={.metadata.name} -n kubeflow]"
time="2020-03-10T11:46:52Z" level=info msg="Saved output parameter: mypvc-name, value: volumeop-sequential-w8gjk-newpvc"
time="2020-03-10T11:46:52Z" level=info msg="[kubectl get PersistentVolumeClaim./volumeop-sequential-w8gjk-newpvc -o jsonpath={.status.capacity.storage} -n kubeflow]"
time="2020-03-10T11:46:52Z" level=info msg="Saved output parameter: mypvc-size, value: "
time="2020-03-10T11:46:52Z" level=info msg="Annotating pod with output"

but step1 didn't start and showed below message
This step is in Pending state with this message: Unschedulable: pod has unbound immediate PersistentVolumeClaims (repeated 2 times)

What did you expect to happen:

pipeline works normally

Environment:

I deployed Kubeflow with official document

KFP version: Build commit ca58b22

KFP SDK version: kfp 0.2.5

Anything else you would like to add:

I found similar questions on StackOverflow, but there was no clear solutions.

/kind bug

[elikatsis]

As you've already mentioned, since it works for ReadWriteOnce PVCs, it looks like your default storage class doesn't support ReadWriteMany PVCs.

You could try one of the following:

1. Modify the DSL code to use RWO mode
2. Set up a storage class that supports RWM and
   
   
   
   • either set it to be your default storage class,
   
   
   • or modify the DSL code to use that storage class
   
   

[kim-sardine]

1. Modify the DSL code to use RWO mode

worked for me. (then volumeop_sequential.py seems to need an update.)

1. Set up a storage class that supports RWM

How can I do this?

I found some documents to set PersistentVolume and PersistentVolumeClaim's access mode to RWM but no StorageClass

my default storage that kfctl made for me is

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2020-03-10T10:58:48Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: standard
  resourceVersion: "311"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
  uid: <UID>
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate

[elikatsis]

worked for me. (then volumeop_sequential.py seems to need an update.)

It doesn't need an update. It's just a different access mode not all clusters support from scratch.

How can I do this?

I guess there are various ways. In the past, I've used these instructions to deploy an NFS provisioner, but I'm not sure whether they are up to date.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.