Katib: ReadWriteMany unsupported for PVCs on GCP

[texasmichelle]

/kind bug

What steps did you take and what happened:
When creating tfevent-pvc, it fails on GCP with this error:
Failed to provision volume with StorageClass "standard": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce ReadOnlyMany] are supported

What did you expect to happen:
The PVC to be created without errors.

Anything else you would like to add:
Since C2D doesn't support v0.7.0 yet, this version (v1alpha2) should work on GCP without failures.

Environment:

• Kubeflow version: v0.6.2 (latest in C2D)
• Minikube version: n/a
• Kubernetes version: (use kubectl version): client: v1.13.11, server: v1.12.10-gke.17
• OS (e.g. from /etc/os-release): MacOS v10.14.6

[gaocegege]

Thanks for the issue.

/cc @hougangliu

[texasmichelle]

Fix incoming 🚀

[hougangliu]

this example is for minikube assuming that the PVC with ReadWriteMany can be created
Since multiple trials/jobs can run in parallel to write TensorFlow event to the PV, and multiple tf-metricscollector Pod will read the data, so the PVC must be ReadWriteMany.

default GCP storage class is gcePersistentDisk driver, which can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed.

@texasmichelle So I think you have to make your cluster support ReadWriteMany PVC somehow, maybe https://github.com/mappedinn/kubernetes-nfs-volume-on-gke helps.

BTW, just curious about what is C2D?

[texasmichelle]

Oh - we only have examples for minikube? Because HP tuning is so resource-intensive and I've had trouble just getting barebones KF to run on its own locally, it didn't even occur to me to try minikube. I think it would be worthwhile to have a cloud version of the examples.

@richardsliu what do you suggest for pvc config on gke?

C2D = click-to-deploy

[hougangliu]

sorry to mislead you, the example can work in most platforms including minikube, and in most platform, ReadWriteMany PVC can be created.
I think GCP also should have solution to support ReadWriteMany PVC (for now I guess default storageclass is gcePersistentDisk not supportingReadWriteMany, can you try if there is other storageclass in GCP to support ReadWriteMany, if there is, just make the example pvc storageclass as the one to run this example).

[gaocegege]

ReadWriteMany is common, I think GCP should support it. Ref https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes

[richardsliu]

Katib-mysql is ReadWriteOnce, how does it work?
https://github.com/kubeflow/katib/blob/master/manifests/v1alpha3/pv/pv.yaml

[richardsliu]

Standard storage class provisoner is GKE PD which doesn't support ReadWriteMany:

NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   6d2h

https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes

So another solution is to try a different storage class. For example: https://developer.ibm.com/tutorials/add-nfs-provisioner-to-icp/

[hougangliu]

Katib-mysql is ReadWriteOnce, how does it work?
https://github.com/kubeflow/katib/blob/master/manifests/v1alpha3/pv/pv.yaml

Only one Pod is running for katib-mysql.

[vpavlin]

It is a common problem among cloud provider and storage backends in general. There are some heavily used (GCP PD, AWS EBS, Ceph) which do not support ReadWriteMany specifically because they won't allow you to attach one block device to multiple nodes, thus if you scale up and scheduler distributes the load, containers won't start because the volume will not be able to attach to multiple nodes.

Maybe it would make sense to note that in the example or Katib documentation that ReadWriteMany for storage backend is required (e.g. NFS), or think about a different option like using object storage somehow.

[gaocegege]

@vpavlin 's advice SGTM.

For katib-mysql, I think the users should have other choices to deploy a SQL DB.

For trial-used volumes, if we do not have a ReadWriteMany volume, the parallelism will be limited.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[edikmkoyan]

/reopen the same is true for azure k8s service.

[andreyvelich]

/lifecycle frozen

[andreyvelich]

We don't have Trial volumes currently.
We can close it.
Feel free to re-open if you see this problem again.