Spark-on-k8s-operator: [proposal] Add volumeClaimTemplates to executor spec.

[ringtail]

We are running spark on ECI (similar with aws fargate). And we want to mount cloud disk as volumes for executors. So volumeClaimTemplates would be helpful for this case. Implement is on the way.

[ringtail]

@liyinan926 @Jeffwan Is this a common case for GKE and AWS?

[Jeffwan]

@ringtail We do have cases using cloud disks for shuffle scratch space, Input & output data.
Webhook allow us specify PV and PVC for driver and executors. Trying to understand anything different in volumeClaimTemplates ? Do I miss something?

[ringtail]

@Jeffwan Do you mean SparkLocalDirVolumePrefix feature?

[Jeffwan]

@ringtail

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go#L422

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml#L31

All volume types are supported.

[ringtail]

What if i want to run each executor with a independent cloud disk.

[themoah]

@ringtail @Jeffwan I have similar workflow, each pod needs its own PVC (stateful streaming). There is also autoscaling and pods are running on spots; thus need some automation to create cloud disks (with defined storageClass).

[ringtail]

@ringtail @Jeffwan I have similar workflow, each pod needs its own PVC (stateful streaming). There is also autoscaling and pods are running on spots; thus need some automation to create cloud disks (with defined storageClass).

I have made it in our release. (https://github.com/AliyunContainerService/spark-on-k8s-operator/blob/alibabacloud-v1beta2-1.1.0-2.4.5/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go#L284). If helpful, I'll open a pr to patch this feature.

[themoah]

@ringtail yes, please!

[itamar-otonomo]

@ringtail That would be great!
I am currently using spot (previously spotinst) for node allocation. One of the shortcomings for that method is that defining host-types (with different RAM size and CPU count) in a "Virtual Node Group" allows to define only a single size for the local disk.
The outcome is, depending on the host type spot chooses for you, you get either too much scratch storage if a small host was chosen (for a single executor) or too little storage if a big host was chosen and it runs several instances of the aforementioned executor.

Furthermore, this would enable us to determine the size of scratch storage per executor, also according to the type of application run and how storage intensive it will get! That would help staying lean and cost effective and also automate the autoscaling and resolution of storage allocation and usage.

Thanks for your work, I say shlap it in there!