Spark-on-k8s-operator: Add support for SPARK-27963 (dynamic resource allocation without an external shuffle service)

[liyinan926]

https://issues.apache.org/jira/browse/SPARK-27963 introduced this enhancement that allows users to enable dynamic resource allocation without needing an external shuffle service. This enhancement was added mainly to address the need of Spark on Kubernetes, which lacks support for dynamic resource allocation due to the lack of an external shuffle service for the Kubernetes deployment mode.

We should add support for SPARK-27963 as an experimental feature. This feature is currently only available in the master branch of Spark.

[qingboc]

@liyinan926 SPARK-27963 is said to be in Spark 3.0.0, do you know when that will be released approximately?

[liyinan926]

The latest info I got points to Q1 2020.

[runzhliu]

Hi @liyinan926 , as we have currently patched this feature Spark-27963 into the internal branch of Spark, we don't see any problem for now, I am wondering what kind of extra support you mentioned above to do in Spark Operator when we trigger on dynamic resource allocation without external shuffle service.

[liyinan926]

I was thinking of adding a simply optional field in spec that when set lets the operator automatically add the necessary config options to enable the feature. It's also useful to add a new field numExecutors in status that tells the number of executors at runtime.

[dineshabbi]

@liyinan926 do we have spark operator image which can support spark 3.0 and the additional config fields you mentioned now ? This is super useful for deployments who want to auto scale up/down on kubernetes.

[liyinan926]

@dineshabbi yes, we now have the following images:

gcr.io/spark-operator/spark:v3.0.0
gcr.io/spark-operator/spark-py:v3.0.0
gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0

[liyinan926]

This limited form ofdynamic allocation will be supported in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/976.