Spark-on-k8s-operator: Please support release Spark 3.0

[AceHack]

It was released yesterday, can you please update
https://spark.apache.org/docs/latest/

I'm happy to create my own base image if this can already be supported

[liyinan926]

Will create a new release based on Spark 3.0 soon. Stay tuned. Thanks.

[liyinan926]

I have just built and pushed the following images:

gcr.io/spark-operator/spark:v3.0.0
gcr.io/spark-operator/spark-py:v3.0.0
gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0

[itamar-otonomo]

Thanks @liyinan926, will there be any changes to the API?

[liyinan926]

@itamar-otonomo There won't be major or backward incompatible changes to the API. There's a couple of new options in Spark 3.0.0 missing in the API. I created https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/954 to add them.

[liyinan926]

We have pretty much added support for most new options in Spark 3.0.0 that are relevant in the context of the operator. We just added support for a limited form of dynamic allocation without an external shuffle service in #976. Closing this and will prepare a new release for 3.0 soon.

[AceHack]

Can you create or point to some docs on the dynamic allocation?