Spark-on-k8s-operator: Spark 3.0.0 operator

[vishnugs]

Would like to know when the helm operator for Spark 3.0.0 will be released ?

[liyinan926]

You meant the Helm chart? We already have a new operator image based on Spark 3.0.0 at gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0. You can use this image by adding the following to your helm install command:

--set operatorVersion=v1beta2-1.1.2-3.0.0

[polyzos]

@liyinan926 does the image support Hadoop 3 binaries?

[liyinan926]

No, the base Spark 3.0.0 image used is pre-built with Hadoop 2.7.

[polyzos]

is it possible to have one with hadoop 3 as well?

[WaterKnight1998]

You meant the Helm chart? We already have a new operator image based on Spark 3.0.0 at gcr.io/spark-operator/spark-operator:v1beta2-1.1.2-3.0.0. You can use this image by adding the following to your helm install command:

--set operatorVersion=v1beta2-1.1.2-3.0.0

I wonder how can I combine this chart with zeppelin? I would like zeppelin notebooks pyspark paragraphs to use spark operator.

Do you know if it is possible?

Does this image comes with google storage connector for pyspark?

[scottydelta]

is private registry supposed to be working with --set operatorVersion=v1beta2-1.1.2-3.0.0?

[AliGouta]

@polyzos Another option would be to build a spark image by yourself from source, then push it to your registry:
http://spark.apache.org/downloads.html. Then call it in your sparkApplication Deployment (you won't have gcs/prometheus connectors but you can add them and build your own image). I am interested to your feedback to know if everything runs fine.

[liyinan926]

@scottydelta you can use a private registry by setting --set operatorImageName=<private registry URL of the image excluding the image tag>. Check out the options available in https://github.com/helm/charts/blob/master/incubator/sparkoperator/values.yaml.

As @AliGouta said, you can always build your own custom Spark image using a pre-built package you can download from the Spark site. Once you have a custom Spark image, you can simply build a custom operator image based on the custom Spark image.

@polyzos I have built and pushed the following Spark images based on Hadoop 3:

gcr.io/spark-operator/spark:v3.0.0-hadoop3
gcr.io/spark-operator/spark-py:v3.0.0-hadoop3

[polyzos]

@AliGouta watching the spark summit i came across this
https://github.com/TomLous/medium-spark-k8s/blob/master/docker/spark-runner/Dockerfile
so alterativaly i was thinking of trying to use this with spark3 image @liyinan926 created, but i saw he build it with hadoop3 as well.
Do you see any potential bottlenecks with the above approach? my goal is to use spark3 in order to connect to gen2 azure datalake

[vishnugs]

currently i see

operatorImageName: gcr.io/spark-operator/spark-operator
operatorVersion: v1beta2-1.1.2-2.4.5
imagePullPolicy: IfNotPresent

[vishnugs]

Is there any config related to feature included in this spark version? Or everything is in spark image.?

[fcenedes]

@AliGouta watching the spark summit i came across this
https://github.com/TomLous/medium-spark-k8s/blob/master/docker/spark-runner/Dockerfile
so alterativaly i was thinking of trying to use this with spark3 image @liyinan926 created, but i saw he build it with hadoop3 as well.
Do you see any potential bottlenecks with the above approach? my goal is to use spark3 in order to connect to gen2 azure datalake

That's what we use, as an approach as well, it works just fine.
We derived from gcr.io/spark-operator/spark, but you could just look at : https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile

[liyinan926]

Closing this, and will continue to use #702 to track any remaining work to support Spark 3.0.0.

[axk7812]

@WaterKnight1998 able to get this to work with Zepplin notebooks?

[ordukhanian]

@scottydelta you can use a private registry by setting --set operatorImageName=<private registry URL of the image excluding the image tag>. Check out the options available in https://github.com/helm/charts/blob/master/incubator/sparkoperator/values.yaml.

As @AliGouta said, you can always build your own custom Spark image using a pre-built package you can download from the Spark site. Once you have a custom Spark image, you can simply build a custom operator image based on the custom Spark image.

@polyzos I have built and pushed the following Spark images based on Hadoop 3:

gcr.io/spark-operator/spark:v3.0.0-hadoop3
gcr.io/spark-operator/spark-py:v3.0.0-hadoop3

@liyinan926 , could you please share the docker file for gcr.io/spark-operator/spark:v3.0.0-hadoop3?