Spark-on-k8s-operator: Python support

[akhurana001]

Hi,

Thanks for the operator.

I was trying to play around and looking at the documentation (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md) I see Python as a supported type in the SparkApplication Spec but i am running into _"Python applications are currently not supported for Kubernetes." ._

I am assuming its stemming from the base spark image: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L332 . Thoughts ?

I can share more details if you believe this can be an issue on my end.

[liyinan926]

@akhurana001 Thanks for the feedback! Yes, the SparkApplication API supports specifying Python dependencies. However, Spark 2.3 does not support PySpark in Kubernetes mode. We plan to add PySpark support into Spark 2.4. The operator is based off the official Spark 2.3 release so that's why you got the error message.

[kumare3]

@liyinan926 thank you for the quick reply. We are planning to support spark on K8s, but majority of our usecases are in python. Is it possible that we can collaborate somehow. I want to understand what changed between 2.2.0 and 2.3.0 that we cannot support python. I want to see if we can do something in the interim, so that we can support python and eventually converge to 2.4.

Also, I would like to see if our team (@Lyft) and your team can collaborate on some of these projects.

[liyinan926]

@kumare3 Actually the operator used to support Spark 2.2 and work with https://github.com/apache-spark-on-k8s/spark. Later it was migrated to the official Spark 2.3 release. It should be fairly easy to create a branch for 2.2 and make it work with that fork. AFAIK, the main changes are names of some Spark configuration properties as https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/commit/bd24ec6b481a73f2646a3d6725d9c24a93cbe7b9#diff-139de0e12f891005bb3eabeba15e0c5a shows. Other notable changes include the removal of the flag --kubernetes-namespace in Spark 2.3. Let me know if you are interested in collaborating on creating a fork for 2.2.

[kumare3]

@liyinan926, that makes sense. And you relieved us. Yes @akhurana001 and I are willing to collaborate and not only that, we are willing to put it through some rigorous testing on our side. Do you want to setup an introductory call. We are already exploring working with couple other GCP projects.

[liyinan926]

@kumare3 @akhurana001 that's awesome! Thanks for contributing! A introductory call sounds good. What about some time next week?

[kumare3]

@liyinan926 how about Tuesday 04/17/2018 afternoon, 2:00 pm? Send me your preferred email address or google address and I can send an invite

[liyinan926]

@kumare3 the time sounds good. You can find my email on my github profile. Talk soon!

[dharmeshkakadia]

+1 We too are really interested in get python for spark-on-k8s going and would like to understand whats needed.

[liyinan926]

@dharmeshkakadia given that the PR https://github.com/apache/spark/pull/21092 for adding PySpark support is out and should be ready for review soon, I would expect it to be merged by the end of the month or early next month.

[ifilonenko]

@dharmeshkakadia I believe it will be merged in a lot sooner :)

[dharmeshkakadia]

Thats definitely encouraging. I was a little scared seeing months :) Do you think next week is good?

[ifilonenko]

We will have a big review tomorrow and I will be trying to resolve issues next week.

[liyinan926]

Python support is now available in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/tree/spark-2.2-support.

[akhurana001]

@ypzhang

I am able to run the example. Can you make sure you are using
*"gcr.io/spark-operator/spark-operator:v2.2.0
http://gcr.io/spark-operator/spark-operator:v2.2.0" *as the operator
image and if possible share the operator logs for this execution (operator
logs spark-submit arguments).

Also are you able to run the non-python examples ?

On Sun, Jun 3, 2018 at 12:18 AM, ypzhang [e-mail] wrote:

Has anybody successfully run the spark-pi-py.yaml example on the 2.2
branch?

I installed spark operator under the manifest directory and:

$ kubectl create -f spark-pi-py.yaml

$ kubectl logs spark-pi-py-1528009901593-driver
++ id -u

• myuid=0
  ++ id -g
• mygid=0
  ++ getent passwd 0
• uidentry=root:x:0:0:root:/root:/bin/ash
• '[' -z root:x:0:0:root:/root:/bin/ash ']'
• /sbin/tini -s -- /bin/sh -c 'SPARK_CLASSPATH="${SPARK_HOME}/jars/" && env | grep SPARK_JAVA_OPT_ | sed '\''s/[^=]=(.*)/1/g'\'' > /tmp/java_opts.txt && readarray -t SPARK_DRIVER_JAVA_OPTS < /tmp/java_opts.txt && if ! [ -z ${SPARK_MOUNTED_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_MOUNTED_CLASSPATH:$SPARK_CLASSPATH"; fi && if ! [ -z ${SPARK_SUBMIT_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_SUBMIT_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi && if ! [ -z ${SPARK_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi && if ! [ -z ${SPARK_MOUNTED_FILES_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_DIR/." .; fi && if ! [ -z ${SPARK_MOUNTED_FILES_FROM_SECRET_DIR} ]; then cp -R "$SPARK_MOUNTED_FILES_FROM_SECRET_DIR/." .; fi && ${JAVA_HOME}/bin/java "${SPARK_DRIVER_JAVA_OPTS[@]}" -cp $SPARK_CLASSPATH -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS'
  Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
  at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:39)
  at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)

Thanks in advance for any input.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/124#issuecomment-394142593,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ag_ElnL3aYSaATFeTg4b-CW-cd6wzVsxks5t4429gaJpZM4TNRGI
.

[ypzhang]

@akhurana001

Thanks for the quick reply. I deleted the original post because I used the wrong image... I didn't realize I had to use different images for driver and executor, and for python too...

And yes, I can run the example-pi.yaml (scala) example.

[ypzhang]

@akhurana001

While we are at this branch, I was just wondering how to proceed to run my own python script. I tried to add my python script in the image and point "mainApplicationFile" to that file but I got this in the driver pod's log:

...
2018-06-04 03:44:05 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 10.244.3.67:38409 with 114.6 MB RAM, BlockManagerId(1, 10.244.3.67, 38409, None)
Exception in thread "main" org.apache.spark.SparkUserAppException: User application exited with 137
        at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:97)
        at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
...

(I can provide more info if necessary)

I noticed there was a dep jar file in the example-pi-py. I wonder if I should generate one too.

So the question is can you share your procedure to run a new python script here?

Btw: the imagePullPolicy in the yaml file does not seem to work. It is always the default behavior even if I set to "Always". Have you tried this option?

[liyinan926]

@ypzhang Regarding image pull policy, it's because the 2.2 fork uses a different Spark config property (spark.kubernetes.docker.image.pullPolicy, see here). This needs to be fixed by changing this line to spark.kubernetes.docker.image.pullPolicy.

[dharmeshkakadia]

I have been running the fork with Python for sometime now. Does executor logs have anything interesting? I have seen a problem where if you remove the example jar from the deps, is does not work. Don't remember exact error message right now. I think it would be worth a try to just add the default one, no need to generate your own example jar and give it shot. I have defaulted to it being always there.

This is an example YAML file for one of the scheduled job:

apiVersion: "sparkoperator.k8s.io/v1alpha1"
kind: ScheduledSparkApplication
metadata:
  name: pyspark-example  # name of the pipeline
  namespace: default
spec:
  schedule: "*/10 * * * *"  # change to your desired schedule
  concurrencyPolicy: Replace
  runHistoryLimit: 3
  template:
    type: Python
    mode: cluster
    mainApplicationFile: "local:///code/pyspark-example.py" # path of your code in the docker image
    Arguments: # command line arguments for your spark python job
    - "wasb://[email protected]/episodes.avro"
    deps:
      jars:
        - "local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar" # You can safely ignore this and just copy paste for python code.
    sparkConf:
      "spark.hadoop.fs.azure.account.key.sparkdkakadia.blob.core.windows.net": "__blobsparkdkakadia__" # make sparkdkakadia(name of a azure blob) accessible for spark job.
    driver:
      cores: 0.5
      image: mdldatadev.azurecr.io/data/spark-py-example:latest # name of your docker image
      memory: "1024m"
      labels:
        version: 2.2.0
      serviceAccount: spark
    executor:
      cores: 2 
      instances: 2 # how many spark executors to launch
      image: dharmeshkakadia/spark-executor-py:latest # you can always use this same
      memory: "1024m" # how much memory executor should get
      labels:
        version: 2.2.0
    restartPolicy: Never

[ypzhang]

@liyinan926

How can I change the pull policy from the the yaml file? Your comment https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/124#issuecomment-394233175 assumes I am building the operator myself?

[liyinan926]

@ypzhang We will fix it and update the operator image. Stay tuned.

[liyinan926]

@ifilonenko on the need to include the example jar as a dependency in the 2.2 fork.

[liyinan926]

Fixes by #178. The operator image gcr.io/spark-operator/spark-operator:v2.2.0 has also been updated.

[ypzhang]

@dharmeshkakadia

It worked after a few retries on my end. Unfortunately I lost track which change did the trick...

Thanks for your info though...

[ypzhang]

Sorry to keep bugging on this closed thread. But does the operator support NFS volume mounting? I tried to replace the hostPath volume in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/spark-2.2-support/examples/spark-pi-py.yaml by nfs volume but it does not have any effect: the pod ends up running but nothing is mounted. (I'll provide the detailed yaml if needed). Thanks!

[liyinan926]

Mounting volumes requires the initializer, which must be enabled (see Configuration). The initializer is an alpha Kubernetes feature and only works on a cluster with alpha features enabled.

[dsterda]

Is it now possible to run PySpark jobs from the master branch? I noticed that @liyinan926 updated the documentation on master to include Python a few weeks ago, but I was having trouble getting it to work (same error as above: _Python applications are currently not supported for Kubernetes_).

I am running operator image gcr.io/spark-operator/spark-operator:v2.3.1-v1alpha1-latest (via kubectl apply -f manifest/spark-operator.yaml), and driver/executor image gcr.io/ynli-k8s/spark-py:v2.4.0-SNAPSHOT (via kubectl apply -f examples/spark-pyfiles.yaml).

I saw this note in the documentation:

Python binding for PySpark will be available in Apache Spark 2.4, and currently requires building a custom Docker image from the Spark master branch.

I'm assuming this refers to both the operator image and the driver/executor image (since it seems that I'm running 2.4 in the driver/executor, but 2.3.1 in the operator). If that's the case, is there any documentation on from where to pull (or how to build) the operator with Spark 2.4 to enable Python support? Or must I use the 2.2 branch?

[liyinan926]

I'm assuming this refers to both the operator image and the driver/executor image (since it seems that I'm running 2.4 in the driver/executor, but 2.3.1 in the operator). If that's the case, is there any documentation on from where to pull (or how to build) the operator with Spark 2.4 to enable Python support? Or must I use the 2.2 branch?

Yes, the operator image must be based on a Spark image built from the master branch of Spark. I will build and push an image tagged gcr.io/spark-operator/spark-operator:v2.4.0-SNAPSHOT-v1alpha1-latest based on gcr.io/spark-operator/spark-py:v2.4.0-SNAPSHOT.

[dharmeshkakadia]

I just published yesterday's operator master built with gcr.io/ynli-k8s/spark-py:v2.4.0-SNAPSHOT spark image at dharmeshkakadia/spark-operator:upstream-8-13. Feel free to try.

[liyinan926]

Thanks @dharmeshkakadia ! I also just pushed gcr.io/spark-operator/spark-operator:v2.4.0-SNAPSHOT-v1alpha1-latest, which is based on gcr.io/spark-operator/spark-py:v2.4.0-SNAPSHOT.

[dsterda]

Thanks for the quick responses @liyinan926 and @dharmeshkakadia ! I was able to get it to work using gcr.io/spark-operator/spark-operator:v2.4.0-SNAPSHOT-v1alpha1-latest (although I'm sure dharmeshkakadia/spark-operator:upstream-8-13 would have worked as well).

[aditanase]

Hi all, quick question - where can one find the Dockerfile used to build the various gcr.io/spark-operator/spark-operator:* images? I'm trying to debug some 2.3.1 vs 2.4.0 issues as well and would like to understand the exact lineage of the docker images.

Thanks!

[liyinan926]

@aditanase Here is the Dockerfile you referred to: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/Dockerfile. The Dockerfile of the base Spark image (the gcr.io/spark-operator/spark:* ones) is at https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.

[aditanase]

@liyinan926 Thanks! That's perfect, I am using the same strategy, I wasn't sure whether you're building from the spark distro Dockerfile or with a custom one.