Spark-on-k8s-operator: Spark-Operator and Jupyter Notebooks

[metrofun]

Greetings,

I'm new to spark-operator and would like to know whether it can be used together with Jupyter Notebooks?

Spark-Operator latest beta seems to use Apache Spark 2.4, which comes with client mode support that allows for notebook-likes. Does it mean spark-operator can be leveraged by Jupyter? Any further pointers/examples to start piping it together?

[liyinan926]

The Spark operator is designed to work well with cluster mode. It currently doesn't support client mode.

[metrofun]

@liyinan926 Thank you for the prompt response

[alfccc]

is there an ticket for this work in the future?

[mattdornfeld]

Is this still true? I see that client mode is an option in the SparkApplication manifest?

[kyprifog]

@liyinan926 How is this intended to be used interactively in a notebook, I am running a spark script in a local jupyter notebook, but it stalls out with

TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

because the operator doesn't spin up any workers when running specifying the master to be the k8s master.

[kyprifog]

@liyinan926 do you have an answer for @mattdornfeld? I was going to take a shot at adding support for client mode but I dont want to duplicate any existing work.

[cwei-bgl]

It would be good to add Jupyter Notebooks support. I am waiting for it also.

[NKcqx]

me too

It would be good to add Jupyter Notebooks support. I am waiting for it also.

[kyprifog]

Just wanted to make those in this thread aware, in the mean time i've been using this helm chart https://github.com/JahstreetOrg/spark-on-kubernetes-helm developed by @jahstreet Its based around Livy which is a better fit architecturally for interactive sessions. spark-on-k8s-operator is still a good fit I think for non-interactive workflows. Happy to have your help working out any rough edges.