Spark-on-k8s-operator: Spark operator roadmap for 2019

[liyinan926]

• [ ] Support for multiple versions of Spark: #182.
• [ ] Support for priority queues and basic priority-based scheduling: #313.
• [ ] Kerberos support that will come in Spark 3.0: #302.
• [ ] sparkctl as a kubectl plugin: #318.
• [ ] Pod template support that will come in Spark 3.0.

[skonto]

I can help with the pod template support, create a poc PR removing the web hook dependency.

[liyinan926]

Thanks @skonto. However, we cannot completely get rid of the webhook as it's still needed for Spark prior to 3.0.

[skonto]

@liyinan926 Maybe we could do breaking changes for new releases, Spark 3.0.0 and the related operator version that supports it does not need to be compatible with 2.4.x.

[AceHack]

Any chance on getting autoscaling on the roadmap?

[liyinan926]

The only kind of autoscaling that make sense at the Spark application level for the native Kubernetes mode is dynamic resource allocation, i.e., autoscaling the number of executors for an application at runtime based on the load. However, Spark on k8s doesn't support dynamic resource allocation yet due to the lack of a proper external shuffle service implementation.

[ChethanUK]

Is there a new Roadmap? It's like one year now..

[liyinan926]

Will create a new one to replace this.