Spark-on-k8s-operator: Integration with job history server with GCS as the backend storage

[liyinan926]

• [ ] Provide a reference template for the job history server Deployment.
• [ ] Add documentation.

[mrow4a]

Spark history server does not need another image, requires only deployment and service yamls. One need to add to yaml command (override the entrypoint of default image) as /opt/spark/sbin/start-history-server.sh with some specific storage configurations, for S3 these are secrets. I can probably PR the proposal we use and iterate over if that fits for GCS (it is a prototype version).

(This is just a try-out version https://github.com/cerndb/spark-on-k8s-operator/commit/e4eaf5442db26b75fc74fffc1f43f5f238510d74)

[liyinan926]

@mrow4a Thanks for the reference to what you have built! For logging to GCS, we need an image that has the GCS connector built-in, e.g., this one. I think we can host different templates here for different backend storage, e.g., one for GCS and one for S3. I'm also thinking of adding a field to SparkApplicationSpec, e.g., eventLogDir that if set means event logging is enabled and the value is used to set spark.eventLog.di. This saves users from having to set both keys in sparkConf.

[mrow4a]

@liyinan926 Have you considered running Spark History as a separate process in the Spark Operator? (https://docs.docker.com/config/containers/multi-service_container/)

[mrow4a]

Another thing to consider with job history server being S3 or GCS is how renames are handled. We observe timeouts with S3 on renaming "*.inprogress" files to normal file when application finishes - in HDFS rename is atomic metadata operation, while in S3 it is data operation, possibly of multi MBs of data.

[liyinan926]

@mrow4a it's an interesting idea about running the history server as a separate process in the operator itself. I think this is a better option than running the history server as a separate Deployment.

[mrow4a]

@liyinan926 there could possibly be also a flag passed to operator in order not to spawn history server process - in case someone wants to customise it and have separate deployment.

[liyinan926]

there could possibly be also a flag passed to operator in order not to spawn history server process - in case someone wants to customise it and have separate deployment.

Makes sense. There could also be a flag for specifying the even log path.

[yuchaoran2011]

To chime in here, I have a PR pending for a history server chart, although I haven't figured out how to fix the CI error (it worked fine during testing on my cluster). It supports using HDFS and PersistentVolumeClaims as event log storage and provides a sample NFS-backed PVC setup. I could also enhance it to support S3 and GCS.

[liyinan926]

@yuchaoran2011 it would be great if you could also make it work with S3 and GCS.

[yuchaoran2011]

@liyinan926 I added support for GCS in my PR and fixed the CI. Feel free to give it a try.