Spark-on-k8s-operator: readiness/liveness probes

[jdelgadillo]

We've been testing upgrades with other pods and noticed that upgrades went more smoothly when we added readiness/liveness probes to our deployments.

We don't see readiness/liveness probes for the SparkApplication CRD. Is there anything we need to add to our SparkApplication definition to ensure minimal or no downtime if we have to update the image we're using with our Spark Application?

[liyinan926]

What kind of readiness/liveness checks do you have in mind? They are generally useful for long-running serving workloads, and less for batch workloads IMO.

[jdelgadillo]

we have long running Streaming Spark Applications that could be affected if we had to switch out the Spark Operator while the Streaming application was running.

[skonto]

@jdelgadillo could you describe what failed during your tests?

[jdelgadillo]

@skonto We don't have failing tests. We're just planning for upgrades. We've found that having readiness/liveness probes helps reduce or eliminate downtime for our deployments. As we were looking into how we'd upgrade the Spark operator, we were wondering if there was a way to configure the probes for our Spark applications--specifically ones that are executing long running streaming jobs.

[jdelgadillo]

An alternative question: What are the best practices for upgrading the Google Spark Operator and Spark Applications?

[liyinan926]

@jdelgadillo the operator can generally be safely upgraded while having running SparkApplictions as long as the new version is based on a compatible version of Spark. For example, if both the new and old versions of the operator are based on the same minor version of Spark, e.g., 2.4.x. While the operator is offline, the status section of your SparkApplictions won't be updated. But things will catch up once the operator is back online. We maintain API backward compatibility in all beta versions. There might be behavior changes between versions that are not backward compatible. But in this case, we will make sure we provide an upgrade guide in the documentation.

[infa-angoel]

I also have similar requirement and need liveness probe to be added for spark application. @jdelgadillo did you find any solution for this ?

[jdelgadillo]

@infa-angoel we wound up using monitoring via prometheus/grafana to ensure our pods were running as expected