Spark-on-k8s-operator: Integration with Prometheus for metrics and monitoring

[liyinan926]

[felipejfc]

any ETA?

[liyinan926]

I don't have an ETA yet. Wondering what kind of metrics people would like to have?

[felipejfc]

things like:

• total active tasks
• total running executors
• completed tasks per executor
• completed tasks per minute
• maybe disk and network metrics
• stages: active, completed, skipped, failed, pending
• tasks: active, completed, skipped, failed, pending
• driver task active, failed, completed
• RDDs

what do you think? I got this examples from:

https://www.datadoghq.com/blog/monitoring-spark/
http://www.hammerlab.org/2015/02/27/monitoring-spark-with-graphite-and-grafana/

[liyinan926]

Some of the metrics can be collected from outside the Spark applications, but looks like most of the metrics have to be collected from inside the applications, e.g., stage/task/rdd metrics. The operator can be extended to expose metrics that can be collected outside the applications.

[liyinan926]

One thing the operator can do to help on collecting metrics inside applications is automatically create and inject the necessary metrics configuration file into an application.

[andrehp]

It would be nice to have the metrics that the spark master exposes, like general metrics from the jobs. Having access to the metrics datadog monitors currently would be great: https://docs.datadoghq.com/integrations/spark/

I am currently using the sparkoperator with a spark streaming application and having metrics like number of restarts, running time and this kind of information would also be great.

[liyinan926]

I'm gonna create a draft design on this and share it here.

[liyinan926]

For inner-application metrics that can only be collected from inside the application driver/executors, the best option is to use a custom sink that pushes the metrics to Prometheus. The spark-metrics library can do this. I created https://github.com/banzaicloud/spark-metrics/issues/14 for a reference Dockerfile with the spark-metrics lib and its dependencies built-in. The operator then can focus on metrics that are across application runs.

[felipejfc]

make sense @liyinan926

[kumare3]

We have started looking into this. We hope to have some implementation soon. We might want to consult for identifying the right metrics to capture. We will update this issue when our branch is ready for review. These metrics will be operator level metrics only

[akhurana001]

Here are some of the initial metrics we think we can support. We can chat more during our sync-up and close out on this:

Metrics:

• spark_app_submit_count
• spark_app_success_count
• spark_app_failure_count
• spark_app_running_count
• spark_app_success_execution_time
• spark_app_failure_execution_time

• spark_app_running_executor_count
• spark_app_failed_executor_count

In-addition we will plug in prometheus as the metricsProvider for workqueue metrics for both SparkApplication and ScheduledSparkApplication.

Label Support:
Prometheus has label support where we can append one or more labels to a metric to get more granular insight across various dimensions. (https://github.com/prometheus/client_golang/blob/master/prometheus/examples_test.go#L50)
We were thinking of letting users specify up-to x (2 ?) labels and apply them while posting metrics. The label names will be provided as arguments to the operator (below) while the values will be provided in the SparkApplication spec.

Configurations:
In-addition, we were thinking of adding some arguments to the operator so that the metrics endpoint etc. are configurable if need be:

• -enable-metrics=true (required)
• -metrics-port=10254 (optional)
• -metrics-endpoint=/metrics (optional)
• -metrics-prefix=service (optional)
• -metrics-labels= label1,label2 (optional)

[liyinan926]

The proposal looks good to me in general. For metric labels, I think we should separate labels supposed to apply to all applications from the per-application ones. Per-application labels and values can be passed in through fields in the SparkApplication spec, whereas global labels are passed in through a command-line flag. Thoughts?

[akhurana001]

Thanks Yinan, I was originally thinking of only supporting global labels.

On the per-application labels, my understanding was that prometheus label names are associated at metric creation and it's not possible to change the labels dynamically. Won't adding per-app labels force us to create a new metric every time ?

[liyinan926]

I was thinking that each SparkApplication has its own set of metrics. It probably doesn't make much sense to have a single spark_app_running_executor_count across all applications?

[liyinan926]

266 adds support for automatic configuration for exposing internal Spark driver and executor metrics to Prometheus.

[liyinan926]

@andrehp @felipejfc @kumare3 @akhurana001 Please check out https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#monitoring.

[felipejfc]

seems good @liyinan926! thanks!