Spark-on-k8s-operator: Spark-pi-prometheus does not expose metrics port 8090

[PavelPaladin]

Tried this example without any changes, and I figured out that spark-operator doesn't expose 8090 port on pods\services even if exposeDriverMetrics + exposeExecutorMetrics is set to "true".

$kubectl describe pod spark-pi-driver
Name: spark-pi-driver
Namespace: default
Priority: 0
PriorityClassName:
Node: node3/68.183.71.40
Start Time: Thu, 11 Apr 2019 11:49:01 +0300
Labels: spark-app-selector=spark-575f1b9b71f74c918fa4d78afc830ad2
spark-role=driver
sparkoperator.k8s.io/app-name=spark-pi
sparkoperator.k8s.io/launched-by-spark-operator=true
version=2.4.0
Annotations: prometheus.io/path: /metrics
prometheus.io/port: 8090
prometheus.io/scrape: true
Status: Running
IP: 10.233.71.22
Containers:
spark-kubernetes-driver:
Container ID: docker://96a8b70f6f5abc5478b5e5c79df784d67319f337e70b20d403be11230ad6a2a6
Image: gcr.io/spark-operator/spark:v2.4.0-gcs-prometheus
Image ID: docker-pullable://gcr.io/spark-operator/spark@sha256:9c585c4a794624439a5c957ae2d2cafe2048fee73c47de1ae28cfe03c9c5584f
Ports: 7078/TCP, 7079/TCP, 4040/TCP

Same with services, no 8090 exposed anywhere:
$kubectl get svc | grep spark
spark-pi-1554972540777-driver-svc ClusterIP None 7078/TCP,7079/TCP 20m
spark-pi-ui-svc NodePort 10.233.21.152 4040:30543/TCP 20m
sparkop-metrics ClusterIP 10.233.27.78 10254/TCP 18h

However, if I jump in pod itself and try some curls to 127.0.0.1:8090 - I will find metrics working.
For now, I can expose 8090 port manually via simple kubernetes svc and scrape metrics without any problems, but just will be happy if spark-operator will expose them itself.
Or did I miss something?

[liyinan926]

Sure, we can expose the port by patching it through the webhook.

[glitch-k8s]

@liyinan926 I am using Sparkoperator to run my Spark applications.
In my SparkApplication, I have exposed REST APIs for some functional need. I am unable to access my REST API port.
Can you please suggest how can I expose port using Sparkoperator framework.

[rolandjohann]

@liyinan926 recently there was a commit at spark that implements custom pod templates for drivers and executors. With this we should be able to omit using webhooks

https://github.com/apache/spark/commit/f6cc354d83c2c9a757f9b507aadd4dbdc5825cca

[liyinan926]

@rolandjohann pod templating is only available starting Spark 3.0. For Spark 2.4 and earlier, the webhook is still needed.

[alexakr]

@liyinan926 is it going to be addressed in the near future?

[arijitdey]

@liyinan926 How can I expose the port 8090 in the driver and executor pods for my PodMonitor or ServiceMonitor to pick it up?

[sahil-sawhney]

Hey @liyinan926 could you suggest some example for achieving " expose the port 8090 by patching it through the webhook."

[liyinan926]

@arijitdey @sahil-sawhney PR #935 will fix this and expose the port.

[sahil-sawhney]

Thanks a lot, @liyinan926 . In which spark operator version this change would be released?

[liyinan926]

It's included in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/releases/tag/v1beta2-1.1.2-2.4.5 and will also be included in the next version (0.7.1) of the Helm chart.

[sahil-sawhney]

Thanks a ton @liyinan926 :)