Charts: [incubator/kafka] Get metrics with jmx_prometheus_javaagent or jmx_prometheus_httpserver?

[ozbillwang]

Is this a request for help?:

It is a discussion about how to get proper metric (memory and CPU usage) in kafka, with current setting, it can't.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

it should be a feature request. With exist command running in metric container in kafka pod, I can see the metrics with java_, and kafka_, but I can't see any jvm_ metrics.

Version of Helm and Kubernetes:

helm v2.9.1
kubernetes v1.10.5

Which chart:

incubator/kafka

What happened:

No jvm_ metrics exported to Prometheus Operator

What you expected to happen:

Get jvm_ metrics, so I can easily view the metric in grafana with dashboard kafka overview

How to reproduce it (as minimally and precisely as possible):

Deploy kafka chart with --set prometheus.jmx.enabled=true,prometheus.kafka.enabled=true

Anything else we need to know:

The owner of jmx-exporter mentioned in README in jmx-exporter

This exporter is intended to be run as a Java Agent, exposing a HTTP server and serving metrics of the local JVM. It can be also run as an independent HTTP server and scrape remote JMX targets, but this has various disadvantages, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

Not really understood what's that meaning, until I saw this comment:

https://github.com/prometheus/jmx_exporter/issues/111#issuecomment-341983150

@brian-brazil can you add some sort of tip to the readme that jvm_* metrics are only exposed when using the Java agent? It took me an hour or two of troubleshooting and searching old issues to figure this out, after playing only with the HTTP server version. Thanks!

So jmx-exporter has to be run with java agent to get jvm_ metrics (for memory and CPU uages). jmx_prometheus_httpserver doesn't support, but it is the default setting in current kafka helm chart.

https://github.com/kubernetes/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82

command:
- sh
- -exc
- |
  trap "exit 0" TERM; \
  while :; do \
  java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 \
  -XshowSettings:vm \
  -jar \
  jmx_prometheus_httpserver.jar \              # <<< here
  {{ .Values.prometheus.jmx.port | quote }} \
  /etc/jmx-kafka/jmx-kafka-prometheus.yml & \
  wait $! || sleep 3; \
  done

What should we do, I am not sure if we need do a if-else condition in chart template to support both ways. Since java agent need be run with kafka in same container, so kafka docker image need be rebuilt. Or there are exist image to support java agent in kafka already. Maybe create a new chart will be better solution. Need your feedback.

[benjigoldberg]

Thanks for filing the issue @ozbillwang and adding these details. I'm happy to look into this in a few days.

/assign @benjigoldberg

[ozbillwang]

@benjigoldberg

There are extra informations about this issue, maybe it will be useful for you. Paste here to take a reference: https://github.com/danielqsj/kafka_exporter/issues/53

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[ozbillwang]

@benjigoldberg

Any updates?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[nitishpt]

Hey @ozbillwang Were you able to get this resolved? I think I am stuck at the same problem. Able to get the kafka_exporter metrics but not any jmx_exporter metrics. Any help would be highly appreciated.

[ozbillwang]

You need run it as java agent (to write a new chart with several command lines changed in the place I pointed (https://github.com/helm/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82) , but I don't have time on it after I raised this ticket.

If you have time, you can work it out and update me.

[ozbillwang]

Seems @niqdev had raised a PR to fix it, but it was closed without merge.

https://github.com/helm/charts/pull/10754

You can take a reference, if you want to follow the fix.

[ozbillwang]

@benjigoldberg

Any updates for us after 2018?

[nitishpt]

Seems @niqdev had raised a PR to fix it, but it was closed without merge.

10754

You can take a reference, if you want to follow the fix.

Yeah, I looked into the PR. Though I want making changes in the chart locally as a last fix. I was going through the cp-helm-charts for kafka-connect and schema-registry as well and I think the issue persists there as well. Though I think I will raise an issue for this there maybe. Thanks!