Cp-docker-images: Kafka Connect JMX : source & sink task metrics missing for tasks in failed state

[mikouaj]

Overview

Kafka Connect exposes various metrics via JMX. Those metrics can be exported i.e. by _Prometheus JMX Exporter_ for further processing.

One of crucial attributes is connector's task status. According to official Kafka docs, status is available as status attribute of following MBean:

kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
status - The status of the connector task. One of 'unassigned', 'running', 'paused', 'failed', or 'destroyed'.

Issue

Generally connector-task-metrics are exposed propery for tasks in running status but not exposed at all if task is failed.

Failed Task appears properly with failed status when queried via REST API:

$ curl -X GET -u 'user:pass' http://kafka-connect.mydomain.com/connectors/customerconnector/status
{"name":"customerconnector","connector":{"state":"RUNNING","worker_id":"kafka-connect.mydomain.com:8080"},"tasks":[{"id":0,"state":"FAILED","worker_id":"kafka-connect.mydomain.com:8080","trace":"org.apache.kafka.connect.errors.ConnectException: Received DML 'DELETE FROM mysql.rds_sysinfo ......"}],"type":"source"}

Failed Task doesn't appear as bean with connector-task-metrics type when queried via JMX:

$ echo "beans -d kafka.connect" | java -jar target/jmxterm-1.1.0-SNAPSHOT-uber.jar -l localhost:8081 -n -v silent | grep connector=customerconnector
kafka.connect:connector=customerconnector,task=0,type=task-error-metrics
kafka.connect:connector=customerconnector,type=connector-metrics
$

Expected result

It is expected, that bean with connector-task-metrics type will appear also for tasks that failed.

Below is example of how beans are properly registered for tasks in Running state:

$ echo "get -b kafka.connect:connector=sinkConsentSubscription-1000,task=0,type=connector-task-metrics status" | java -jar target/jmxterm-1.1.0-SNAPSHOT-ube r.jar -l localhost:8081 -n -v silent
status = running;
$

[bmoe24x]

Has anybody found a workaround to this? Seems like a pretty significant operational issue... especially when trying to utilize Prometheus and AlertManager. The REST API gives back a FAILED status when a task fails but JMX does not... Would be very very handy for both situations to match

[l0n3r4n83r]

@bmoe24x it's not ideal but we alert on missing data when a task/connector fails the jmx metrics stop publishing

[bmoe24x]

@bmoe24x it's not ideal but we alert on missing data when a task/connector fails the jmx metrics stop publishing

So you essentially set up an alert that triggers when that metric is missing?

[l0n3r4n83r]

@bmoe24x it's not ideal but we alert on missing data when a task/connector fails the jmx metrics stop publishing

So you essentially set up an alert that triggers when that metric is missing?

Yes

[strigona-worksight]

This issue is being addressed in Kafka here: https://github.com/apache/kafka/pull/8502 and tracked here: https://issues.apache.org/jira/browse/KAFKA-9066

I don't see an ETA yet.

[bmoe24x]

@bmoe24x it's not ideal but we alert on missing data when a task/connector fails the jmx metrics stop publishing

So you essentially set up an alert that triggers when that metric is missing?

Yes

Thanks for the idea!

As for everyone else, thanks to @mikouaj there is a Jira issue here

...and a pull request with some recent traction thanks to @C0urante found here Would love to get this included in the next release, would save a lot of effort on a hacky solution!

[bmoe24x]

This issue is being addressed in Kafka here: apache/kafka#8502 and tracked here: https://issues.apache.org/jira/browse/KAFKA-9066

I don't see an ETA yet.

Beat me to it by 13 seconds it looks like! Lets hope this functionality gets pulled in soon 🤞