telegraf docker plugin fails to parse the metric when running under kubernetes

[thedebugger]

Bug report

Possibly related to 2780, telegraf docker plugin errors out when pulling stats from docker

unable to parse 'docker_container_cpu,annotation.kubernetes.io/config.seen=2017-08-08T08:38:22.137024605-07:00,annotation.kubernetes.io/config.source=api,annotation.kubernetes.io/created-by={\"kind\":\"SerializedReference\"\\,\"apiVersion\":\"v1\"\\,\"reference\":{\"kind\":\"DaemonSet\"\\,\"namespace\":\"lk-system\"\\,\"name\":\"splunk-forwarder\"\\,\"uid\":\"ef4fc440-7bad-11e7-967d-005056abb62d\"\\,\"apiVersion\":\"extensions\"\\,\"resourceVersion\":\"123506\"}}': missing fields\nunable to parse ',app=splunk-forwarder,container_image=gcr.io/google_containers/pause-amd64,container_name=k8s_POD_splunk-forwarder-0x41k_lk-system_9c1ae0bc-7c4f-11e7-90dc-005056abb62d_0,container_version=3.0,controller-revision-hash=1134785655,cpu=cpu11,engine_host=xxxx,host=xxxx,io.kubernetes.container.name=POD,io.kubernetes.docker.type=podsandbox,io.kubernetes.pod.name=splunk-forwarder-0x41k,io.kubernetes.pod.namespace=lk-system,io.kubernetes.pod.uid=9c1ae0bc-7c4f-11e7-90dc-005056abb62d,pod-template-generation=1 container_id=\"e9345f77b9367340512726c83186f267223de8dbb5e0055cd010d0d6b2ae7148\",usage_total=0i 1502239585000000000': missing measurement

Relevant telegraf.conf:

[[inputs.docker]]
endpoint = "unix:///var/run/docker.sock"
container_names = []
timeout = "180s"
interval = "180s"

System info:

1. telegraf v1.1.0
2. Linux xxxxx 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
3. docker 1.12.6

Steps to reproduce:

1. Install telegraf, docker, telegraf docker plugin and k8s
2. Check telegraf logs

Expected behavior:

It should parse and send metrics to output sink

Actual behavior:

it fails to parse

Other info

We are running kubernetes 1.7

Let me know if you need more information.

[danielnelson]

Can you retest with the latest release?

[danielnelson]

Please reopen after testing with latest release.

[pilsy]

You need to exclude annotation.kubernetes* from that list, what's happening is sometimes the kubernetes annotations have no value which causes that error.

[rossedman]

@pilsy how do you exclude it?

[danielnelson]

I think adding this option to your docker plugin will work:

[[inputs.docker]]
  docker_label_exclude = ["annotation.kubernetes*"]

[roffe]

@danielnelson thank you, got rid of my errors.

I'm on kubernetes 1.8

[danielnelson]

I added this to the docker README: https://github.com/influxdata/telegraf/commit/ad4a5aa7a0409a528782cbfeb6b0038b629efeb1