Datadog-agent: Tagger error on Kubernetes

Created on 10 Jan 2018  路  17Comments  路  Source: DataDog/datadog-agent

Describe what happened:
I am trying to set up log collection for Docker containers on a Kubernetes cluster.

Snippet from my config:

    logs:
    - type: docker
      image: "datadog/agent"
      servie: datadog
      source: datadog

Logs are not being sent to Datadog, and I see a lot of these errors in the DD container log:

2018-01-10 16:47:50 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cacb7a1688be922adcb89f39950d4526cafb1c0e92ee050f630fe0e90132b0b not found in podlist

Describe what you expected:
Working log collection.

Steps to reproduce the issue:
Deploy DD agent on Kuberentes with Daemonset.

Additional environment details (Operating System, Cloud provider, etc):

Kubernetes 1.8.6
Docker 0.13

agent status

Agent (v6.0.0-beta.7)

Status date: 2018-01-10 16:49:09.760419 UTC
Pid: 7
Python Version: 2.7.14
Logs:
Check Runners: 10
Log Level: info

Paths
=====
Config File: /etc/datadog-agent/datadog.yaml
conf.d: /etc/datadog-agent/conf.d
checks.d: /etc/datadog-agent/checks.d

Clocks
======
NTP offset: 0.00326513 s
System UTC time: 2018-01-10 16:49:09.760419 UTC

Host Info
=========
bootTime: 2018-01-09 20:04:23.000000 UTC
kernelVersion: 4.4.65-k8s
os: linux
platform: debian
platformFamily: debian
platformVersion: 9.3
procs: 63
uptime: 74247
virtualizationRole: guest
virtualizationSystem: xen

Hostnames
=========
ec2-hostname: ip-172-20-174-85.ec2.internal
hostname: i-0d84be79a0113f36b
instance-id: i-0d84be79a0113f36b
socket-fqdn: dd-agent-j6b4h
socket-hostname: dd-agent-j6b4h

=========

Collector

Running Checks
==============
cpu
---
Total Runs: 29
Metrics: 6, Total Metrics: 168
Events: 0, Total Events: 0
Service Checks: 0, Total Service Checks: 0

disk
----
  Total Runs: 29
  Metrics: 160, Total Metrics: 4640
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

docker
------
  Total Runs: 29
  Metrics: 266, Total Metrics: 7342
  Events: 0, Total Events: 5
  Service Checks: 1, Total Service Checks: 29

file_handle
-----------
  Total Runs: 29
  Metrics: 1, Total Metrics: 29
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

io
--
  Total Runs: 29
  Metrics: 52, Total Metrics: 1472
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

kube_dns
--------
  Total Runs: 29
  Metrics: 41, Total Metrics: 1189
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

load
----
  Total Runs: 29
  Metrics: 6, Total Metrics: 174
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

memory
------
  Total Runs: 29
  Metrics: 14, Total Metrics: 406
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

network
-------
  Total Runs: 29
  Metrics: 20, Total Metrics: 580
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

ntp
---
  Total Runs: 29
  Metrics: 1, Total Metrics: 28
  Events: 0, Total Events: 0
  Service Checks: 1, Total Service Checks: 29

uptime
------
  Total Runs: 29
  Metrics: 1, Total Metrics: 29
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

Loading Errors
==============
docker_daemon
-------------
Core Check Loader:
Check docker_daemon not found in Catalog

  JMX Check Loader:
    check is not a jmx check, or unable to determine if it's so

  Python Check Loader:
    No module named docker_daemon

========

JMXFetch

Initialized checks
==================
no checks

Failed checks
=============
no checks

=========

Forwarder

CheckRunsV1: 29
IntakeV1: 6
RetryQueueSize: 0
Success: 64
TimeseriesV1: 29

API Keys status
===============
https://6-0-0-app.agent.datadoghq.com,*******4aa08: API Key valid

=========

DogStatsD

Checks Metric Sample: 16694
Event: 6
Events Flushed: 6
Number Of Flushes: 29
Series Flushed: 12074
Service Check: 377
Service Checks Flushed: 393
Dogstatsd Metric Sample: 697

[deprecated] tealogs componentagger teacontainers
Source
picture macat
👍3

Most helpful comment

I got the same error.

[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:14 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://3ab71114c06ebd2ed0b641f604ef7e13034399c86de2bf38d323139e3db86735 not found in podList
  • Kubernetes v1.8.7-gke
  • Datadog Agent v6.0.2
  • Install agent using Helm (stable/datadog)


values.yaml

image:
  repository: datadog/agent               # Agent6
  tag: 6.0.2  # Use 6.0.0-jmx to enable jmx fetch collection
  pullPolicy: IfNotPresent

daemonset:
  enabled: true
  updateStrategy: RollingUpdate

deployment:
  enabled: false
  replicas: 1

kubeStateMetrics:
  enabled: true

datadog:
  apiKey: xxxxx
  name: dd-agent
  logLevel: WARNING
  collectEvents: false
  env:
    - name: DD_LOGS_ENABLED
      value: "true"
  leaderLeaseDuration: 600s
  confd:
    kubernetes.yaml: |-
      init_config:
      instances:
        - port: 4194
          collect_events: True
          namespace_name_regexp: .*
    docker_daemon.yaml: |-
      logs:
        - type: docker
          service: docker
          source: docker
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 256m
      memory: 512Mi

rbac:
  create: true
  serviceAccountName: default

tolerations: []

kube-state-metrics:
  rbac:
    create: true
    serviceAccountName: default

picture dtan4 on 8 Mar 2018
👍11

All 17 comments

Hey @macat, thanks for opening up this issue!
As you are running the beta7, the log agent binary is embedded in the image.
Though, it is still running in a separate binary. Hence, I'd like to make sure that it is running.

To this extent, could you share with us:

  • Your datadog.yaml.
  • The manifest you use for your daemonset.
  • The output of: cat /var/log/datadog/agent.log | grep "logs-agent".

Finally, would it be possible to open a ticket with our support team [email protected] with the aforementioned and a reference to this ticket ?

Also, from the upcoming beta.8 the logs-agent will run as a goroutine in the main agent process.

Best,
.C

CharlyF picture CharlyF on 10 Jan 2018

I deployed the agent using daemonset.

Configmap:

apiVersion: v1
data:
  datadog.api.key: oo
  datadog.yaml: |
    log_level: debug
    log_enabled: true
    listeners:
    - name: docker
    config_providers:
    - name: kubelet
      polling: true
  docker_daemon.yaml: |
    init_config:
      docker_root: /host
    logs:
    - type: docker
      image: "drone/agent:0.8.2"
      service: drone
      source: drone
    - type: docker
      image: "datadog/agent:6.0.0-beta.7"
      servie: datadog
      source: datadog
    - type: file
      path: /host/var/log/syslog
      service: syslog
      source: syslog

DaemonSet:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: dd-agent
spec:
  template:
    metadata:
      labels:
        app: dd-agent
      name: dd-agent
    spec:
      containers:
        - env:
            - name: DD_API_KEY
              valueFrom:
                configMapKeyRef:
                  key: datadog.api.key
                  name: datadog
            - name: KUBERNETES
              value: "true"
            - name: DD_LOG_ENABLED
              value: "true"
          image: datadog/agent:6.0.0-beta.7
          imagePullPolicy: Always
          name: dd-agent
          ports:
            - containerPort: 8125
              name: dogstatsdport
              protocol: UDP
          resources:
            limits:
              cpu: 300m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi
          volumeMounts:
            - mountPath: /var/run/docker.sock
              name: dockersocket
            - mountPath: /host/proc
              name: procdir
              readOnly: true
            - mountPath: /host/sys/fs/cgroup
              name: cgroups
              readOnly: true
            - mountPath: /conf.d
              name: datadog-config-volume
              readOnly: true
            - mountPath: /opt/datadog-agent/run
              name: datadog-run
            - mountPath: /host/var/log
              name: varlog
      volumes:
        - hostPath:
            path: /var/run/docker.sock
          name: dockersocket
        - hostPath:
            path: /proc
          name: procdir
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroups
        - hostPath:
            path: /opt/datadog-agent/run
          name: datadog-run
        - hostPath:
            path: /var/log
          name: varlog
        - configMap:
            items:
              - key: docker_daemon.yaml
                path: docker_daemon.yaml
              - key: datadog.yaml
                path: datadog.yaml
            name: datadog
          name: datadog-config-volume
macat picture macat on 11 Jan 2018


Log output

ks logs dd-agent-dzhxw | grep "logs-agent\|tagger"
2018-01-11 19:42:27 UTC | INFO | (tagger.go:80 in Init) | starting the tagging system
2018-01-11 19:42:27 UTC | INFO | (tagger.go:153 in tryCollectors) | docker tag collector successfully started
2018-01-11 19:42:27 UTC | INFO | (tagger.go:153 in tryCollectors) | kubelet tag collector successfully started
2018-01-11 19:42:28 UTC | INFO | (autoconfig.go:458 in GetChecks) | Core Check Loader: successfully loaded check 'logs-agent'
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748494615972 [Debug] Unable to get the hostname from the config file: host name is empty
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748494641947 [Debug] Trying to determine a reliable host name automatically...
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748494646361 [Debug] GetHostname trying GCE metadata...
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748495565347 [Debug] Unable to get hostname from GCE: unable to retrieve hostname from GCE: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748495580233 [Debug] GetHostname trying Docker API...
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748587468716 [Debug] GetHostname trying EC2 metadata...
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748589274044 [Debug] GetHostname trying EC2 metadata...
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | Starting logs-agent
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | Opening /host/var/log/syslog
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | Can't tail file twice: /host/var/log/syslog
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | Connecting to the backend: intake.logs.datadoghq.com:10516
2018-01-11 19:42:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699748992915488 [Info] starting the tagging system
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749290575132 [Info] kubelet tag collector successfully started
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749290600619 [Info] docker tag collector successfully started
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749293850318 [Debug] ecs tag collector cannot start: cannot find ECS agent
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749294009356 [Debug] candidate list empty, stopping detection
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749489652636 [Debug] found 7 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:29 UTC | INFO | (logs-agent.go:75 in func1) | 1515699749489757180 [Debug] listener message: [%!s(*collectors.TagInfo=&{kubelet docker://348cbab3e08e3418e0e8a914ed0f0950a39f127e7a91fa013986cf5834fc8604 [pod_name:dd-agent-dzhxw] [kube_namespace:kube-system kube_container_name:dd-agent kube_daemon_set:dd-agent] false}) %!s(*collectors.TagInfo=&{kubelet docker://d1bf0d65b578b67679ed4f8467dbef7b5482f41f42737a124b6d6f20a21d7d70 [pod_name:nvml-metrics-4fk67] [kube_namespace:kube-system kube_container_name:nvml-metrics kube_daemon_set:nvml-metrics] false}) %!s(*collectors.TagInfo=&{kubelet docker://46f0dec114fe102b11980b4f7d05ba7bd307eed307f5df58173391ca2d22452d [pod_name:jssgen-99163c49-2fd2-4051-b667-368f63d8ba77] [kube_namespace:jobs kube_container_name:99163c49-2fd2-4051-b667-368f63d8ba77] false}) %!s(*collectors.TagInfo=&{kubelet docker://0a69de8e5a92532112079d3bf25c3b0162274e34d17bdd3e7b975cd1d016db5d [pod_name:research-1703865212-2mrqp kube_replica_set:research-1703865212] [kube_namespace:research kube_container_name:research kube_deployment:research] false}) %!s(*collectors.TagInfo=&{kubelet docker://5c99e50b93513832910e89dd02b39c0dde082f2221e8ce21063405edef64aa46 [kube_job:airtable-sync-1515698940 pod_name:airtable-sync-1515698940-n6kq7] [kube_namespace:warehouse kube_container_name:airtable] false}) %!s(*collectors.TagInfo=&{kubelet docker://f9855ed3fb8f9e371f9dd3c5f1ac905d8ff1de23a055cc0cab5308fa9f77ca6c [pod_name:filebeat-5ng5s] [kube_container_name:filebeat kube_daemon_set:filebeat kube_namespace:kube-system] false})]
2018-01-11 19:42:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699753496937454 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699753496953995 [Debug] listener message: []
2018-01-11 19:42:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699758496681896 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699758496704813 [Debug] listener message: []
2018-01-11 19:42:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699763497810529 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699763497833833 [Debug] listener message: []
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a846cdd4bc3e21216e05674303f861478377f98087996a596a350f2e8b37b55e not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from docker: Error: No such container: 606941227fb96e01f28def3d231a4d60f89fe82ffac647235aa25661ffdb6a02
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://606941227fb96e01f28def3d231a4d60f89fe82ffac647235aa25661ffdb6a02 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from docker: Error: No such container: 9f9f53077f853a5d8b5596954e65824b3dcba6a7f51a437401987cd0b5741b42
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9f9f53077f853a5d8b5596954e65824b3dcba6a7f51a437401987cd0b5741b42 not found in podlist
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from docker: Error: No such container: 954cd70152938f486cc2de30816d110be43ee2a938701403d5b3f6bff22b8e4c
2018-01-11 19:42:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://954cd70152938f486cc2de30816d110be43ee2a938701403d5b3f6bff22b8e4c not found in podlist
2018-01-11 19:42:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699768496257064 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699768496296456 [Debug] listener message: []
2018-01-11 19:42:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699773496055666 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699773496071865 [Debug] listener message: []
2018-01-11 19:42:58 UTC | INFO | (logs-agent.go:75 in func1) | 1515699778497265903 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:42:58 UTC | INFO | (logs-agent.go:75 in func1) | 1515699778497282025 [Debug] listener message: []
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:42:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:43:03 UTC | INFO | (logs-agent.go:75 in func1) | 1515699783587632006 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:03 UTC | INFO | (logs-agent.go:75 in func1) | 1515699783587673100 [Debug] listener message: []
2018-01-11 19:43:08 UTC | INFO | (logs-agent.go:75 in func1) | 1515699788496328285 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:08 UTC | INFO | (logs-agent.go:75 in func1) | 1515699788496365944 [Debug] listener message: []
2018-01-11 19:43:13 UTC | INFO | (logs-agent.go:75 in func1) | 1515699793497242199 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:13 UTC | INFO | (logs-agent.go:75 in func1) | 1515699793497272329 [Debug] listener message: []
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:43:18 UTC | INFO | (logs-agent.go:75 in func1) | 1515699798496529791 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:18 UTC | INFO | (logs-agent.go:75 in func1) | 1515699798496567311 [Debug] listener message: []
2018-01-11 19:43:23 UTC | INFO | (logs-agent.go:75 in func1) | 1515699803497270762 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:23 UTC | INFO | (logs-agent.go:75 in func1) | 1515699803497306594 [Debug] listener message: []
2018-01-11 19:43:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699808497331010 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699808497370508 [Debug] listener message: []
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:43:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699813499422198 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699813499457242 [Debug] listener message: []
2018-01-11 19:43:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699818496725127 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699818496780184 [Debug] listener message: []
2018-01-11 19:43:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699823497689167 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699823497721130 [Debug] listener message: []
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:43:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699828495935820 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699828495971916 [Debug] listener message: []
2018-01-11 19:43:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699833496544864 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699833496582811 [Debug] listener message: []
2018-01-11 19:43:58 UTC | INFO | (logs-agent.go:75 in func1) | 1515699838499593235 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:43:58 UTC | INFO | (logs-agent.go:75 in func1) | 1515699838499610699 [Debug] listener message: []
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:43:58 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:44:03 UTC | INFO | (logs-agent.go:75 in func1) | 1515699843496805405 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:03 UTC | INFO | (logs-agent.go:75 in func1) | 1515699843496909130 [Debug] listener message: []
2018-01-11 19:44:08 UTC | INFO | (logs-agent.go:75 in func1) | 1515699848496602123 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:08 UTC | INFO | (logs-agent.go:75 in func1) | 1515699848496639616 [Debug] listener message: []
2018-01-11 19:44:13 UTC | INFO | (logs-agent.go:75 in func1) | 1515699853497268377 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:13 UTC | INFO | (logs-agent.go:75 in func1) | 1515699853497291496 [Debug] listener message: []
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:13 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:44:18 UTC | INFO | (logs-agent.go:75 in func1) | 1515699858496557794 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:18 UTC | INFO | (logs-agent.go:75 in func1) | 1515699858496624389 [Debug] listener message: []
2018-01-11 19:44:23 UTC | INFO | (logs-agent.go:75 in func1) | 1515699863496349412 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:23 UTC | INFO | (logs-agent.go:75 in func1) | 1515699863496386349 [Debug] listener message: []
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699868498557460 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:28 UTC | INFO | (logs-agent.go:75 in func1) | 1515699868498686565 [Debug] listener message: []
2018-01-11 19:44:28 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:44:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699873496997867 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:33 UTC | INFO | (logs-agent.go:75 in func1) | 1515699873497031524 [Debug] listener message: []
2018-01-11 19:44:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699878495956404 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:38 UTC | INFO | (logs-agent.go:75 in func1) | 1515699878495994126 [Debug] listener message: []
2018-01-11 19:44:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699883497032039 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:43 UTC | INFO | (logs-agent.go:75 in func1) | 1515699883497053155 [Debug] listener message: []
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://9eeab2830e2a526a4c0ce52f52c97c40dc038b6978fd0675f912fce120ef652c not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cbfb827ea55552efc60b63dc76f203c5348bf1028916d0c23778a4eafd19279 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a51034f4f20248665dd99277b02cdc508fa8a2e514b5fa2e6cabcc6798d98a0a not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://bd58ab00d44a0c0a6b777f6edbdcda3146fc10030f934c6e3d243871c0cb0112 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://fcdd306123ca4e5aef511868911d1b77e53f3d718607d35a781b8a531b3d6f34 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://96b41ef7d1efd3414340ac37664eb2756955e2e50d9c4c14ea2125efc3d63e1c not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://5e2241019efa7232b4fa48c947904c685bbb1d975edb2e3fb3a04c1050811654 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://a6510f1f71327d0ac2d94677c650347c0a37287bffaf924b14e81baec56d3744 not found in podlist
2018-01-11 19:44:43 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://2119a51a8e7a99532f92f6cac8f344416fa9296b57b62c35f49088615c127715 not found in podlist
2018-01-11 19:44:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699888496382337 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:48 UTC | INFO | (logs-agent.go:75 in func1) | 1515699888496408525 [Debug] listener message: []
2018-01-11 19:44:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699893496835227 [Debug] found 0 changed pods out of 7, new resversion 72898050
2018-01-11 19:44:53 UTC | INFO | (logs-agent.go:75 in func1) | 1515699893496867396 [Debug] listener message: []

macat picture macat on 11 Jan 2018

Hey @macat,

I wanted to keep you updated on this - As it stands the log agent does not fully support the Autodiscovery feature in kubernetes.
It is possible to specify an image as you did for the agent to tail the logs of the containers running it with Docker, but the support with an orchestrator overlay is not guaranteed.

We are currently working on improving this and have a better support for Autodiscovery on our roadmap.

Would you mind if we get your details through a ticket (to our support-team) and open a Feature Request so that we can keep you updated when this is released ?

Best,

.C

CharlyF picture CharlyF on 12 Jan 2018

OK, thank you. Actually, I realized that error does not prevent my logs to be shipped. After removing the image filter, all logs are being transferred to DataDog, which is great. The agent even picked up the Kubernetes attributes from the containers, so I'm very happy.

macat picture macat on 12 Jan 2018
🎉1

I'm having this same issue with beta 8, and I am not seeing logs from my service containers. I'm using namespaces, btw. Not sure if that makes any difference... @macat would you mind posting your updated config?

berryp picture berryp on 15 Jan 2018

Simply removed the image attr and all docker logs now fly to datadog. Realized it is possible to send all logs after inspecting this method: https://github.com/DataDog/datadog-agent/blob/master/pkg/logs/input/container/scanner.go#L124

apiVersion: v1
data:
  datadog.api.key: ooo
  datadog.yaml: |
    log_enabled: true
    listeners:
    - name: docker
    config_providers:
    - name: kubelet
      polling: true
  docker_daemon.yaml: |
    init_config:
      docker_root: /host
    logs:
    - type: docker
      service: docker
      source: docker
    - type: file
      path: /host/var/log/syslog
      service: syslog
      source: syslog
macat picture macat on 15 Jan 2018

Indeed, removing the image works - But as one does not have control over the logs that are collected via a filter, it can result in a lot of undesired logs.
We are working on improving this for a future release though :).

Thanks again for contributing !
.C

CharlyF picture CharlyF on 15 Jan 2018

I'm facing the same issue on beta9, except I'm not getting logs even with the same config as from 2 comments above, i.e. lots of the following in the agent logs:

datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://338ee23f86516d2740c17fd5a920655316f9ed1597d00b46bab600033e80573f not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://6c85647890f782587a4066cad6ad8a38e48a1ff448cced139d43d5670d5cc279 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://6c85647890f782587a4066cad6ad8a38e48a1ff448cced139d43d5670d5cc279 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://e85550c58ba96f0c73df8a5fe0e5220a320da601a6626e2a97ce5a36362f606c not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://e85550c58ba96f0c73df8a5fe0e5220a320da601a6626e2a97ce5a36362f606c not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://e447687e5547fb6e5a003c2baf67566df809516c90b5f34d03b6c67536f44d60 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://e447687e5547fb6e5a003c2baf67566df809516c90b5f34d03b6c67536f44d60 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://ae191ca31af5253190bd7e531d1dd221a8c3190d0036dd67d95be3d0a06a106d not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://8c4afa89049125610a9b8d0459259c5495e79a6d7d52f03a7a9748fc72578835 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://012eee5cf8a6880b3ddbdc931192ed1fdfa58a7f1e5d517a40ade75c0ed6f418 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://4f4d0fd57b579145253bb04dfc955c1cebabf7924af20f38ec8114e4dee8214e not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://2cbf1bea1ffea441c6787800bc23b15597a868dcacd7a4671fc08fb77df9b77a not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://3a79db3bf02f5a428bcf79624965b26dd4ca042187b8d55c18084ac226c34e78 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://f727753e7b87975e37783994b0a2342d28a192c93d26b572bc4be40cb56ec81f not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://f727753e7b87975e37783994b0a2342d28a192c93d26b572bc4be40cb56ec81f not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://8b1a01e0dea30d6197ed9edca8dfbe49d5ba818c19d099541f2cdd9c56fe9013 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://4d08f6a6b058d0475a74b123c619596b892b69fceefe867dd7c51c1f46fee326 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://4d08f6a6b058d0475a74b123c619596b892b69fceefe867dd7c51c1f46fee326 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://98d55025d794ef4fe1ae42854bb2ad21934d83c89144661b29aa0dade14d3a29 not found in podlist
datadog-agent-6wnqb datadog-agent 2018-02-12 13:21:27 UTC | WARN | (tagger.go:247 in Tag) | error collecting from kubelet: container docker://4b02e64f81611660cb49d0ac403fa439051338c3777bd33001a06c5203d95f69 not found in podlist

Pod metrics/tagging seem to work fine though.

Edit: disregard everything, I wasn't paying attention close enough, and I was mounting the checks configs in /opt/datadog-agent/conf.d instead of just /conf.d.

When I mount them in the correct location, I get some different errors though:

datadog-agent-kjlsf datadog-agent 2018-02-12 13:52:02 UTC | ERROR | (integration_config.go:105 in buildLogsSources) | Invalid file path: ..2982_12_02_13_52_00.704149069/datadog.yaml
datadog-agent-kjlsf datadog-agent 2018-02-12 13:52:02 UTC | ERROR | (integration_config.go:105 in buildLogsSources) | Invalid file path: ..2982_12_02_13_52_00.704149069/docker_daemon.yaml

They're under these weird paths because that's how k8s mounts configmaps as files. Despite these errors it seems that logs-agent does attempt to read container logs though:

datadog-agent-kjlsf datadog-agent 2018-02-12 13:52:23 UTC | ERROR | (scanner.go:118 in listContainers) | Can't tail containers, Error response from daemon: client is newer than server (client API version: 1.25, server API version: 1.24)
datadog-agent-kjlsf datadog-agent 2018-02-12 13:52:23 UTC | ERROR | (scanner.go:119 in listContainers) | Is datadog-agent part of docker user group?

It seems like our Docker (1.12.6, CoreOS) is too old? @macat if you don't mind me asking, what Docker version are you running? In your initial comment you said 0.13, but that seems wrong, I'm guessing it's 1.13.x?

andor44 picture andor44 on 12 Feb 2018
👍1

Hi everyone,

A little message just to signal you that my DataDog agent is reporting the same kind of errors. I have an opened ticket that talk about that here: https://help.datadoghq.com/hc/en-us/requests/130699 (with logs from the agent)

I have a k8s 1.9.2 cluster. My Datadog agent is the v6 one and it is deployed thanks to the stable chart you provide.

I deploy my agent thanks to this command:

helm install --name ${DD_AGENT_RELEASE_NAME} -f deploy/resources/datadog/values.yml stable/datadog

with the following values.yml:

# Copied from here: https://github.com/kubernetes/charts/blob/master/stable/datadog/values.yaml

# Default values for datadog.
image:
  # This chart is compatible with different images, please choose one
  # repository: datadog/docker-dd-agent  # Agent5
  repository: datadog/agent          # Agent6 (beta)
  # repository: datadog/dogstatsd      # Standalone DogStatsD6 (beta)
  tag: latest-jmx
  pullPolicy: Always

# NB! Normally you need to keep Datadog DaemonSet enabled!
# The exceptional case could be a situation when you need to run
# single DataDog pod per every namespace, but you do not need to
# re-create a DaemonSet for every non-default namespace install.
# Note, that StatsD and DogStatsD work over UDP, so you may not
# get guaranteed delivery of the metrics in Datadog-per-namespace setup!
daemonset:
  enabled: true
  ## Bind ports on the hostNetwork. Useful for CNI networking where hostPort might
  ## not be supported. The ports will need to be available on all hosts. Can be
  ## used for custom metrics instead of a service endpoint.
  ## WARNING: Make sure that hosts using this are properly firewalled otherwise
  ## metrics and traces will be accepted from any host able to connect to this host.
  # useHostNetwork: true

  ## Sets the hostPort to the same value of the container port. Can be used as
  ## for sending custom metrics. The ports will need to be available on all
  ## hosts.
  ## WARNING: Make sure that hosts using this are properly firewalled otherwise
  ## metrics and traces will be accepted from any host able to connect to this host.
  useHostPort: true

  ## Annotations to add to the DaemonSet's Pods
  # podAnnotations:
  #   scheduler.alpha.kubernetes.io/tolerations: '[{"key": "example", "value": "foo"}]'

  ## Allow the DaemonSet to schedule on tainted nodes (requires Kubernetes >= 1.6)
  # tolerations: []

  ## Allow the DaemonSet to perform a rolling update on helm update
  ## ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
  # updateStrategy: RollingUpdate

# Apart from DaemonSet, deploy Datadog agent pods and related service for
# applications that want to send custom metrics. Provides DogStasD service.
#
# HINT: If you want to use datadog.collectEvents, keep deployment.replicas set to 1.
deployment:
  enabled: false
  replicas: 1

## deploy the kube-state-metrics deployment
## ref: https://github.com/kubernetes/charts/tree/master/stable/kube-state-metrics
##
kubeStateMetrics:
  enabled: true

datadog:
  ## You'll need to set this to your Datadog API key before the agent will run.
  ## ref: https://app.datadoghq.com/account/settings#agent/kubernetes
  ##
  apiKey: 'TOCHANGE'

  ## dd-agent container name
  ##
  name: dd-agent

  ## Set logging verbosity.
  ## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
  ##
  logLevel: WARNING

  ## Un-comment this to make each node accept non-local statsd traffic.
  ## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
  ##
  # nonLocalTraffic: true

  ## Set host tags.
  ## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
  ##
  # tags:

  ## Enables event collection from the kubernetes API
  ## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
  ##
  collectEvents: true

  ## Un-comment this to enable APM and tracing, on ports 7777 and 8126
  ## ref: https://github.com/DataDog/docker-dd-agent#tracing-from-the-host
  ##
  apmEnabled: true

  ## The dd-agent supports many environment variables
  ## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
  ##
  env:
    # https://docs.datadoghq.com/guides/process/
    - name: DD_PROCESS_AGENT_ENABLED
      value: "true"
    # https://app.datadoghq.com/logs/onboarding/container
    - name: DD_LOGS_ENABLED
      value: "true"
    # https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#kubernetes-integration
    - name: DD_LEADER_ELECTION
      value: "true"
    # https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#event-collection
    - name: DD_COLLECT_KUBERNETES_EVENTS
      value: "true"

  ## The dd-agent supports detailed process and container monitoring and
  ## requires control over the volume and volumeMounts for the daemonset
  ## or deployment.
  ## ref: https://docs.datadoghq.com/guides/process/
  ##
  volumes:
    - hostPath:
        path: /etc/passwd
      name: passwd
  volumeMounts:
    - name: passwd
      mountPath: /etc/passwd
      readOnly: true

  ## Enable leader election mechanism for event collection
  ##
  leaderElection: true

  ## Set the lease time for leader election
  ##
  # leaderLeaseDuration: 600

  ## Provide additonal service definitions
  ## Each key will become a file in /conf.d/auto_conf
  ## ref: https://github.com/DataDog/docker-dd-agent#configuration-files
  ##
  # autoconf:
  #   kubernetes_state.yaml: |-
  #     docker_images:
  #       - kube-state-metrics
  #     init_config:
  #     instances:
  #       - kube_state_url: http://%%host%%:%%port%%/metrics

  ## Provide additonal service definitions
  ## Each key will become a file in /conf.d
  ## ref: https://github.com/DataDog/docker-dd-agent#configuration-files
  ##
  confd:
  #   redisdb.yaml: |-
  #     init_config:
  #     instances:
  #       - host: "name"
  #         port: "6379"
  # https://app.datadoghq.com/logs/onboarding/container
  # https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#configuration-file-example
    logs.yaml: |-
      init_config:
      instances:
        [{}]
      logs:
        - type: docker
          service: myapp
          source: myapp-logs

  ## Provide additonal service checks
  ## Each key will become a file in /checks.d
  ## ref: https://github.com/DataDog/docker-dd-agent#configuration-files
  ##
  # checksd:
  #   service.py: |-

  ## dd-agent resource requests and limits
  ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources:
    requests:
      cpu: 100m
      memory: 64Mi
    limits:
      cpu: 256m
      memory: 256Mi

rbac:
  ## If true, create & use RBAC resources
  create: false

  ## Ignored if rbac.create is true
  serviceAccountName: default

tolerations: []

kube-state-metrics:
  rbac:
    create: false

    ## Ignored if rbac.create is true
    serviceAccountName: default
guizmaii picture guizmaii on 21 Feb 2018
👍2

I got the same error.

[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:14 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://3ab71114c06ebd2ed0b641f604ef7e13034399c86de2bf38d323139e3db86735 not found in podList
  • Kubernetes v1.8.7-gke
  • Datadog Agent v6.0.2
  • Install agent using Helm (stable/datadog)


values.yaml

image:
  repository: datadog/agent               # Agent6
  tag: 6.0.2  # Use 6.0.0-jmx to enable jmx fetch collection
  pullPolicy: IfNotPresent

daemonset:
  enabled: true
  updateStrategy: RollingUpdate

deployment:
  enabled: false
  replicas: 1

kubeStateMetrics:
  enabled: true

datadog:
  apiKey: xxxxx
  name: dd-agent
  logLevel: WARNING
  collectEvents: false
  env:
    - name: DD_LOGS_ENABLED
      value: "true"
  leaderLeaseDuration: 600s
  confd:
    kubernetes.yaml: |-
      init_config:
      instances:
        - port: 4194
          collect_events: True
          namespace_name_regexp: .*
    docker_daemon.yaml: |-
      logs:
        - type: docker
          service: docker
          source: docker
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 256m
      memory: 512Mi

rbac:
  create: true
  serviceAccountName: default

tolerations: []

kube-state-metrics:
  rbac:
    create: true
    serviceAccountName: default

dtan4 picture dtan4 on 8 Mar 2018
👍11

Got this on kube 1.7.10 in Azure.

sylr picture sylr on 15 Mar 2018

Is there any workaround to disable this feature while it is being fixed? Maybe using one of these env vars? https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#tagging
This is currently filling up my log server.

mausch picture mausch on 15 Mar 2018

Hi everyone, we have a fix https://github.com/DataDog/datadog-agent/pull/1345 that should resolve the logging issues and will be included in the next 6.1 release.

We're aware of an issue on kubernetes side where static pods are not correctly updated in the kubelet podlist (#1447) but we'll keep an eye on this to see if there could be other issues.

mfpierre picture mfpierre on 15 Mar 2018
👍4

@mfpierre when will the 6.1.1 release be available as a Helm chart?
Why doesn't every new release result in a new version of the Helm chart?
Related to #1447

torbjornvatn picture torbjornvatn on 3 Apr 2018
👍4

@torbjornvatn I just set image.tag: "6.1.1" as a value when installing/upgrading the helm chart. Personally I do this with pretty much all helm charts... charts and images are typically versioned separately.

mausch picture mausch on 8 Apr 2018

Hello there,

The release 6.1.1 is available for a while now (as a Helm Chart as well) and no new reports have been filled.
Therefore I'll go ahead and close this issue.

Feel free to reach out to [email protected] if you have any questions or experience any problem.

Thanks again for the report and helping us improve the Datadog Agent.

NBParis picture NBParis on 30 Jan 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ralmsdeveloper picture ralmsdeveloper  路  4Comments
SleepyBrett picture SleepyBrett  路  4Comments
dignajar picture dignajar  路  3Comments
tunguyen9889 picture tunguyen9889  路  4Comments
amlwwalker picture amlwwalker  路  5Comments