Beats: Autodiscover for docker is not working

[vbohata]

I am testing new autodiscover for docker logs in filebeat 6.1.1 but it is not working. My configuration:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.name: apache
          config:
            - type: docker
              container.ids:
                - "${data.docker.container.id}"

output.console:
  pretty: true

I assume it should automatically detect running container with name apache and start displaying its docker json logs. But nothing happens.

I can collect logs statically using this:

- type: docker
  containers.ids:
    - "*"

But autodiscover is not working. Is it a bug or am I doing something wrong?

[exekias]

Hi @vbohata,

docker.container.name contains the whole Docker image path, including version, so it may look something like: apache:$VERSION. You can use other conditions for autodiscovery, try contains instead of equals

Also, you can debug the content of Autodiscover events by running Filebeat with -d autodiscover flag

[vbohata]

Also does not work with contains.

[exekias]

Could you please share the output of running Filebeat with this flag: -d autodiscover,docker?

[vbohata]

No stdout/stderr output. In logs/filebeat:

2017-10-16T23:24:27+02:00 INFO Home path: [/root/filebeat-6.1.1-linux-x86_64] Config path: [/root/filebeat-6.1.1-linux-x86_64] Data path: [/root/filebeat-6.1.1-linux-x86_64/data] Logs path: [/root/filebeat-6.1.1-linux-x86_64/logs]
2017-10-16T23:24:27+02:00 INFO Beat UUID: 7d746998-e84b-4aa1-b021-19a1f2bdd414
2017-10-16T23:24:27+02:00 INFO Setup Beat: filebeat; Version: 6.1.1
2017-10-16T23:24:27+02:00 INFO Metrics logging every 30s
2017-10-16T23:24:27+02:00 INFO Beat name: rhel4test
2017-10-16T23:24:27+02:00 INFO filebeat start running.
2017-10-16T23:24:27+02:00 INFO Registry file set to: /root/filebeat-6.1.1-linux-x86_64/data/registry
2017-10-16T23:24:27+02:00 INFO Loading registrar data from /root/filebeat-6.1.1-linux-x86_64/data/registry
2017-10-16T23:24:27+02:00 INFO States Loaded from registrar: 2
2017-10-16T23:24:27+02:00 WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2017-10-16T23:24:27+02:00 INFO Loading Prospectors: 1
2017-10-16T23:24:27+02:00 WARN EXPERIMENTAL: Docker prospector is enabled.
2017-10-16T23:24:27+02:00 INFO Starting prospector of type: docker; ID: 7362274609644250151 
2017-10-16T23:24:27+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-10-16T23:24:27+02:00 INFO Starting Registrar
2017-10-16T23:24:50+02:00 INFO Stopping filebeat
2017-10-16T23:24:50+02:00 INFO Stopping Crawler
2017-10-16T23:24:50+02:00 INFO Stopping 1 prospectors
2017-10-16T23:24:50+02:00 INFO Prospector ticker stopped
2017-10-16T23:24:50+02:00 INFO Stopping Prospector: 7362274609644250151
2017-10-16T23:24:50+02:00 INFO Crawler stopped
2017-10-16T23:24:50+02:00 INFO Stopping Registrar
2017-10-16T23:24:50+02:00 INFO Ending Registrar
2017-10-16T23:24:50+02:00 INFO Total non-zero values:  beat.info.uptime.ms=22822 beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=3185976 beat.memstats.memory_total=3185976 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.output.type=console libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=2 registrar.writes=1
2017-10-16T23:24:50+02:00 INFO Uptime: 22.822877732s
2017-10-16T23:24:50+02:00 INFO filebeat stopped.

[exekias]

Sorry, my fault, use -e -v -d autodiscover,docker

[vbohata]

Still same output, just going to stdout/stderr instead to file.

[exekias]

It's werid, the logs says nothing about autodiscover, but still, there is a prospector running, could you share your full filebeat.yml? Something doesn't add up here

[vbohata]

autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: apache
          config:
            - type: docker
              container.ids:
                - "*"

filebeat.prospectors:
- type: docker
  containers.ids:
    - "SOME_RANDOM_STR_BECAUSE_THERE_HAS_TO_BE_SOME_CONFIG"

output.console:
  pretty: true

logging.level: debug

[exekias]

Ok, I see the error now, try by replacing autodiscover with filebeat.autodiscover

[vbohata]

OK, that make the change. But still not working. It needs path but it is not a required parameter for docker prospector.
[autodiscover] Failed to create runner with config &{{<nil> } <nil> 0xc42044f400}: each prospector must have at least one path defined

[exekias]

I see there is another typo: try container.is -> containers.ids

[vbohata]

Thanks. Now it is working well.

This typo is also presented here https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

[exekias]

OMG, I'll open a PR to fix it, thanks for finding that!

[ianseyer]

Interestingly, I am having the same issue:

filebeat.autodiscover:
  providers:
    - type: docker
      container.ids:
        - "*"

Running on AWS ECS. /var/run/docker.sock is mounted.

running ./filebeat -d autodiscover,docker produces no output besides INFO Setting Elasticsearch and Kibana URLs. I have stopped and started several containers while this is runing and it produces nothing.

No logs are produced.

Note: I am running filebeat from within docker, hence the mounted sock. However, I have tried running it on the host directly to the same result.

[jakommo]

AFAIK you need to add a prospect for this as well.
Just tried with your config and after adding the following it worked for me.

filebeat.prospectors:
- type: docker
  containers.ids:
    - "*"

[exekias]

@ianseyer same typo is in place for your config, as @jakommo pointed out, it should be containers.ids instead of container.ids

[sgreszcz]

I can't get this working either, although metricbeat is grabbing docker metrics fine. I am getting system, audit, and other custom logs fine from filebeat, just not docker using this configuration (now that prospectors are depreciated):

#=========================== Filebeat autodiscover
==============================
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      containers.ids:
        - "*"
#=========================== Filebeat Inputs ==============================

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"