Beats: Logs with kubernetes autodiscover are not parsed

[jsoriano]

With the basic hints configuration for kubernetes autodiscover to use modules with pods, logs are not being parsed by the module pipeline, it has been reported with the filebeat nginx module, but it could be general.

Config:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true

Annotations:

      annotations:
        co.elastic.logs/module: nginx
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/fileset.stderr: error

Some observations:

• In affected environments logs are collected and pipelines are installed, but logs are not parsed. When tested to parse the logs with the simulate API, logs are correctly parsed, so it doesn't look like an issue on log format or the module pipelines.
• It may be related with the use of streams in docker input. If pipeline is set in the elasticsearch output to the nginx pipeline, then the logs are parsed.
• Docker autodiscovery with and without hints don't show this problem.

Reported also in discuss:

• https://discuss.elastic.co/t/nginx-logs-on-kubernetes-not-parsed/143446
• https://discuss.elastic.co/t/filebeat-deployment-in-kubernetes-docker/113759/27

[ITBlogger]

Yep, also having this issue. https://discuss.elastic.co/t/filebeat-hint-based-autodiscovery-not-working/144985

[exekias]

I just tested this and it's working for me, it may be a lack of documentation. Did you remove the existing filebeat.config.inputs configuration? Autodiscover should replace it, they won't work together

[ITBlogger]

@exekias Working for you with what configuration exactly?

It's definitely not working for me with logs going to logstash and settings as detailed in my thread.

[exekias]

Can you try again after removing these lines? https://gist.github.com/ITBlogger/f50632d643ec4cb241bdd41355b295ba#file-filebeat-configmaps-L19-L23

Also, check annotations, as they are attached to the Deployment, where they should be present in the Pod template. Full manifests would help here, as they are just fragments.

[ITBlogger]

I tried adding annotations to the pod template, but got errors from kubectl

[ITBlogger]

Removing those lines didn't make a difference, by the way

[ITBlogger]

OK, moving the annotations down to pod spec along with removing the config lines seems to have done the trick...thanks...that should definitely be changed in the documentation

[exekias]

I've opened https://github.com/elastic/beats/pull/8029, I'm open to input on more docs to update.

[ITBlogger]

https://github.com/elastic/beats/blob/f33982b31694f7003e1e002a9c01009009814f23/filebeat/docs/autodiscover-hints.asciidoc should be changed so that:

annotations:

becomes

spec:
  template:
    metadata:
      annotations:

in both examples

[elasticmachine]

Pinging @elastic/infrastructure

[exekias]

Closing as this turned out to be a miss configuration

[havlan]

@exekias is it not worth changing the docs to this? Took a while for me to figure this out when starting out with autodiscover. That the annotations should really be

spec:
  template:
    metadata:
      annotations:

Or at least clarify where filebeat looks for annotations (pods, deployments, replica sets?)