Beats: Filebeat can't read CRI-O logs.

[jareksm]

• Version: filebeat latest build from master branch, CRI-O latest build from master branch.
• Operating System: CoreOS
• Discuss Forum URL:
• Steps to Reproduce:

Filebeat doesn't collect logs generated by CRI-O since they are in /var/log/pods/<pod uid>/<container name>/<index starting with 0>.log. I've created this issue and looked at cri-o code and it has no options to change the path. It accepts it from kubelet and does as kube decides.

I've checked filebeat code, but the way it derives path is pretty complex unless someone can guide me and I can create a pull request. 1st thing I need is how to get a Pod UID (not ID). It looks like they are disabled by default, then I need to get a name of a container and where to use both of them to generate new path to the log.

[jsoriano]

We'd probably need to add explicit support for CRI-O, not sure if only changing the path would be enough.

[exekias]

I'm guessing autodiscover can be leveraged here, a config like this may work (I didn't test it):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      include_pod_uid: true
      templates:
        - condition.regexp:
            kubernetes.container.name: '.+'
          config:
            - type: docker
              containers:
                path: "/var/log/pods/${data.kubernetes.pod.id}/"
                ids:
                  - "${data.kubernetes.container.name}"

Also, add_kubernetes_metadata has powerful matching rules, I'm guessing that a combination of dissect + add_kubernetes_metadata processors could work here. Something like:

filebeat.inputs:
  - type: log
    paths:
      - /var/log/pods/*/*/*.log
    processors:
      - dissect:
        tokenizer: "/var/log/pods/%{kubernetes.pod.uid}/%{kubernestes.container.name}/%{index}.log"
        field: "source"
      - add_kubernetes_metadata:
        indexers:
          - pod_uid:
        matchers:
          - fields:
              lookup_fields: ["kubernetes.pod.uid"]

[jareksm]

Autodiscover worked like a charm, it discover the path and read the log files.

But now hints are broken (the used to work) and the correct filebeat modules are not assinged to logs through annotations. The below config used to work (I'm running latest version of filebeat from master).

Here's the final config I've used:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      include_pod_uid: true
      in_cluster: true
      hints.enabled: true
      include_annotations: '*'
      templates:
        - condition.regexp:
            kubernetes.container.name: '.+'
          config:
            - type: docker
              combine_partial: true
              cri.parse_flags: true
              cri.force: true
              containers:
                path: "/var/log/pods/${data.kubernetes.pod.uid}/"
                ids:
                  - "${data.kubernetes.container.name}"

processors:
  - add_cloud_metadata: ~
  - add_kubernetes_metadata:
      in_cluster: true

I've added extra logging lines to filebeat to see the reader configuration:

2018-11-09T05:33:12.154Z    INFO    readjson/docker_json.go:69  New all reader configuration options. Partial log entries: true, Force CRI logs: true, CRI flag parsing: true
2018-11-09T05:33:12.154Z    INFO    log/input.go:138    Configured paths: [/var/log/pods/91d51733-e33c-11e8-9f88-063461221a7e/prometheus-node-exporter/*.log]
2018-11-09T05:33:12.154Z    INFO    input/input.go:114  Starting input of type: docker; ID: 12251713129776511561 
2018-11-09T05:33:12.154Z    INFO    log/input.go:138    Configured paths: [/var/log/pods/f0f6865f-e3e0-11e8-9f88-063461221a7e/filebeat/*.log]
2018-11-09T05:33:12.154Z    INFO    input/input.go:114  Starting input of type: docker; ID: 6509717892180129881 
2018-11-09T05:33:12.155Z    INFO    readjson/docker_json.go:69  New all reader configuration options. Partial log entries: true, Force CRI logs: true, CRI flag parsing: true
2018-11-09T05:33:12.155Z    INFO    log/harvester.go:253    Harvester started for file: /var/log/pods/f0f6865f-e3e0-11e8-9f88-063461221a7e/filebeat/0.log
2018-11-09T05:33:12.156Z    INFO    log/input.go:138    Configured paths: [/var/log/pods/be325514-e322-11e8-9f88-063461221a7e/nginx-ingress-controller/*.log]
2018-11-09T05:33:12.156Z    INFO    input/input.go:114  Starting input of type: docker; ID: 9626636941847764779 
2018-11-09T05:33:12.156Z    INFO    kubernetes/watcher.go:198   kubernetes: Resource sync done
2018-11-09T05:33:12.156Z    INFO    kubernetes/watcher.go:242   kubernetes: Watching API for resource events
2018-11-09T05:33:12.157Z    INFO    log/input.go:138    Configured paths: [/var/log/pods/11fc4867-e3c4-11e8-9f88-063461221a7e/aws-node/*.log]
2018-11-09T05:33:12.157Z    INFO    input/input.go:114  Starting input of type: docker; ID: 13299610620757492503 
2018-11-09T05:33:12.158Z    INFO    log/input.go:138    Configured paths: [/var/log/pods/481bfb4d-e31f-11e8-9f88-063461221a7e/calico-node/*.log]
2018-11-09T05:33:12.158Z    INFO    input/input.go:114  Starting input of type: docker; ID: 5048832850434355580 
2018-11-09T05:33:12.158Z    INFO    readjson/docker_json.go:69  New all reader configuration options. Partial log entries: true, Force CRI logs: true, CRI flag parsing: true

[jareksm]

@exekias Is it possible to do similar configuration for hints autodiscovery?

[exekias]

This is closed by https://github.com/elastic/beats/pull/12193, that should be available with 7.2