Fluentd-kubernetes-daemonset: Support containerd log format

[byrnedo]

Hi, I'm running k3s using containerd instead of docker.
The log format is different to docker's.
AFAIK it would just involve changing the @type json to a regex for the container logs, see https://github.com/rancher/k3s/issues/356#issuecomment-485080611
Would anyone be up for doing this? With maybe some kind of env var to switch on the container d support, eg CONTAINER_RUNTIME=docker as default, with containerd as an alternative

[arthurdarcet]

you can add the env variable FLUENT_CONTAINER_TAIL_PARSER_TYPE with value /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/ - it'll make this daemonset ok with the containerd logs

[strigazi]

The above regex worked for me! thanks!

Could we make it work for both containerd and docker without setting the type?

[faust64]

Hi,

It may look like it works, though having dealt with OpenShift a lot lately: you're missing something. Eventually, you'll see log messages being split into several records.

I've had to patch the /fluentd/etc/kubernetes.conf file.

We could indeed set FLUENT_CONTAINER_TAIL_PARSER_TYPE to /^(?<time>.+) (?<stream>stdout|stderr) (?<logtag>[FP]) (?<log>.+)$/.

However we also need to add the following:

    <filter kubernetes.**>
      @type concat
      key log
      partial_key logtag
      partial_value P
      separator ""
    </filter>

Note that I'm setting a logtag field, from the F and P values, that @arthurdarcet drops with a [^ ]*.
We actually need those re-constructing multi-line message (P means you have a partial log, while F notes the last part of a message).

[vfolk30]

I have enabled rancher logging with fluentd for containerD , but still getting issue.Below our the enn variable i have pasted in daemon set
https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/logging/

env:
- name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
value: /var/log/containers/fluentd*
- name: FLUENTD_SYSTEMD_CONF
value: disable
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
value: /^(?.+) (?stdout|stderr) [^ ]* (?.*)$/

output:

2020-08-11 16:07:29 +0000 [warn]: #0 pattern not matched: "2020-08-11T18:07:28.606198265+02:00 stdout F 2020-08-11 16:07:28 +0000 [warn]: #0 pattern not matched: \"2020-08-11T18:07:27.620512318+02:00 stdout F 2020-08-11 16:07:27 +0000 [warn]: #0 pattern not matched: \\\"2020-08-11T18:07:26.541424158+02:00 stdout F 2020-08-11 16:07:26 +0000 [warn]: #0 pattern not matched: \\\\\\\"2020-08-11T18:07:25.531461018+02:00 stdout F 2020-08-11 16:07:25 +0000 [warn]: #0 pattern not matched: \\\\\\\\\\\\\\\"2020-08-11T18:07:24.528268248+02:00 stdout F 2020-08-11 16:07:24 +0000 [warn]: #0 pattern not matched: \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"2020-08-11T18:07:23.524149263+02:00 stdout F 2020-08-11 16:07:23 +0000 [warn]: #0 pattern not matched: \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"2020-08-11T18:07:23.187045754+02:00 stdout F 2020-08-11 16:07:23.186 [INFO][57] int_dataplane.go 976: Finished applying updates to dataplane. msecToApply=1.434144\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\"\""

[arren-ru]

@arthurdarcet @faust64 How the regex string supposed to work in FLUENT_CONTAINER_TAIL_PARSER_TYPE if that variable translated to @type value in the configuration of parser?
kubernetes.conf inside container contains:

…
<source>
  @type tail
  …
  <parse>
    @type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>
…

Allowed @types https://docs.fluentd.org/configuration/parse-section#type

[DarkBlaez]

Maybe this should just be addressed with a flag. The issue has been present for such a long time and impacts other vendors that choose to spin value-added products around this. Word to the wise, docker is not the only front-end to containers and container evolution continues. Addressing this now external to the sloppy work arounds with regex or manipulation would be a good thing. Better to get in front of the issue than lag behind.

DB

[repeatedly]

We can put additional plugin into plugins directory, e.g. https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/v1.11/debian-elasticsearch7/plugins
So if anyone provides containerd log format parser, we can configure it via FLUENT_CONTAINER_TAIL_PARSER_TYPE.

[DarkBlaez]

That would work as I am willing to write a custom parser for this to contribute and save others the same issues. Or re-phrased, that is perhaps the best option, an additional plugin for this specific use case

[faust64]

@arren-ru : you are right, my mistake. FLUENTD_CONTAINER_TAIL_PARSER_TYPE should be set to regexp, and then you'ld set anexpression, with your actual regexp.

Either way, that's not something you can currently configure only using environment variables.
You're looking for something like this: https://github.com/faust64/kube-magic/blob/master/custom/roles/logging/templates/fluentd.j2#L31-L51

[arren-ru]

@faust64 I solved this by overriding kubernetes.conf with configmap mounted in place of original configuration with modified content, this gives basic working solution

      <source>
        @type tail
        @id in_tail_container_logs
        path /var/log/containers/*.log
        pos_file /var/log/fluentd-containers.log.pos
        tag "#{ENV['FLUENT_CONTAINER_TAIL_TAG'] || 'kubernetes.*'}"
        exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
        read_from_head true
        <parse>
          @type regexp
          expression /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<flags>[^ ]+) (?<message>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </parse>
      </source>

      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        kubernetes_url "#{'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
      </filter>

[i300543]

The kind of solutions presented here will cause json log to be parsed as a string, and no fields defined in json itself will be recongnized as Elasticsearch fields correct ?

[arren-ru]

The kind of solutions presented here will cause json log to be parsed as a string, and no fields defined in json itself will be recongnized as Elasticsearch fields correct ?

Not sure understood you, but CRI logs are represented as a string line, this is not a docker logs, so if you want to parse json further you may want to add pipelined parser or filter

[micktg]

I got an issue, that my logfile was filled with backslashes. I am using containerd instead of docker. I solved it by putting in the following configuration:

- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

[m-usmanayub]

I got an issue, that my logfile was filled with backslashes. I am using containerd instead of docker. I solved it by putting in the following configuration:

- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

Did not work for me on 1.20.1 hosted on VMs. Still the same backslashes full of error

[vipinjn24]

I am using containerd as the CRI for kubernetes and used FLUENT_CONTAINER_TAIL_PARSER_TYPE env var.
But it seems that now the logs are somewhat readable but the time format is incorrect so the error is shown for that.

Any solution to this problem or can we change the time format by any env var?

[vipinjn24]

Ok, got it on how to fix this one.

First we know that we need to change the logging format as containerd do not use json format and is a regular text format.
so we add the below environment variable to the daemonset.

- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: /^(?<time>.+) (?<stream>stdout|stderr) (?<logtag>[FP]) (?<log>.*)$/

Now when we do this, it still shows error with the time format.
To solve this we will extract the kubernetes.conf file from a running fluentd container and copy the contents to a config map and mount that value to the kubernetes.conf location i.e. /fluentd/etc/kubernetes.conf.

  volumeMounts:
  - name: fluentd-config
    mountPath: /fluentd/etc/kubernetes.conf
    subPath: kubernetes.conf
volumes:
- name: fluentd-config
  configMap:
    name: fluentd-config
    items:
    - key: kubernetes.conf
      path: kubernetes.conf

So to fix the error, we update the following value inside the source.

<source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag "#{ENV['FLUENT_CONTAINER_TAIL_TAG'] || 'kubernetes.*'}"
      exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
      read_from_head true
      <parse>
        @type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
        time_format %Y-%m-%dT%H:%M:%S.%N%:z
      </parse>
</source>

time_format %Y-%m-%dT%H:%M:%S.%NZ
to
time_format %Y-%m-%dT%H:%M:%S.%N%:z

Now deploy the daemonset, it will work.

[cosmo0920]

I'd published the new parser included images: https://github.com/fluent/fluentd-kubernetes-daemonset/pull/521, https://github.com/fluent/fluentd-kubernetes-daemonset/commit/2736b688e405c9da4675eb4e23ade916d009ec53

With FLUENT_CONTAINER_TAIL_PARSER_TYPE, we can specify cri type parser for parsing CRI format Logs.

ref: https://github.com/fluent/fluentd-kubernetes-daemonset#use-cri-parser-for-containerdcri-o-logs

[hari819]

we are facing this issue with slashes "\\" , we use v1.12-debian-elasticsearch7-1 version of daemonset , we are currently testing the workarounds mentioned in this issue ,

would like to know if there would be a newer version of the daemonset after fixing the issue or do we need to use the workarounds permanently,

thanks,