Fluentd-kubernetes-daemonset: multiline message (like ruby exception) is divided into single line messages.

[zerthimon]

I'm seeing that a ruby exception comes in form of separate messages for each line. Does it work as intended ?

[repeatedly]

It is docker limitation. Docker generates json logs per line.
See also: https://docs.fluentd.org/v0.12/articles/docker-logging#additional-step-2:-concatenate-multiple-lines-log-messages

[zerthimon]

@repeatedly What do I have to do to get this concat snippet into the fluentd config template ? Is there an option to configure this via a configMap? Or is there another way around ?

[bdashrad]

@zerthimon since @include conf.d/* was added here you can add your own configmap and a volume to the daemonset yaml:

configmap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: YOUR-CONFIG-MAP
data:
  00_docker_concat.conf: |-
    <filter docker.**>
      @type concat
      key log
      stream_identity_key container_id
      multiline_start_regexp /^-e:2:in `\/'/
      multiline_end_regexp /^-e:4:in/
    </filter>

Excerpt of what you need to add to daemonset:

spec:
  template:
    spec:
      containers:
          volumeMounts:
            - name: kubesconf
              mountPath: /fluentd/etc/conf.d
      volumes:
      - name: kubesconf
        configMap:
          name: YOUR-CONFIG-MAP

Be aware that the example filter block is for fluentd v0.12 and would need to be changed for v1.x

[bdashrad]

Although for exception handling this might be better: https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions but it it not currently part of the plugins installed here.

[zerthimon]

@bdashrad Wow, this is awesome! Thanks a lot, I'll give it a try!

[bdashrad]

@zerthimon i had some time today to test this, and unfortunately both fluent-plugin-concat and fluent-plugin-detect-exceptions are not in the base docker image, but you could create a Dockerfile with the base image here as the source and add it yourself.

[bdashrad]

https://www.fluentd.org/guides/recipes/docker-logging has the 1.0 config for concat

[zerthimon]

@bdashrad Thanks for the research!

Are there plans to get these plugins into official image of fluent/fluentd-kubernetes-daemonset ?
In the meantime I'll make my own image, no problem with that.
Got a question though: Do I need to be using filter kubernetes.** instead of docker.** in a kubernetes setup ?

[zerthimon]

@bdashrad the filter block you posted earlier looks the same as the one at https://www.fluentd.org/guides/recipes/docker-logging am i missing something ?

[bdashrad]

@zerthimon I think that config would work. It also looks like @repeatedly added the concat plugin to the image in #268 so it should be easier to implement this now.

[tanel-andreson]

Are there plans to get these plugins into official image of fluent/fluentd-kubernetes-daemonset ?
In the meantime I'll make my own image, no problem with that.

Voting up.
I'll go with private images until then as well...

[NareshAnaparthi]

Hi, we are also facing issue of handling multiline logs (Java stack traces) . Could some one point to fluend image having fix/config in place

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days