Fluentd-kubernetes-daemonset: How to extract fields from kubernetes app logs - efk stack

[yishaihl]

I'm using FluentD (deployed as DaemonSet) to stream k8s app (containers) logs to elasticsearch. i want extract specific keys from the 'log' key string in elastic such as logKey: ... or statusCode:.. (inside the 'log' key below)
Please see the log:
https://i.stack.imgur.com/zMn3x.png

i saw that i should use FluentD plugin to parse the log before it sent to elastic, but unfortunately i didn't find a way to do it.
this is my current ConfigMap containers section :

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/app.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

Thanks a lot for your help.

[golgoth31]

Hi,
You only parse the docker log format, you should add a filter to parse the "@message" key:

@type parser
key_name log
....

[poonai]

I have a follow-up question in that

I want to use kubernetes_metatdata.
I'm using the following config.

<filter kubernetes.**>
  @type kubernetes_metadata
  @id filter_kube_metadata
  kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
  verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
  ca_file "#{ENV['KUBERNETES_CA_FILE']}"
</filter>

<filter kubernetes.var.log.containers.kubernetes-parallel-job**>
  @type parser
    <parse>
      @type json
      json_parser json
    </parse>
  replace_invalid_sequence true
  key_name log
</filter>

Not able to parse.

Hey @yishaihl If you found the solution already. Could you share your config? It will be very helpful

[rfleschenberg]

@balajijinnah That config seems to work for me, however I lose the standard Kubernetes fields. Any idea how to merge the application log with the Kubernetes fields?

[cosmo0920]

@balajijinnah That config seems to work for me, however I lose the standard Kubernetes fields. Any idea how to merge the application log with the Kubernetes fields?

Try to use reserve_data true:

<filter kubernetes.var.log.containers.kubernetes-parallel-job**>
  @type parser
    <parse>
      @type json
      json_parser json
    </parse>
  replace_invalid_sequence true
  key_name log
  reserve_data true
</filter>

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days