Fluentd-kubernetes-daemonset: json log not getting parsed to the output record fields

[ericln]

I have tried using both fluent/fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch and fluent/fluentd-kubernetes-daemonset:elasticsearch images in my k8s cluster with default fluent.conf and kubernetes.conf. However, the json format log fields doesn't get parsed into output record fields when sending to elasticsearch.

am I missing some configurations ?

just a note, i have another older cluster running with fluent/fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch image that was pull couple month prior is parsing the json log fine.

thanks

[repeatedly]

You need to put filter parser in your pipeline: https://docs.fluentd.org/v1.0/articles/filter_parser
See https://github.com/fluent/fluentd/issues/2073 for more details.

We will add README about this point.

[lentzi90]

To help others coming here looking for solutions: Note that there are two different versions for fluentd and that the configuration syntax has changed for the filter plugin.
Compare v0.12 and v1.0.

This comment worked for me with v1.4.2-debian-elasticsearch-1.0, but you may be interested in this if you run some other version.

[ericln]

@lentzi90 thanks, that worked for me too by adding the following

<filter kubernetes.var.log.containers.**>
  @type parser
  format json
  key_name log
  reserve_time true
  reserve_data true
  emit_invalid_record_to_error false
</filter>