Fluentd: Json in 'log' field not parsed/exploded after migration from 0.12 to 1.2

Created on 15 Jun 2018  路  9Comments  路  Source: fluent/fluentd
  • fluentd or td-agent version: docker image fluent/fluentd-kubernetes-daemonset:v1.2-debian-elasticsearch
  • Environment information, e.g. OS: kubernetes 1.9
  • Your configuration: default
  • Your problem explanation. If you have an error logs, write it together.
    I'm pushing microservices logs into Elasticsearch using fluentd. Our microservices are making json logs to stdout where Kubernetes catch them and stream to json files making 'json in json'. In version 0.12 those logs were automatically 'exploded' so I had possibility to easily filter by fields in those logs in Elasticsearch. In Fluentd 1.12 I see only json string in Kubernetes 'log' field. What happend?
incomplete
Source
picture jakubbujny
👍6

Most helpful comment

Fix for that in new version can be achieved using
https://docs.fluentd.org/v1.0/articles/filter_parser
https://docs.fluentd.org/v1.0/articles/parser_json
as mentioned in https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/79
Probably it can look like:

<filter foo.bar>
  @type parser
  format json
  key_name log
</filter>
picture jakubbujny on 2 Jul 2018
👍2

All 9 comments

What happend?

I'm not sure. We need actual problematic log content for investigation.
Could you paste it?

In local testing, there is no result difference.

  • v1.2
2018-06-20 11:33:33 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/path/to/json_in_json.log"
    tag "tail.test"
    read_from_head true
    <parse>
      @type json
    </parse>
  </source>
  <match tail.**>
    @type stdout
  </match>
</ROOT>
2018-06-20 11:33:33 +0900 [info]: starting fluentd-1.2.2 pid=68076 ruby="2.5.1"
...snip...
2018-06-20 11:33:33.644152000 +0900 tail.test: {"k1":"v1","log":"{\"k2\":\"v2\"}"}
  • v0.12
2018-06-20 11:32:07 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path /path/to/json_in_json.log
    tag tail.test
    format json
    read_from_head true
  </source>
  <match tail.**>
    @type stdout
  </match>
</ROOT>
2018-06-20 11:32:07 +0900 [info]: following tail of /path/to/json_in_json.log
2018-06-20 11:32:07 +0900 tail.test: {"k1":"v1","log":"{\"k2\":\"v2\"}"}
repeatedly picture repeatedly on 20 Jun 2018

Thank you @repeatedly for answer!
Here is Log:
{"log":"{\"Timestamp\":\"2018-06-11T14:44:57.3591704+00:00\",\"Level\":\"Warning\",\"MessageTemplate\":\"The LINQ expression '{expression}' could not be translated and will be evaluated locally.\",\"RenderedMessage\":\"The LINQ expression '\\\"where ([x] == [dummy].something)\\\"' could not be translated and will be evaluated locally.\"", "stream":"stdout","time":"2018-06-11T14:44:57.364059249Z"}

In 0.12 I was able to easily filter in Elasticsearch using for example Level field. Now after migration I can't as in Elasticsearch I see only "log" field with whole json string which I provided

jakubbujny picture jakubbujny on 21 Jun 2018

We're having the exact same issue.

On 1.1.3 our JSON logs were being parsed correctly. Using 1.2.2, which fixes a stack trace problem we were having, no JSON log parsing is occuring.

Neither of the following configurations work for us on 1.2.x:

    <source>
      @id source_kubernetes_containers
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd/kubernetes-containers.pos
      read_from_head true
      tag kubernetes.*
      <parse>
        @type json
        json_parser yajl
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <source>
      @id source_kubernetes_containers
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd/kubernetes-containers.pos
      read_from_head true
      tag kubernetes.*
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>
jcardoso-bv picture jcardoso-bv on 21 Jun 2018
👍1

I confirm that downgrade to 1.1.3 fix the problem

jakubbujny picture jakubbujny on 22 Jun 2018

https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/pull/128

I found the root cause is fluent-plugin-kubernetes-metadata-filter removes automatic json merging feature since latest version.

In v1.1.3, image uses older version.

diff -u docker-image/v1.1/debian-elasticsearch/Dockerfile docker-image/v1.2/debian-elasticsearch/Dockerfile
--- docker-image/v1.1/debian-elasticsearch/Dockerfile   2018-05-14 22:41:50.000000000 +0900
+++ docker-image/v1.2/debian-elasticsearch/Dockerfile   2018-06-13 20:50:47.000000000 +0900
@@ -1,7 +1,7 @@
 # AUTOMATICALLY GENERATED
 # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/Dockerfile.erb

-FROM fluent/fluentd:v1.1.3-debian
+FROM fluent/fluentd:v1.2.2-debian

 MAINTAINER Eduardo Silva <[email protected]>
 USER root
diff -u docker-image/v1.1/debian-elasticsearch/Gemfile docker-image/v1.2/debian-elasticsearch/Gemfile
--- docker-image/v1.1/debian-elasticsearch/Gemfile  2018-05-03 00:30:14.000000000 +0900
+++ docker-image/v1.2/debian-elasticsearch/Gemfile  2018-06-13 20:50:49.000000000 +0900
@@ -3,7 +3,7 @@

 source "https://rubygems.org"

-gem "fluentd", "1.1.3"
+gem "fluentd", "1.2.2"
 gem "oj", "3.5.1"
 gem "fluent-plugin-elasticsearch"
 gem "fluent-plugin-kubernetes_metadata_filter"
diff -u docker-image/v1.1/debian-elasticsearch/Gemfile.lock docker-image/v1.2/debian-elasticsearch/Gemfile.lock
--- docker-image/v1.1/debian-elasticsearch/Gemfile.lock 2018-05-14 22:41:50.000000000 +0900
+++ docker-image/v1.2/debian-elasticsearch/Gemfile.lock 2018-06-13 20:50:58.000000000 +0900
@@ -13,30 +13,30 @@
     dig_rb (1.0.1)
     domain_name (0.5.20180417)
       unf (>= 0.0.5, < 1.0.0)
-    elasticsearch (6.0.2)
-      elasticsearch-api (= 6.0.2)
-      elasticsearch-transport (= 6.0.2)
-    elasticsearch-api (6.0.2)
+    elasticsearch (6.0.3)
+      elasticsearch-api (= 6.0.3)
+      elasticsearch-transport (= 6.0.3)
+    elasticsearch-api (6.0.3)
       multi_json
-    elasticsearch-transport (6.0.2)
+    elasticsearch-transport (6.0.3)
       faraday
       multi_json
     excon (0.62.0)
-    faraday (0.15.0)
+    faraday (0.15.2)
       multipart-post (>= 1.2, < 3)
-    ffi (1.9.23)
-    fluent-plugin-elasticsearch (2.9.2)
+    ffi (1.9.25)
+    fluent-plugin-elasticsearch (2.10.3)
       elasticsearch
       excon
       fluentd (>= 0.14.20)
-    fluent-plugin-kubernetes_metadata_filter (2.0.0)
+    fluent-plugin-kubernetes_metadata_filter (2.1.2)
       fluentd (>= 0.14.0, < 2)
       kubeclient (~> 1.1.4)
       lru_redux
-    fluent-plugin-systemd (1.0.0)
+    fluent-plugin-systemd (1.0.1)
       fluentd (>= 0.14.11, < 2)
-      systemd-journal (~> 1.3)
-    fluentd (1.1.3)
+      systemd-journal (~> 1.3.2)
+    fluentd (1.2.2)
       cool.io (>= 1.4.5, < 2.0.0)
       dig_rb (~> 1.0.0)
       http_parser.rb (>= 0.5.1, < 0.7.0)
@@ -83,7 +83,7 @@
       sigdump (~> 0.2.2)
     sigdump (0.2.4)
     strptime (0.2.3)
-    systemd-journal (1.3.1)
+    systemd-journal (1.3.2)
       ffi (~> 1.9)
     thread_safe (0.3.6)
     tzinfo (1.2.5)
@@ -103,7 +103,7 @@
   fluent-plugin-elasticsearch
   fluent-plugin-kubernetes_metadata_filter
   fluent-plugin-systemd
-  fluentd (= 1.1.3)
+  fluentd (= 1.2.2)
   oj (= 3.5.1)

 BUNDLED WITH
Common subdirectories: docker-image/v1.1/debian-elasticsearch/conf and docker-image/v1.2/debian-elasticsearch/conf
Common subdirectories: docker-image/v1.1/debian-elasticsearch/hooks and docker-image/v1.2/debian-elasticsearch/hooks
Common subdirectories: docker-image/v1.1/debian-elasticsearch/plugins and docker-image/v1.2/debian-elasticsearch/plugins
repeatedly picture repeatedly on 25 Jun 2018

Fix for that in new version can be achieved using
https://docs.fluentd.org/v1.0/articles/filter_parser
https://docs.fluentd.org/v1.0/articles/parser_json
as mentioned in https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/79
Probably it can look like:

<filter foo.bar>
  @type parser
  format json
  key_name log
</filter>
jakubbujny picture jakubbujny on 2 Jul 2018
👍2

@jcardoso-bv I'm running version 1.1.3 for now. Could not get json_parser plugin to work in 1.2.4. Can I see the complete conf? Are you using the filter parser with the source you have?

We're having the exact same issue.

On 1.1.3 our JSON logs were being parsed correctly. Using 1.2.2, which fixes a stack trace problem we were having, no JSON log parsing is occuring.

Neither of the following configurations work for us on 1.2.x:

    <source>
      @id source_kubernetes_containers
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd/kubernetes-containers.pos
      read_from_head true
      tag kubernetes.*
      <parse>
        @type json
        json_parser yajl
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

<source> @id source_kubernetes_containers @type tail path /var/log/containers/*.log pos_file /var/log/fluentd/kubernetes-containers.pos read_from_head true tag kubernetes.* <parse> @type json time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> </source>

bamb00 picture bamb00 on 7 Feb 2019

Im also having the same issue and tried couple of filter parser configs suggested here, but didn't work.. Any solution ?

Dinidu picture Dinidu on 27 Sep 2019

FIX : https://github.com/fluent/fluentd/issues/2073#issuecomment-409469227

Dinidu picture Dinidu on 9 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

agrozyme picture agrozyme  路  3Comments
JayTeli picture JayTeli  路  4Comments
t-osakada picture t-osakada  路  6Comments
aleks-mariusz picture aleks-mariusz  路  7Comments
khteh picture khteh  路  4Comments