Fluentd: Problem in nested json parsing for unwanted back slash when getting logs from docker

Created on 12 Jan 2018 · 7Comments · Source: fluent/fluentd

TD Agent version: td-agent 0.14.25

We have some nodejs and python applications that are running as Docker container. When using fluentd log driver, our json logs get "stringified",so structure is lost. I searched to find a solution using a fluentd plugins but can't. It doesn't seem to be fixable with elasticsearch mapping either.

Actual Application Log:

{"cid": "1c2675fc-82dd-4a7d-ae7c-7082d48dabbe", "source": "USSD", "timestamp": "2017-12-21 10:42:23.012765", "action": "USSD_BROWSING", "msisdn": null, "browsing": {"UMBSessionId": null, "response": {"code": 200}, "request": {"url": "http://localhost/", "path": "/", "queryString": ""}}}

When we are using json log driver from Docker, the log is converted as below (When fluentd receives it, it looks like below)

{"source":"stderr","log":"{\"cid\": \"6aca9d54-a4e0-47d9-af24-f9d29b39d262\", \"source\": \"USSD\", \"timestamp\": \"2018-01-07 12:45:45.557018\", \"action\": \"USSD_BROWSING\", \"request_type\": \"browsing\", \"msisdn\": \"8801711082722\", \"UMBSessionId\": \"201801071845450422752340\", \"response\": {\"code\": 302, \"location\": \"/retailer/choice.html\"}, \"request\": {\"url\": \"http://10.172.66.2/ussd/index.php?SessionID=201801071845450422752340&msisdn=8801711082722&service=454\", \"path\": \"/ussd/index.php\", \"queryString\": \"SessionID=201801071845450422752340&msisdn=8801711082722&service=454\"}}","container_id":"51ee5f9a9536c26389dbca75df3fe749486bdf4a504e23462ace58f8908708e4","container_name":"/ussd"}

N.B: We are using tail inside directive to read the logs from /var/lib/docker/containers/*

Here I am mentioning how do we want to process our logs.

Tail container log

Grep with specific message and proceed with those logs only

Remove unwanted back slash from nested json

Flatten Json message from nested json

Match output to stdout or elasticsearch

Sudo config file is given below for your convenience:

<source>
  @type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S 
  tag all.docker.*
  format none
</source>

<filter all.docker.*>
  @type grep
  <regexp>
    key source
    pattern (USSD|WEB)
  </regexp>
  tag staging.ussd.grepped
</filter>

<filter staging.ussd.grepped>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
  tag staging.ussd.nested_json
</filter>

<filter staging.ussd.nested_json>
  @type flatten_hash
  separator .
</filter>

<match test.**>
  @type stdout
</match>

*I have given our input log sample. Would be a great If you can help me by instructing me how the fluent config file should be. *

Application Log:
{"cid": "1c2675fc-82dd-4a7d-ae7c-7082d48dabbe", "source": "USSD", "timestamp": "2017-12-21 10:42:23.012765", "action": "USSD_BROWSING", "msisdn": null, "browsing": {"UMBSessionId": null, "response": {"code": 200}, "request": {"url": "http://localhost/", "path": "/", "queryString": ""}}}

Final Expected Log:
{"cid": "1c2675fc-82dd-4a7d-ae7c-7082d48dabbe", "source": "USSD", "timestamp": "2017-12-21 10:42:23.012765", "action": "USSD_BROWSING", "msisdn": null, "browsing.UMBSessionId": null, "browsing.response.code": 200, "browsing.request.url": "http://localhost/", "browsing.request.path": "/", "browsing.request.queryString": ""}

@repeatedly @sugimura @sohel2020 @tagomoris

Source