Fluentd: Standard way for nested record support

[repeatedly]

This is for fixing #649, #902 and related issues.

Currently, many plugins can't handle nested record because there is no standard way.
Many users want to access nested record. For example, grep, rewrite-tag-filter, parser and more plugins.
Fluentd core should provide the way to handle these cases.

Here is a starting point. We need the feedback and suggestion!

Features

• Standard syntax for configuration
• Provide plugin helper for nested record

Configuration

What is the best syntax for configuration? There are several approaches.
Use grep filter plugin as an example:

jsonpath like synax

regexp1 key1 cool  #record['key1']`
regexp2 $.key1.key2[0] hot # record['key1']['key2'][0]

. separated key

regexp1 key1 cool  #record['key1']`
regexp2 key1.key2.0 hot # record['key1']['key2'][0]

How to handle record["key1"]["key2"] and record["key1.key2"] pattern?

support array

regexp1 key1 cool  #record['key1']`
regexp2 ['key1', 'key2', 0] hot # record['key1']['key2'][0]

record_transformer with enable_ruby syntax

regexp1 record["key1"] cool  #record['key1']`
regexp2 record["key1"]["key2"][0] hot # record['key1']['key2'][0]

key1 is same as record["key"] for backward compatibility

or good syntax?

Plugin helper

For internal implementation, We can use dig method to access nested values since ruby 2.4. For ruby 2.2 or earlier, we can use ruby_dig or backport_dig.

For plugin API, there are several approaches:

Return accessor object

accessor = create_record_accessor(conf_param) # conf_param is $.key1.key2[0]
value = accessor.call(record) # access record['key1']['key2'][0]

Provide access method

This is similar to inject helper.

create_record_accessor(:name, conf_param) # conf_param is $.key1.key2[0]
value = access_record(:name, record) # access record['key1']['key2'][0]

Plugin helper needs to maintain accessor list internally.

or good API?

[sonots]

+1 for jsonpath-like syntax, but we need to add bracket–notation to support a key containing . as

regexp1 key1 cool  # record['key1']`
regexp2 $.key1.key2[0] hot # record['key1']['key2'][0]
regexp3 $['key1']['this.is.key3'] warm # record['key1']['this.is.key3']

[chancez]

I'm a big fan of the enable_ruby approach because it generally can help solve this, but also offers a lot of other niceties I'd like to see in other plugins.

That said, I think it's probably a bit of a bigger scope and I'd still like to see this handled without the enable_ruby approach which can have potentially much higher overhead if all you want to do is access nested keys.

Overall I think jsonpath looks very reasonable, although feels somewhat inconsistent with how you access fields that aren't nested which I dislike, but I guess there's not many options unless you throw out backwards compatibility.

[repeatedly]

No more comment. So I wrote a patch for it: https://github.com/fluent/fluentd/pull/1637

[chancez]

Awesome, I like that it's got the two notations to handle cases with dots, and such.

[jhuenges]

@repeatedly how can this be used with remove_keys in the record_transformer filter?

[repeatedly]

@jhuenges Now way for now because this helper doesn't provide nested deletion.

[jhuenges]

Is there any way to delete something that is not on the top level of a json?
Example:

 {
  "request": {
    "headers": {
       ...
       "cookie": "abcdf",
       ...
    }
  }

And I want to remove the request.headers.cookie but not all other elements of request.headers.

[dannyk81]

@jhuenges Were you able to find a solution for this?

@repeatedly I have a similar issue, we have this structure:

{
        "timestamp": "1513650893625",
        "level": "FATAL",
        "http": {
            "user_agent": "Apache-HttpClient/4.3.3 (java 1.5)",
            "method": "POST",
            "env": {
                "REQUEST_START_TIME": 1513650858,
                "HTTP_ACCEPT_ENCODING": "gzip,deflate",
                "HTTP_CONNECTION": "close",
                "HTTP_CONTENT_LENGTH": "231",
                "HTTP_CONTENT_TYPE": "application/json; charset=UTF-8"
           }
       }
}

From above, I would like remove from the record ["http"].["env"] entirely.

so the result shoule be:

{
        "timestamp": "1513650893625",
        "level": "FATAL",
        "http": {
            "user_agent": "Apache-HttpClient/4.3.3 (java 1.5)",
            "method": "POST",
       }
}

[dannyk81]

Ok, I figured it out :)

Seems it was failing because not all records have the http key, so I used the trick mentioned by @repeatedly to check if it exists and delete the nested http.env key if true.

<filter udp.php>
  @type record_modifier
  <record>
    _dummy_ ${if record.has_key?('http'); record['http'].delete('env') ; end; nil}
  </record>
  remove_keys _dummy_
</filter>

[repeatedly]

Nested delete support: https://github.com/fluent/fluentd/pull/1800

[vikranth06]

can we update nested fields of a json??