Graylog2-server: How to use select_jsonpath function to generate fields

[sayume]

Expected Behavior

In pipeline we should select json path from json parse result and turn its children into fields.

Current Behavior

I use following code to get json from string field

rule "parse json"
when
  true
then
  let json_tree = parse_json(to_string($message.message));
  let json_result = select_jsonpath(json_tree, {json_body:"$..*"});
  set_fields(json_result);
end

I can use this pipeline to generate a field named "json_body", but what I want is to generate fields with all of the children of the root node. For example, for json "{"p1":"v1", "p2":"v2"}, what I want is to generate fields with name p1 and p2, not the field I named with "json_body". I am not sure this function is supported with the current functions.

Possible Solution

A new function to generate fields with json path children, like input json extractors.

Your Environment

• Graylog Version: 2.2.1
• Elasticsearch Version: 2.3
• MongoDB Version: 3.3
• Operating System: ubuntu14.04
• Browser version: Chrome

[kroepke]

Yes, this is currently not supported. We'll consider it for a future release.

[ralberto]

+1

[Bragolgirith]

This seems like a quite the oversight: as it stands it is currently not possible to parse messages like this:

2017-06-26T14:58:00,455Z {"payload":{"id":"1","text":"Hi!"}}
2017-06-26T14:58:00,455Z {"person":{"name":"John","age":5}}

(messages where the JSON object can have a varying structure)

We can't use a Pipeline, because parse_json() and select_jsonpath() only work for rigid, pre-defined JSON objects.
And we can't directly use the JSON Extractor, because only the part of the log message after the timestamp is valid JSON.

[joschi]

@Bragolgirith You can extract the JSON part of your message with a regular expression or a Grok pattern and then use the result in parse_json().

[Bragolgirith]

@joschi The issue is that by using a combination of parse_json(), select_jsonpath() and set_fields() we can only extract fields whose name we know in advance (correct me if this is not true).

The JSON Extractor is much more flexible in that regard and can take any JSON structure and convert it to field names automatically, however it can only parse messages that are valid JSON and thus will fail to parse the messages from the above example.

It is currently not possible to have the best of both worlds: be able to both grok for a JSON pattern within a message and automatically extract fields from an unknown JSON structure.

P.S. I've slightly updated my previous comment to make the examples more clear.

[joschi]

It is currently not possible to have the best of both worlds: be able to both grok for a JSON pattern within a message and automatically extract fields from an unknown JSON structure.

@Bragolgirith Sure it is. You can either use a set of extractors or even combine extractors and pipeline rules.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

[jalogisch]

rule "extract-json"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let json = parse_json(to_string($message.message));
    let map = to_map(json);
    set_fields(map);
end