Graylog2-server: http JSON input -> JSON extractor, difficult to extract structured data values

[kmansoft]

In reference to:

https://groups.google.com/d/msg/graylog2/js9dBgEHaDA/-jk09IHrAAAJ

Expected Behavior

I've got a JSON HTTP API which returns several data items which are of interested to me, something like this:

{
  "objs": { "devs": 100, "subs": 200 },
  "dbstats": { "read": 300, "write": 400 }
}

I've defined an HTTP JSON _input_ in Graylog, and was hoping to then use the JSON extractor to map the values of subs, devs, read, write into properties of the collected messages (for monitoring, charts, etc.)

Current Behavior

Let's focus on subs and devs for now.

The HTTP JSON input has its JSON path of data to extract set to $.objs.

And then I'm trying to use JSON extractor to break that apart into devs and subs.

But that does't work because the HTTP JSON input stores the data (in result) as some sort of Java map.toString(), not valid JSON, and then the JSON extractor can't work wit that:

Expected values in result:

{ "devs": 100, "subs": 200 }

Actual values in result:

{ devs=100, subs=200 }

Possible Solution

1 - Change the HTTP JSON extractor so it can store its output (result) as JSON. Maybe a setting so it's optional, preserving the current behavior.

OR

2 - Add a way to configure JSON path -> value mapping right in the HTTP JSON extractor, allowing for multiple values.

So instead of the current (still not working, but maybe possible with solution #1):

JSON HTTP input -> calls the HTTP JSON API -> takes $.objs and stores as JSON -> JSON extractor takes this JSON apart and adds the values of subs and devs to each message --

-- it would be possible to do this:

JSON HTTP input -> calls the HTTP JSON API -> takes $.objs.devs and stores as devs, takes $.objs.subs and stores as subs, etc.

A JSON Extractor would not be needed then here.

Context

Trying to collect structured typed values from an HTTP JSON API using a HTTP JSON input.

Your Environment

• Graylog Version: 2.0.3
• Elasticsearch Version: 2.3.5
• MongoDB Version: 3.2.9
• Operating System: Debian Linux 8.5
• Browser version: Google Chrome 52

[towolf]

Yeah, that is not helpful; at all. I want to get at the numbers there.

[ermak83]

Any news on this?

[kiwimato]

+1 on news, does anyone know of a workaround ? It could be tackled using multiple extractors, as long as the child JSON format doesn't change

[joschi]

@ninjaslikecheese You can use parse_json() and select_jsonpath() to extract the information you need in pipeline rules.

[rdoering]

Yeah I also ran into this pit.

[kiwimato]

@joschi I tried that, but for some reason when I changed the order, moving Pipelines before Chains, caused it to drop the messages. I now noticed that for some reason the extractors works for JSONs that don't contain arrays. It doesn't work for {"args": [{"id": 240, "organization_id": 10}] }

[kiwimato]

For those interested in version 2.4.3 it worked, if I stripped the [] chars using "Replace with regular expression" in Extractors, and then use the JSON extractor it works, though I don't know if it works using multiple items in that list, or if they will be overwritten..

[piellick]

Hi everyone, no news about it ?

Someone can provide an exemple of workaround through a pipeline on Graylog 3.x ? I have trouble to parse a full ($.*) Json structure.

Here is an exemple, "select_jsonpath" function return an error :

rule "JSON_extractor"
when
has_field("result")
then
let json = parse_json(to_string($message.result));
let fields = select_jsonpath(json, '$.*');
set_fields(fields);
end

an other exemple who doesnt work :

rule "JSON_extractor"
when
true
then
let json = parse_json(to_string($message.result));
end

I had done many test but Graylog JSON path from HTTP API is still removing quote from JSON.

There are plenty of discussion but no responses:
https://community.graylog.org/t/json-path-from-http-api/9015

Thanks

[piellick]

hello @joschi can you explain your workaround please ?