Loki: Provide the option to extract all keys from a json input

[rrajkomar]

Is your feature request related to a problem? Please describe.
When parts of a log line are generated by application code that is from a library or a framework, internal json keys may not be known in advance.
However being able to search with those keys as labels them might prove useful

Describe the solution you'd like
It would be great if the json pipeline_stages has an option to auto-extract all keys from the json input (i.e. if no expressions keys are provided)

It would then require the labels stage to also accept using all keys extracted from a given source to be then used as labels

Something like this :

      - regex:
          expression: '^\[(?P<time>[^\]]+)\] (?P<channel>[a-z]+)\.(?P<level>[A-Z]+): (?P<message>.+) (?P<context>.+) (?P<extra>.+)$'
      - json:
          expressions:
          source: context
          output: context_keys
      - json:
          expressions:
          source: extra
          output: extra_keys
      - labels:
          channel:
          level:
          message:
          context:
          extra_sources:
                  - context_keys
                  - extra_keys

[rrajkomar]

Note : also it would be nice if the json parser was recursive (to avoid having to set up multiple json stage configurations instead of a single one at the beginning)

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[rrajkomar]

Are there any plans to make this feature available ?

[pomazanbohdan]

Also interested in this question.
Configured windows events to a log file using nxlog and save them in json format.
And since there are a lot of different data, I would like to automatically lay it out in labels

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[rrajkomar]

Please reopen this issue as we've received no information on whether it is currently possible nor if (and when) this feature might be available

Also are there any plans to be able to handle gelf-like logging ?
In a way that could allow sending for a example a PHP exception object and get somethign that's usable in Loki UI ?

[cyriltovena]

We think https://github.com/grafana/loki/pull/1848 is a better alternative as it won't require you to configure anything client side.

Gelf could be added to parsers from the PR above but I believe, the glob will already help.

[rrajkomar]

Mmmm I'm fine with waiting to see how it plays out but I'm not entirely it will fit our needs.
In the meantime I may hav found an alternate solution for my use case (still needs some work though)