Loki: Add support for Geo IP

[neilmunday]

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
It would be great if promtail/loki had a GEO IP feature like LogStash. E.g. regex identifies IP addresses in log message and performs GEO IP look-up to add additional fields to store location. This could then be used by the Grafana World Map plugin - though this plugin may also need updating.

Describe alternatives you've considered

ELK. It already has this feature -> LogStash Geo IP filter + Kibana world map.

Additional context
Add any other context or screenshots about the feature request here.

[adityacs]

@cyriltovena @slim-bean This is a good feature to support. However, this requires us to package the geolite2 or any other similar database file along side Promtail. Also, we should figure out if the license allows us to do this.

WDYT?

[WarraxUA]

nginx with the module ngx_http_geoip_module can writes geodata tag to the access log
just need a new "Worldmap Panel Plugin" for Grafana with support as a datasource - Loki

nginx log WITH GEODATA TAG -> Promtail -> Loki -> Grafana

P.S. I’m just surprised that the Grafana lab didn’t realize such a simple thing even at the time of the announcement of Loki

[wardbekker]

Hi @WarraxUA and folks. I've been using an preview branch of the upcoming metrics and field extraction feature. This allowed me to build the below dashboard, with metrics on high cardinality fields. For the Worldmap I've added the GEOIP module to Nginx, and added the country name to the log output. With the following expression I was able to sum by countryname as input for the worldpanel. (syntax pending to change, and it's a bit double escaped sum by (country_code) (count_over_time({filename=\"/var/log/nginx/access.log\"} | regexp \"HTTP\\\\/1\\\\.1\\\" (?P<statuscode>\\\\d{3}) (?P<bytessent>\\\\d+) (?P<refferer>\\\".*?\\\") \\\"(?P<useragent>.*)\\\" \\\"(?P<country_code>.*)\\\"\"[$__interval]))

[neilmunday]

Looking good!

[Wnthr]

Hi @WarraxUA and folks. I've been using an preview branch of the upcoming metrics and field extraction feature. This allowed me to build the below dashboard, with metrics on high cardinality fields. For the Worldmap I've added the GEOIP module to Nginx, and added the country name to the log output. With the following expression I was able to sum by countryname as input for the worldpanel. (syntax pending to change, and it's a bit double escaped sum by (country_code) (count_over_time({filename=\"/var/log/nginx/access.log\"} | regexp \"HTTP\\\\/1\\\\.1\\\" (?P<statuscode>\\\\d{3}) (?P<bytessent>\\\\d+) (?P<refferer>\\\".*?\\\") \\\"(?P<useragent>.*)\\\" \\\"(?P<country_code>.*)\\\"\"[$__interval]))

That looks better than good to me, is the preview branch you speak of available publically, alternatively, is there a time frame in which it will be available? I am currently in the prototyping stage of my project, so running unreleased isn't a concern.

[cyriltovena]

Here is the repo https://github.com/cyriltovena/demo/blob/master/logql/docker-compose.yaml#L8

There’s a small readme but also I gave a talk at GrafanaCon about this https://grafana.com/go/grafanaconline/loki-future/ see at the end, when you hear my weird and funny french accent you found it 😂

For ETA this is hard we’re still trying to make sure the syntax is easy to use and learn as we will live with this forever.

So soon TM.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[DanielFuchs98]

Are here any updates? It's a important feature

[afletch]

Being able to enrich data either upon collection in promtail (via a plugin?) or when that data lands in Loki, is really very important.

[adeleglise]

I would love seeing this too.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[neilmunday]

Any news?

[jonkristian]

Here's just a thought. Wouldn't it be more preferable to enrich data after the logs are collected, then one would not have to add extra overhead on the web server. To be honest though I really don't know how much overhead the geoip would add, but if you have many sites it could impact.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[neilmunday]

A comment to keep this issue open.

[WarraxUA]

example dashboard from wardbekker1 with Geo_IP
https://grafana.com/grafana/dashboards/12559

[afletch]

The dashboards shared in this thread are very nice, but they don't address the issue highlighted by the OP, which is; there is currently no way to enrich data either within Promtail or at the point of ingestion into Loki. GeoIP is a good example of this, but it would apply to any enrichment of collected log data using external lookups.
So, if you have a GeoIP field in the source log data, extracting it (and displaying it) is easy enough. If you don't have GeoIP in the source, then adding this label data is not possible in the flow at present.

(this is possible using fluend as a client, but then you're stepping outside the stack)

Hope this helps clarify what is being requested here, as things seem to have got muddied over time.