Loki: Loki and alerts

[soum1234]

Suppose I have to alert when there is an error log.let me know how to so it

[tomwilkie]

Hi @soum1234! We don't have a good answer for this yet, but it is something we plan on supporting. The current idea is we're going to first add a Prometheus-style query language for Loki, then use Prometheus-style rules and alerts on top of this.

[soum1234]

Thanks a lot for the response

[soum1234]

could we expect it in the next release?also could you let me know if there are some documents on querying in boltdb pls send the link
thanks

[BushnevYuri]

Prometheus style alerting for logs sounds truly amazing. Hopefully, this feature will be added soon.

[mcamou]

Has there been any work done on this? Any idea on when we can expect to be able to generate alerts in Grafana from Loki?

[cyriltovena]

I will start investigating this since LogQL has moved forward.

[JnMik]

yeah that would be nice to be able to send alerts if a specific query return results.
I would build queries looking up for "error" keywords and stuff like that.

[JnMik]

By the way, have you made any progress @cyriltovena ?

[cyriltovena]

Not yet merged, but you can use promtail to create those alerts in the meantime.

You can create metrics in promtail based on log retrieved from files, then scrape those promtail using a prometheus and create an alert.

(https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md)

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[mcamou]

How about if we're not using promtail? We're streaming logs directly into the Loki endpoint.

[cyriltovena]

We have plan to support Prometheus alert style.

[kalingals]

Hi @cyriltovena can you please tell me when do you plan to release Loki alerting feature?

[tomwilkie]

Loki already accepts PromQL like expressions (LogQL) on a Prometheus-compatible API, so the rough plan is to use the Cortex Ruler to evaluate Prometheus-style alert rules against logs. The Cortex Rules already exists, is horizontally scalable & highly available, so the integration should be relatively straight forward we hope.

This plan is in its super early stages mind you; input is more than welcome!

[bukowa]

Not yet merged, but you can use promtail to create those alerts in the meantime.

You can create metrics in promtail based on log retrieved from files, then scrape those promtail using a prometheus and create an alert.

(https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md)

Link is dead :(

[migueleliasweb]

https://github.com/grafana/loki/tree/master/docs/clients/promtail/stages

You are looking for the stages section. With that, you can define metrics based on matches against the log lines. Then you use Prometheus & Alertmanager as you normally would to create the alert itself ;)

I reckon this issue can be closed.

[bukowa]

@migueleliasweb thanks a lot, ill do that but i guess the issue should not be closed to provide built-in solution to this.

[migueleliasweb]

Hey @bukowa, I'm glad I helped :wink: .

Regarding the built-in solution for alerting, I would be surprised if this will ever be a feature. The integration with Prometheus/Alertmanager is already there and in every release it gets improved. Replicating the funcionality of creating and managing alerts would be highly deprioritized or even unwanted in this context. But that's just my view.

[bukowa]

@migueleliasweb My use case is loki for docker logs:
````yaml

services:
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml

promtail:
image: grafana/promtail:latest
volumes:
- /var/log:/var/log
command: -config.file=/etc/promtail/docker-config.yaml
depends_on:
- loki
- grafana

grafana:
image: grafana/grafana:master
ports:
- "3000:3000"

web:
image: nginx:alpine
ports:
- 85:80
logging:
driver: loki
options:
loki-url: "http://localhost:3100/loki/api/v1/push"
loki-external-labels: "container_name={{.Name}}.{{.ID}}"
loki-batch-size: "5000"
depends_on:
- grafana
- loki
- promtail
labels:
test: 1
````

Everything works great (great project thank you!) but now i cannot setup alerts for these logs (i really love simple solutions). So here goes my idea that this could and should be supported.

[peimanja]

Really interested in Prometheus-style rules and alerts and possibly ability to send the alerts to Alertmanager so same receivers and alert routes get applied there.

[cyriltovena]

This is great idea we want to implement.

[g574]

Hello,
I have tried these:

1. Get logs in Promtail > create metrics > Prometheus > Alertmanager > email. This works, but I find it very complex and I'm not good in regexes :)
2. Set up an alert in Grafana with an Alertmanager notification channel ( I didn't know one exists ). This way you can manage the alert in the GUI and aggregate the alerts in Alertmanager.
   I think the second option is near ideal to me.

[tonykimani]

any progress on this at all? Setting up alerts via Grafana based on a loki query.

[tonykimani]

@tomwilkie - is there a rough ETA? maybe mention this in GrafanaCONline

[cyriltovena]

We will talk about it during the Loki future talk at GrafanaCon. ETA beta around June.

[MalloZup]

@cyriltovena and others:
right now afaik we have this https://github.com/grafana/loki/issues/340#issuecomment-618851642

I was wondering if it will come soon somehting more minimalistic like prometheus alert-manager.yaml file and alerts.

Good work so far thx for feedback

[cyriltovena]

Yes @owen-d is working on alert manager style alerts. see here for more details https://youtu.be/TcmvmqbrDKU?t=1771

[MalloZup]

thx !

[JnMik]

Simple question here, why use Alert manager when Grafana can already trigger alerts ?

What's the benefits of using directly the alert manager of prometheus ? Does it offer more features ? Does grafana actually use AlertManager behind the same and that's why we must use a Prometheus Datasource ?

Using Grafana to alert on logs is not very user friendly at the time (Workaround of using Loki as Prometheus data source etc..) but I already have all my metrics alerts setup in Grafana, with webhooks to slack and pager duty and I obviously prefer to keep all the alert in the same tool.
I suppose Grafana will evolve in a way where alerting on logs become a simple matter ?

Thanks !

[cyriltovena]

Yes alert with grafana will evolve and improve. The grafana team have plan for it.

But we still want alertmanager for people who are used to this experience, which has definitively more features such as routing, grouping, silencing and high availability (deduping). See https://prometheus.io/docs/alerting/latest/alertmanager/

So there’s definitely two type of users and we want both to have a stellar experience.

Stay tuned.

[pakita]

Hi,
May I ask you to be little bit more specific. When do you plan to provide Loki release that would support true Grafana interface including ability to setup alarms? The latest “production” release v.1.5.0 can not support this option. I am asking this question because we are reviewing our log monitoring options and this caveat may put on hold production loki implementation. By any chance can I use loki API on my own to build alarm channel?
Thank you,
B

[owen-d]

We've been running a horizontally scalable version in a dev environment successfully for a week. It's based off the https://github.com/grafana/loki/pull/2275. Since then, there's been some discussion regarding which parts to include in Loki vs Cortex (our upstream dependency). I'm currently refactoring the approach, but the internal logic should stay similar to the approach detailed in that PR.

Suffice it to say this should be coming soon :)

[kycfeel]

Hi all.

I'm currently handling this by using Grafana built-in alert with Loki added as Prometheus (PromLoki). The alert itself is getting triggered well. I'm able to get it through the Slack.

But I also want to set the notification contains the actual log lines which have been queried. For now, It just displays the name of the Kubernetes container (in my case) and the hit number.

Is there a way to make this possible?

Thanks.

[pakita]

Well, I could establish PromLoki access but when I tried to setup Alert and tested the rule I got error message (below). The query that triggered this alert also provided below. I can see graphic representation, but it seems alarm query can not get over provided query. Any idea why?
Regards,
count_over_time(({source="syslog", instance =~ "."} |= "error" |= "timeout")[1m])
{
"firing": true,
"state": "pending",
"conditionEvals": " = true",
"timeMs": "1.158ms",
"error": "tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in #1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected $end\n|...",
"logs": [
{
"message": "Condition[0]: Query",
"data": {
"from": 1594217468353,
"queries": [
{
"refId": "A",
"model": {
"expr": "count_over_time(({source=\"syslog\", instance =~ \".\"} |= \"error\" |= \"timeout\")[1m])\r\n",
"instant": false,
"interval": "1m",
"legendFormat": "\" error timeout\"",
"refId": "A"
},
"datasource": {
"id": 3,
"name": "Prometheus-Loki"
},
"maxDataPoints": 0,
"intervalMs": 0
}
],
"to": 1594217768353
}
}
]
}

[pakita]

P.S regarding case above . Original query seems Ok at least from inspector point. Can not figure out why alert display error by suing this query
Regards,
{
"request": {
"url": "api/datasources/proxy/3/api/v1/query_range?query=count_over_time((%7Bsource%3D%22syslog%22%2C%20instance%20%3D~%20%22.*%22%7D%20%7C%3D%20%22error%22%20%7C%3D%20%22timeout%22)%5B1m%5D)%0D%0A&start=1594218360&end=1594218660&step=60",
"method": "GET"
},
"response": {
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"source": "syslog"
},
"values": [
[
1594218360,
"68"
],
[
1594218420,
"41"
],
[
1594218480,
"41"
],
[
1594218540,
"24"
],
[
1594218600,
"21"
],
[
1594218660,
"59"
]
]
}
]
}
}
}

[cyriltovena]

Hi all.

I'm currently handling this by using Grafana built-in alert with Loki added as Prometheus (PromLoki). The alert itself is getting triggered well. I'm able to get it through the Slack.

But I also want to set the notification contains the actual log lines which have been queried. For now, It just displays the name of the Kubernetes container (in my case) and the hit number.

Is there a way to make this possible?

Thanks.

@owen-d What do you think we could do there ? Adding some more metadata to the alert ? I like the use case but that's not easy to do.

[cyriltovena]

@pakita

Update Loki to 1.5.

[noob-master147]

Greetings,
I wanted to set up Webhook alerts with Loki, could not find relevant documentation about it.
@cyriltovena can you guide me to relevant docs?

[cyriltovena]

Only possible with Grafana Alert for now.

[MalloZup]

@cyriltovena I guess this issue is a duplicate of this https://github.com/grafana/loki/issues/1753 or the other way around but both concern same topic.
:beer:

[owen-d]

Closing, please see https://grafana.com/docs/loki/latest/alerting/ for docs!