Loki: Delete log streams via API

[cyriltovena]

Is your feature request related to a problem? Please describe.

When the users will want to tweak Promtail scrape configuration (relabeling, pipeline, discovery) to work with their own deployments, they'll probably generate some unused or unwanted Loki streams.

It would be nice if we could search/filter through streams and ordered them by last chunk received, throughput and total size. Then be able to deleted those we don't want anymore.

This could also help users to create some sort of cron job to apply retention. @sandlis

We could use prometheus to scrape promtail and who creates those metrics. Loki could then use this to query for use within the API.

Describe the solution you'd like

1 - List all streams ordered by last data seen and throughout over the last 5minutes, we should be able provide label matcher {app="mongod" cluster="us-east1"} to narrow down the search, this should be a query limited by 100 results by default.
2 - Delete a list of stream via their labelset and may be from a specific time.

I feel like this could be useful to have in Grafana some sort of stream management UI @davkal ?

[tomwilkie]

Lets try and match the Prometheus API here: https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

[cyriltovena]

We should start with GET, https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers

/cc @davkal

[slim-bean]

related to #113

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[blocke]

This ticket was closed but #113 only covers the GET portion. I'm hoping the DELETE portion gets included too as I just spent a couple hours fighting an incorrect timestamp parsing pipeline. Should a reminder for DELETE be added to #113 or can this ticket be saved from the stale bot?

Thanks.

[bastjan]

I've hacked together a rough prototype of the delete API after seeing cortexproject/cortex#2103 and blowing up my index with an erroneously parsed timestamp.

Is anyone working on the delete API on the Loki team? I could finish up my prototype over the next 2-3 weeks if needed.

[sandeepsukhani]

Not yet. I appreciate your efforts! We would like to get it working for Cortex and then reuse most of the code in Loki.
Also out of curiosity, was the issue that you were facing introduced by code from that Cortex PR? I would like to fix it if it is.

[bastjan]

My issue was unrelated. I parsed the "time since boot" counter from the linux kernel log as label by accident.

The cortex PR allowed me to use the new storage.DeleteSeriesIDs. Which made it quite easy to implement the remainder :)

[daveram]

Is there any way to manually go in and delete all instances of a label? We're running into the scenario where we started collecting the wrong logs under the wrong label name, promtail reconfig was nice and easy but now we have the old labels hanging out with invalid logs. Would like to clean up, and since the API doesn't support this like Prometheus yet, hoping there's a more manual way.

We're using two different setups, one with filesystem and boltdb for chunks/index, the other with s3 and cassandra. Both have the same scenario happening

[ayZagen]

We also need this also. It would be great to have something like in prometheus. link

[tibin-mfl]

Do we have any update on this, Is delete api available on Loki ?
What can we do if we want to put some on demand logs to loki for some some issue debugging and later delete that log stream.

For Example :
We push DB logs to loki for analysing some production issue after issue debugging I don't want the logs so I need to delete that logs to reduce unnecessary storage.

[cyriltovena]

It's actively being worked on by @sandeepsukhani

Feel free to give him support, it's OpenSource !

[Tschis]

@cyriltovena Hi! Thanks for the info! 💪

But while it is not done, do you know of a workaround that can be used?
Due to some circumstances we created labels on a field that went above 100,000 entries. Now we are stuck with an error message cardinality limit exceeded for logs{log_code}; 123474 entries, more than limit of 100000.

Since we hold our logs for many days, is there currently any way I can manually delete these label values?

[diemus]

I have to delete all data to solve the "cardinality limit exceeded for logs ....more than limit of 100000" problem. and I have no idea which label is high cardinality. is there a tool to see this? I just found the index size grow fast these days, but can not find the cause

[cyriltovena]

We recently released a way to see that with logcli. (aka --analyze-labels)

see:

• https://grafana.com/blog/2020/08/13/loki-1.6.0-released-metric-query-performance-up-to-10x-faster-push-logs-from-any-client-to-promtail-query-language-and-logcli-enhancements-and-more/
• https://github.com/grafana/loki/pull/2497

It does exactly what you want ✌️

[AMoghrabi]

@cyriltovena Thanks for the info. Are you aware of a way to actually delete the labels? I couldn't find information on that in the PR or blog. Thanks!

[tmikaeld]

Same problem here, need to clear the logs due to GDPR request.

Seems silly to wipe all logs, when there are only a few labels we need to clear.