Loki: Allow out of order log submission

[shughes-uk]

I'd love to use Loki in a distributed system easier and without being forced relatively high cardinality labels based on something like process ID. This goes double for systems like AWS Lambda.

This main obstacle to this for me is being unable to submit 'out of order' log lines, it would be great if loki could have a feature that would enable this.

At one point I found an old issue relating to this request but it was closed with "not something we need before releasing". Perhaps it is time to revisit this?

Cheers

[cyriltovena]

Hello,

To be totally honest this is how Cortex storage works and Loki is based on it. I'm not saying we are not willing to work on it but the amount of work required is consequent. So far we're trying to find alternatives instead of reworking the whole foundation of Loki.

I'm curious to know how you are sending logs to Loki ? Are you using Promtail or in-app API call ? Do you care about ordering ?

Something that could be done is using a server side timestamp in Loki.

[shughes-uk]

In-app API calls, this library to be exact https://github.com/GreyZmeem/python-logging-loki .

I would say the ordering is not ultra critical as my data does contain it's own timestamps. Our app essentially proxies the loki websocket to a front end UI, and having the UI do sorting on new log lines as they arrive is trivial.

For querying exact time ranges we would just expand the range by a value (the largest amount of time we would expect a log to be 'late' by) and filter it down to the desired time range.

So yes a flag for loki to just use it's own timestamps would be perfect for us.

[slim-bean]

As @cyriltovena mentioned, trying to buffer the data and sort it inside Loki is a big task, and would come with a big consequence in memory usage, especially with a large amount of streams. This is primarily why we aren't too excited about implementing this and have been deferring the task of sorting to the clients.

However, the idea of applying the timestamp by Loki at the time of ingestion is certainly intriguing!

If ok with you @shughes-uk I'd like to rename this issue to reflect this new feature?

[slim-bean]

(or if you'd like you can create a separate issue for this and keep this one for posterity)

[shughes-uk]

Fine with me! Both solutions are totally workable for my use case, and I think it makes sense to do the easy one first.

[tmikaeld]

I'm using Loki to log stack traces from distributed serverless workers, so this keeps coming up as a problem. Applying timestamp @ the time of ingestion would work well for this scenario.

[pstibrany]

If we do this, let’s make sure that time is applied by distributor, so that all ingesters get the same value.

[cyriltovena]

How do we handle batches ? For performance reason, data is sent in batches. Should we just realign out of order entries ?

[imtrobin]

This is quite a concern for me. I'm planning to use HTTP API calls, if it drops out of order message, how do you account for network latency? We should not use server timestamp as it could be incorrect.

[shughes-uk]

I imagine using server timestamps for out of order submissions could be a flag that you can disable/enable? Still not the perfect solution but for people who don't need exact precision it would get a solution out there faster.

[judemorrissey]

hello everyone, apologies if there's already another issue that has more information on this topic, are there any developments or further discussions on this? we realized we were getting a fair amount of loglines being dropped, and while we acknowledge that we could use more cardinality as we are probably too conservative with labels, this issue is of interest to us

thank you in advance!

[verticelo]

This would be a great feature and one of the major blockers for our organization to adopt Loki. Is there any setting that a timestamp can't be in the future? Suppose there is a bug or a manual entry or something stupid, is there a way of stopping the minimum timestamp being in the future so logs won't be rejected for a long time?

[Pizzahd88]

For Arduino projects where the controllers has no idea of the time of day, this would be a fantastic feature - letting Loki set the timestamp.
It would make IOC devices require less power while having the possibility of using the logging features in Loki.