Loki: Memory keeps growing

[negbie]

During my tests with about 1000 pps I see randomly following errors in the explore tab:

err="unrecognised chunkTimeRangeKey version: ''"
err="invalid chunk time range value: 330066616b653a6431373930323a684a6a7665707647615836574843464d4c374b557430574c48476271577179427070542b7947312b77376b003033626239623933"

Checking my system stats I see that loki is eating all my memory:
19945 root 20 0 17,360g 6,479g 0 S 4,6 86,3 0:39.02 loki

Docker logs:

ts=2019-01-06T17:25:27.4395411Z caller=spanlogger.go:36 org_id=fake method=SeriesStore.Get level=error msg=lookupChunksBySeries err="unrecognised chunkTimeRangeKey version: '2002fake:d17902:logs:node_id'"
ts=2019-01-06T17:27:01.015789765Z caller=spanlogger.go:36 org_id=fake method=SeriesStore.Get level=error msg=lookupChunksBySeries err="unrecognised chunkTimeRangeKey version: ''"
ts=2019-01-06T17:27:12.15825352Z caller=spanlogger.go:36 org_id=fake method=SeriesStore.Get level=error msg=lookupChunksBySeries err="unrecognised chunkTimeRangeKey version: ''"
ts=2019-01-06T17:28:32.362907914Z caller=spanlogger.go:36 org_id=fake method=SeriesStore.Get level=error msg=lookupChunksBySeries err="invalid chunk time range value: 330066616b653a6431373930323a684a6a7665707647615836574843464d4c374b557430574c48476271577179427070542b7947312b77376b003033626239623933"
level=error ts=2019-01-06T17:36:02.760980878Z caller=consul_client.go:182 msg="error getting path" key=ring err=null
level=warn ts=2019-01-06T17:36:03.347543711Z caller=gokit.go:43 msg="GET /api/prom/label (500) 1.998232ms Response: \"empty ring\n\" ws: false; Accept-Encoding: gzip; User-Agent: Go-http-client/1.1; "

[billimek]

Confirmed here too:

[idobry]

I can confirm this when running Loki in Kubernetes and while sending him logs through the API (from external machine) the memory usage is sky high

[SvenDowideit]

yup, I see the same thing using https://github.com/hikhvar/journaldtail

[kovetskiy]

Is this PR going to solve the issue?
https://github.com/grafana/loki/pull/209

[nothinux]

i have same issue, loki eating all of my memory when i querying log data from grafana

fatal error: runtime: out of memory

[mriedmann]

same here. Loki crunched 22G of memory overnight. Pretty unusable atm.

[ThomasADavis]

anyone know what version to roll back to that does not have this memory leak?

[kovetskiy]

@ThomasADavis nope, there is no such version.

[ThomasADavis]

Well, I just noticed there was bunch of merges, so I re-downloaded and rebuilt it.. and now it's working without eating all the memory.

[negbie]

Can confirm that the memory issue is gone or at least rising very slowly. Needs more longterm investigations. But when the traffic grows this came up
https://github.com/grafana/loki/issues/279

[kovetskiy]

Same for me, with #209 it works much better.

[tomwilkie]

Thanks folks, sorry for the silence on this. #209 was intended to fix this, thanks for the testing! We're working hard to make this more useable.

[wapmorgan]

Got similar problem: #960