Loki: ingestion rate limit exceeded

[rndmh3ro]

Describe the bug

I'm getting the following lines in loki when sending logs from promtail (using static_config to scrape logfiles):

level=warn ts=2020-04-09T09:15:05.866134665Z caller=client.go:242 component=client host=172.29.95.195:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): ingestion rate limit (8388608 bytes) exceeded while adding 311 lines for a total size of 102169 bytes"

I don't quiet understand this line or is it misleading?

It says "adding 311 lines for a total size of 102169 bytes". But the total size of 102169 bytes is less than the ingestion limit of 8388608 bytes.
Or does it mean that it tries to store 311 * 102169 = ‭31.774.559‬ bytes of data, thus exceeding the ingestion rate limit?

To Reproduce
Steps to reproduce the behavior:

1. Started Loki v1.0.4
2. Started Promtail v1.0.4
3. let promtail parse a huge logfile

Expected behavior
I'd like to understand what this error exactly means.

And also how to avoid it. :)

Environment:

• Infrastructure: bare-metal

Screenshots, Promtail config, or terminal output
Loki limits config:

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 8

Promtail loki-batch-size is set to default.

[owen-d]

The message is describing which request triggered the rate limiter. Likely this is because previous batches sent from promtail consumed the remainder of the ingestion budget. If you aren't seeing these regularly, it's ok - promtail is designed to handle backoffs and continue ingestion. If these messages are common, increase the ingestion_rate_mb and/or the ingestion_burst_size config :)

https://github.com/grafana/loki/tree/master/docs/configuration#limits_config

[rndmh3ro]

I had this problem fairly often during my tests and the initial setup if promtail. Probably because it was indexing all these huge system-logfiles at the same time.
However increasing the ingestion_burst_size to 16mb seems to have fixed even these problems.

Thanks for your help!