Loki: Fluentbit forwards from Journald to Loki experience 'out of order' error

[tfchan0]

Describe the bug
Fluentbit taking log entries from Journald and output to Loki.
From the logs:
level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2020-02-07 14:51:17.592162 +0000 UTC ignored, reason: 'entry out of order' for stream: {job=\"fluent-bit\"},"

To Reproduce
Steps to reproduce the behavior:

1. Loki (v1.3.0) deployed
2. Fluentbit deployed with Loki plugin, with input from systemd plugin

Expected behavior
Understand that this is not tested yet but wonder if any plan to support in future.

Environment:

• Infrastructure: K8s
• Deployment tool: helm

Screenshots, Promtail config, or terminal output

    [INPUT]
        Name              systemd
        Tag               host.*
        Parser            docker
        Path              /var/log/journal
        #DB                /var/log/fluentbit.db
        Mem_Buf_Limit     140MB
        storage.type      filesystem
    [OUTPUT]
        Name                loki
        Match               host.*
        Url                 http://loki:3100/loki/api/v1/push
        Labels              {job="fluent-bit"}
        LineFormat          json
        LogLevel            info
        BatchWait           1
        BatchSize           1001024

Errors:

level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=429 error="server returned HTTP status 429 Too Many Requests (429): ingestion rate limit (4194304 bytes) exceeded while adding 849 lines for a total size of 1000850 bytes"
level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2020-02-07 14:51:17.592162 +0000 UTC ignored, reason: 'entry out of order' for stream: {job=\"fluent-bit\"},"
level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2020-02-07 14:54:23.122453 +0000 UTC ignored, reason: 'entry out of order' for stream: {job=\"fluent-bit\"},"
level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2020-02-07 14:58:42.818418 +0000 UTC ignored, reason: 'entry out of order' for stream: {job=\"fluent-bit\"},"
level=error caller=client.go:246 id=0 component=client host=loki:3100 msg="final error sending batch" status=429 error="server returned HTTP status 429 Too Many Requests (429): ingestion rate limit (4194304 bytes) exceeded while adding 1138 lines for a total size of 999989 bytes"

[cyriltovena]

Your Loki is rate limited, you should increase those limits see https://github.com/grafana/loki/tree/master/docs/configuration#limits_config it might be the reason for the other errors.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[tfchan0]

Sorry for late response. I have tried setting a large ingestion limit but similar issue of "entry out of order". Is it because there's no guaranteed that systemd input is in order? Or other config that I may have missed?

[tfchan0]

Tried with the latest version built and it seems good so far. Closing the issue.

[stvnwrgs]

@tfchan0 which version have you used?

[TPX01]

Asking myself the same thing. I'm having the same issue with latest version. Fluent-bit Loki plugin 1.4.1..