Fluent-bit: [Feature] Expose metrics in Prometheus format

[sdurrheimer]

Hi there,

As a hosted project of the CNCF, it would be nice to expose native Prometheus metrics like it has been done with a plugin for Fluentd.

[edsiper]

thanks. Definitely something to do.

[prune998]

where is it located in the todo-list ?

[edsiper]

@prune998 this issue, it's labeled as enhancement.

[lpic10]

I'm waiting for this feature to migrate from Fluentd

[prune998]

waiting is the new dying :)

[edsiper]

Work in process, current status

• [x] create internal generic metrics interface
• [x] input plugins use metrics interface
• [x] metrics exporter
• [ ] http server interface (work in process, this week)
• [ ] metrics exporter, register HTTP endpoints and share metrics

[edsiper]

Some progress, HTTP Server functional

[prune998]

That's great news ! thanks for the effort !

[edsiper]

We are getting closer:

• [x] create internal generic metrics interface
• [x] input plugins use metrics interface
• [x] metrics exporter
• [x] http server interface (work in process, this week)
• [ ] metrics exporter, register HTTP endpoints and share metrics

the last part is to add Prometheus format, we have plain JSON for now:

[edsiper]

looks like we are done

[prune998]

that's great ! thanks

[jsravn]

Any plans to incorporate fields into this? E.g. in Kubernetes it'd be nice to see the various values per pod or namespace.

[azweb76]

This is awesome. Are we ready to merge?

[azweb76]

Looks like this is already available. Thanks all! Should this be resolved?

[Quentin-M]

Hi there,

How can we get the queue length, in other words, how can we estimate how far behind fluent-bit gets reading/processing/sending the logs to the output(s)?

I would intuitively do input / output but something tells me that input only increases if output progresses too, as the pipeline must be moving forward for fluent-bit to read more.

[discordianfish]

@edsiper Guess this can be closed then?
Probably makes sense to track further improvements in #516

[edsiper]

Metrics already implemented:

https://docs.fluentbit.io/manual/configuration/monitoring

Fixed.

[bobykus31]

Wonder why metrics show up with strange numbers

cat /etc/td-agent-bit/td-agent-bit.conf

[SERVICE]
    Flush               10
    Daemon              off
    Log_Level           debug
    HTTP_Server         On
    HTTP_PORT           2020

[INPUT]
    Name                systemd
    Max_Entries         5000
    DB                  /var/log/systemd.db
    Mem_Buf_Limit       5MB
    Tag                 journal.0

[OUTPUT]
    Name                forward
    Match               *
    Host                forward.com
    Port                24888

[FILTER]
    Name                record_modifier
    Match               *
    Record              host me.com

curl -s http://127.0.0.1:2020/api/v1/metrics | jq

{
  "input": {
    "systemd.0": {
      "records": 11,
      "bytes": 6540
    }
  },
  "filter": {
    "record_modifier.0": {
      "drop_records": 0,
      "add_records": 0
    }
  },
  "output": {
    "forward.0": {
      "proc_records": **0**,
      "proc_bytes": **101869**,
      "errors": 0,
      "retries": 0,
      "retries_failed": 0
    }
  }
}

So the same is curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus, why fluentbit_output_proc_records_total is zero while fluentbit_output_proc_bytes_total is 101869 or whatever?