Kube-prometheus: Add support for logs (loki/fluentbit)

[atmosx]

What is missing?

Loki and fluentbit integration.

NOTE: From a quick look at promtail's release notes looks it is bug-ridden which makes sense as it's a young project. Fluentbit on the other hand is a battle-tested alternative.

Why do we need it?

To add logging server to grafana. This way can have a full featured stack instead of just metrics.

Jaeger would also be a nice addition in the future, since Grafana now has integrated support for traces.

Environment

any

Anything else we need to know?:

I would like to know if you would be interested in the idea of adding Loki as part of the stack.

[paulfantom]

kube-prometheus project is about delivering monitoring stack based on prometheus. Logging as such is out of scope for the project.

[brancz]

While I agree with @paulfantom I think there have been various folks who have created integrations with loki. I do think it belongs in a separate project but I agree it would awesome to have well integrated stacks. This one is a little outdated but I know there are some folks working on stuff in this direction :)

[brancz]

The integration is pretty trivial actually. You just setup your loki stack and add loki as a datasource to your kube-prometheus Grafana instance:

  {                                                                                                                                                                                                                                           
    _config+:: {
      grafana+:: {                                                                                                                                                                                                                            
        datasources+: [{                                                                                                                                                                                                                      
          name: 'loki',                                                                                                                                                                                                                       
          type: 'loki',                                                                                                                                                                                                                       
          access: 'proxy',                                                                                                                                                                                                                    
          org_id: 1,                                                                                                                                                                                                                          
          url: 'http://loki.loki:3100',                                                                                                                                                                                                       
          version: 1,                                                                                                                                                                                                                         
          editable: false,                                                                                                                                                                                                                    
        }],                                                                                                                                                                                                                                   
      },                                                                                                                                                                                                                                      
    },
  }

[jobh]

For those looking for a quick recipe, I tried it just now using the suggested datasource configuration in the comment above. I had to change the url address to http://loki:3100 (instead of loki.loki). The following helm commands add the loki stack:

helm repo add loki https://grafana.github.io/loki/charts
helm repo update
helm upgrade --install loki loki/loki-stack --namespace monitoring

[paulfantom]

I think we would like to have this as an example jsonnet file in examples/ directory. Possibly with some documentation. Any takers for doing this? :)

[xpepermint]

It would be great to have loki and promtail alert presets as addons.

[brancz]

At this point it is unclear how integrating loki would our ability to distribute kube-prometheus as apache 2 license. For the time being it will need to be an extension hosted outside of this repository. That shouldn't discourage people from doing so outside of this repo though and link to it! :)