Loki: Multi cluster support?

[omerlh]

Is there an official recommendation regarding running Loki on multiple clusters? Currently, each time we’re deploying a cluster, we need to set up a new data source in Grafana manually (we have single Grafana showing data from all clusters). It’s a bit hard to scale, maintain and use (cannot loon for data from all clusters).
Are there any existing solutions? It’s not the same issue as tenancy, more like the issue that Thanos/federation solved for Prometheus - one instance that can be used for querying all clusters.

[cyriltovena]

Right now the only scale path we offer is tanka in this repository production/ksonnet/, it's a distributed cluster where you can send all your logs from any other kubernetes cluster. (That version scales easily to 150MB/sec)

In the future we're looking at improving helm for that same use case and we're also thinking about loki to loki remote writes.

Regarding the datasource you can provision it via the Grafana API, but that doesn't solve looking at data from all clusters.

Federation is a good idea, but I'm not sure the current helm chart can handle multiple cluster load.

[omerlh]

We’re using jsonnet to install Loki today, so if federation is possible I’ll be happy to test it.
Where I can read more about the distributed cluster architecture? Does it mean that one cluster is the master, and the rest just write to it?

[cyriltovena]

No you just run a single cluster and it can eat all you're sending. It's not federation.

I assume you were using helm, if you already use jsonnet, then as a workaround for now you could just have a single big cluster and ingest all your k8s cluster with it.

[omerlh]

So promtail will send all the logs to a centralized loki instance?

[cyriltovena]

Yes that’s how we run it.

[slim-bean]

Closing this issue as I _think_ this is resolved, please comment if that's not the case. Thanks!

[k8s42]

In my case, I've multiple Kubernetes clusters which host apps which specific teams support. In each cluster, I've Loki and Promtail deployed. I've a centralized Grafana from which I create a data source to each cluster's Loki endpoint. It's too many Lokis to manage. What I'd like to do is only have promtail deployed to each Kubernetes cluster which will push data to a centralized Loki instance. Problem with this approach is the centralized Loki instance will contain data for all Kubernetes clusters, which in turn makes all the data available from all K8S clusters to all teams. I'd rather not filter data in Grafana dashboards by namespace for Loki logs to display to each team. Is there a way to control this at Loki level?

[migueleliasweb]

I have a very similar case as @omerlh .

For me, the more important feature would be a single data source to Loki API from Grafana, including the ability to aggregate data scattered between different Loki instances.

I don't think there's any documentation regard running Liki in multiple cluster setups =/. Not sure if multi tenancy is quite suitable in this scenario.

Edit:

As far as I know, these are the current possibilities in deploying Loki in a multi cluster setup. I'm yet to test the second one in a week or so if priorities allow:

• Have promtails from each cluster to send logs to a centralised Loki installation.

  • Pros: Easy setup
  • Cons: All logs are centralized. If something happens to the cluster where the logs are persisted, it's dooms day.

• (not tested yet) Run Loki in it's distributed form (separate components deployed) across multiple clusters. Each cluster would have it's down instance of distributor, ingester and querier. This would require some level of orchestration to ensure all components are properly configured at all times - maybe a small operator would be necessary.

  • Pros: This would probably give you more reliability against failure of a given cluster but I'm not sure how Loki would behave if one of it's index/chucks is not accessible.
  • Cons:
  • Possibly requires custom code to keep all components able to talk to all the instances.
  • Substantially more complicated to deploy/maintain
  • In the case of geographically separated clusters, data transfer between far regions can be slow and costly.

[onhachoe]

I'm looking along the same lines with @migueleliasweb, where I want single API endpoint for aggregation across geographically spread out log generation.
I was looking into the single stretched loki cluster, but wasn't fully sure how perf. would be impacted, so holding off this route.
Looking into some alternative solution, I could setup loki in each region, and maybe have some sort of fanout layer to query multiple lokis and aggregate results in single contained response. Would be nice if this can be supported natively.

[slim-bean]

There are a couple different concerns here and use cases here I can provide some feedback and my opinions on so I'll try to break them out.

Currently the recommended and really only way to use Loki over many clusters is to run a single Loki instance and send logs from all your clusters all over the world to that centralized Loki, this is how we run Loki. We attach a cluster label at each promtail and are able to query all our logs using Grafana and a single datasource.

I was looking into the single stretched loki cluster, but wasn't fully sure how perf. would be impacted, so holding off this route.

Turnaround from promtail to logs in Loki is on average less than 2s for all our clusters, Promtail sends batches either when they are full or after a 1s timeout which likely adds to a fair amount of this 2s delay. I wouldn't expect performance of a centralized Loki cluster to be an issue.

Cons: All logs are centralized. If something happens to the cluster where the logs are persisted, it's dooms day.

Yes but this is a bit nuanced, as if you run Loki separated in all your clusters, if anything happens to any of your clusters the logs in those clusters are gone, so while you are minimizing the scope of the impact it's not really addressing the problem. To get multiple region or availability zone durability with Loki you would need to run separate clusters in separate regions and configure promtail to send to both of them (there are some caveats with this approach in how promtail is coded today), or run separate promtail instances for each cluster (probably the best approach). Or you could look at replicating the chunk and index data between regions by a different mechanism and setup a standby Loki cluster and use some kind of failover DNS, etc.

Mostly you would need to use any of the methods you would use today to make a system available/durable against region or AZ wide issues and I don't think this is really an argument against running a centralized Loki. That being said, and I'm not even sure what form this would take yet, I'm sure there is more we should do and will hopefully do to make it easier to have better Loki availability over different cloud provider regions.

Run Loki in it's distributed form (separate components deployed) across multiple clusters. Each cluster would have it's down instance of distributor, ingester and querier. This would require some level of orchestration to ensure all components are properly configured at all times - maybe a small operator would be necessary.

This could be made possible, the distributed hash ring used by Loki could probably be made location aware to make sure replicated data is sent to different locations but this kind of setup would work very poorly for querying and I don't think would be a good fit for Loki without some big changes into how the querying works. I think solving the HA over multiple regions can probably be solved in other ways a bit easier in other ways.

In my case, I've multiple Kubernetes clusters which host apps which specific teams support. In each cluster, I've Loki and Promtail deployed. I've a centralized Grafana from which I create a data source to each cluster's Loki endpoint. It's too many Lokis to manage. What I'd like to do is only have promtail deployed to each Kubernetes cluster which will push data to a centralized Loki instance. Problem with this approach is the centralized Loki instance will contain data for all Kubernetes clusters, which in turn makes all the data available from all K8S clusters to all teams. I'd rather not filter data in Grafana dashboards by namespace for Loki logs to display to each team. Is there a way to control this at Loki level?

I think this is an interesting question and there is probably a good feature request for Grafana in here which is provisioning a Loki datasource and specifying certain labels that are require for each query. This is already possible in a different way today just using dashboards, you can make a dashboard with a log panel and template variables for changing labels, but in the query editor you can hard-code certain labels. Take a look at this blog for some more details.

I'm not sure if there are other advantages or reasons to do it at a datasource level or not but I hadn't though of that before.

_some additional comments:_

First thanks for the discussion and feedback everyone! This is an area that clearly needs some better documentation, guidance and work.

TL;DR Multi cluster Loki support is best obtained via a single Loki instance and remote agents sending with a cluster specific label. HA for Loki over regions is currently going to be best accomplished by running separate Loki instances and sending data to both or replicating the Loki storage data (chunks and index) via another means to a different region to a standby Loki cluster.

In related features though I do think we want to support a federation model in the future of some kind with Loki where you could run a Loki instance per cluster and also have that instance send data to a centralized Loki instance (either all the data or a subset).

[migueleliasweb]

Thank you so much, @slim-bean ! That was some very useful ideas!

I will be spiking one of theses setups in following week. Cheers!

Edit:

Just dumping an idea here.

I wonder if the approach of having a single frontend for multiple standalone Loki instances would be easier to implement as a first version of the distribution to multiple clusters

Loki already provides an API so by just creating a client that can aggregates the data from multiple instances into one result would (with my limited knowledge of Loki internals) solve the problem of having a single point of contact. If would be also interesting due the fact that the setup could be more gradual with many standalone Loki installations (single binary running everything) could still be treated as a single unit with this single frontend.

Throughput would potentially be a problem and live tailing logs would probably be a challenge across multiple instances but maybe it's worth the try.

[lipmanp]

The actual query will be distributed to nodes, and results will be aggregated back in query client. it would be similar to how Elasticsearch query works.

[yanirj]

Regarding @lipmanp's comment - is it possible to run a single hub Loki query frontend (https://grafana.com/docs/loki/latest/configuration/query-frontend/) and have all other clusters' Loki instances point to the hub frontend? (let's assume multi cluster service discovery works or a LoadBalancer is in place)

I'm asking because I'm not sure if having answers back from multiple Loki instances would be really aggregated or the original query will simply be split to N parts of different time frames will be sent to each Loki instance. I'm not interested in parallelization here to accelerate queries (query frontend's original goal), just aggregation. Basically similar to a Thanos setup.

[tivvit]

Regarding @lipmanp's comment - is it possible to run a single hub Loki query frontend (https://grafana.com/docs/loki/latest/configuration/query-frontend/) and have all other clusters' Loki instances point to the hub frontend? (let's assume multi cluster service discovery works or a LoadBalancer is in place)

I don't think that it is possible now but I am thinking about this solution for a while. I will try to implement some POC over the weekend.

[lipmanp]

regarding to @slim-bean comments on how to achieve HA, it seems only 2 ways:
1) to run separate Loki instances in parallel or
2) set up a mechanism to replicate index/chunks to another location.
but in practice, I still need some guidance:
for 1), you could switch data source to the healthy Loki instance, but the question is how to rebuild the failure one and having data sync with the working instance
for 2), I'm not sure if index/chunks always contain latest data, e.g. data in memory within working instance is in sync with physical index/chunks files?

without recovering quickly, only one Loki instance running, no another instance to be switched to, if it is failed luckily.