Loki: Documentation, examples?

[setevoy2]

Hi!
First of all - thanks for the Loki - really great.
But still few questions, please.

1. "federation" analog?

Is there any way to configure Loki's instances in the same way as Prometheus uses its federation ability?

I.e. - I have a Stage environment, with 3 AWS EC2 instances running.
On each EC2 - I have Prom's exporters running + Prometheus itself to collect metrics from local exporters.
Then Prometheus exposes them via its _:9090/federation_ to the central Prometheus instance.

So I'd like to make the same with Loki - collect logs with promtail and Loki on some environment's instance, and pass them to the "central" Loki on the host with Prometheus server and Grafana.

Is it possible? Can't see any docs about such approach, unfortunately.

UPD: "Solved" this: Loki and Grafana runs on one host, promtail "worker" - on another, and just has url: http://loki.example.com:3100/api/prom/push.

2. Loki's storages

Found only one doc where storages configuration was mentioned - here.

Can I use local filesystem instead of AWS S3? I think so - just map new volume via Docker like:

  loki:
    image: grafana/loki:master
    volumes:
      - /some/local/storage/:/tmp/loki/chunks/

Is it correct? And the same for indexes.

And is there any way to use MySQL/Postgres etc as chunks storage?
And is there any way to use something else instead of DynamoDB etc as indexes storage?

3. Documentation, examples?

And finally - the main question... Docs :-)

Where can I see storage_config description, available options etc for the Loki, for example? Or - scrape_configs for the promtail? The only way I found since yesterday was an NGINX's "example" here.

Or it is absolutely similar to the Prometheus?

Thanks in advance and sorry for posting all question to the same issue.

[base698]

Would be nice to get some promtail examples. I was unclear that was even the log agent. Can you pull labels out of the log files themselves? Or would that require preprocessing?

[yubozhao]

@setevoy2 Traditional database is not a good solution for chunk storage, typically cloud object storage is better for this task. You can use local file(boltdb) as index table, or Google's big table. Any reason you are not looking for DynamoDB?

[setevoy2]

@yubozhao Yah, I agreed about traditional databases. Just wondering if Loki can (will can) deal with something else as storages.

[Bfoster-melrok]

I want to second the request for better documentation and examples.

I've read through this blog post about loki's scalability but cant any documentation on how to actually achieve this.
documentation around using s3 and dynamodb are great, but what about users using clusters not on aws? I'd love to use my own local object store and cassandra(this is mentioned in the blog post) for my storage if possible.
I know there is the "our production setup" Doc, but not everyone likes or uses ksonnet, and heptio isn't even working on it anymore since they were acquired by vmware.

[yubozhao]

hi @Bfoster-melrok I agree with more documentations. I am not very clear what documentations you are looking for. From what you wrote, I think you are looking for :

1. a deep dive of how Loki achieve scalability
2. non k8s setup for Loki/Promtail, and with different storage options
3. a different deployment setup? I assume for k8s.

Is that correct?

[Bfoster-melrok]

@yubozhao Almost,

1. an example of setting up a production loki cluster (including distributor/ingester/querier) inside kubernetes without ksonnet
2. documentation on promtail and loki config options (including different storage options)

[yubozhao]

For example setup without using ksonnet, you could use helm --dry-run to print out the raw YAML files. That should be a good starting point. You can also use helm as well.

Documentation for configuration is definitely lagging behind. ATM you would have to dig in the code and find what you want.

[Bfoster-melrok]

I have tried the helm chart and it only seems to be able to do a single loki instance setup with local storage

[Sniliev]

It would be nice to get a bit more information about how Loki scales. As far as I understand you are supposed to run multiple instances of Loki. Each of these instances taking on one(or possibly more) of the roles of distributor, ingestor or querier. However I cannot find how this should be achieved.

[rrfeng]

Need docs

[sciffer]

It's worse than that, Loki cluster requires also hashicorp consul cluster + an AWS storage (S3 or dynamodb). S3 can be emulated using multiple opensource solutions(like minio, edgefs, etc..) - but not really tested to the best of my knowledge. Anyhow - that means lots more overhead than should be!!
Docs are definitely missing, but it should be fairly simple to figure it out from the ksonnet docs if you are familiar with kubernetes manifest files.

[yubozhao]

If you don't have heavy workload, I would advice not use the consul to setup a multi instance Loki. As far as S3/Dynamo those stuff, refer to the upstream support. Those open source projects are not part of the support AFAIK.

[rrfeng]

Detailed docs help users try Loki easy and developers support the development.

[ratio91]

agree

especially docs on how to scrape logs from other docker containers (when promtail is running in docker itself) is required.
the sample config is not really working (no "container_names")
https://github.com/grafana/loki/blob/master/cmd/promtail/promtail-docker-config.yaml

[thirdeyenick]

I just looked through different issues here to learn more about how to deploy loki in a production environment as I could not find documentation. My current understanding is:

• the current helm chart deploys one single loki instance which does not seem to be production ready
• a production deploy means that all components (distributors, queriers, ingesters, (table-managers?)) are deployed separately (which can be done by adding the "-target" option to the loki command)
• consul is needed if distributors and ingesters are deployed separately
• as the queriers need access to the index DB something like googles bigtable or AWS DynamoDB is also a requirement
• this PR (https://github.com/grafana/loki/pull/713) seems to make it possible to scale even when using a local indexDB or chunk storage (consul seems to be still needed)

Are my findings correct so far?

[cyriltovena]

Yes we're going to make scaling on helm possible as we don't want you to have to handle this case:

a production deploy means that all components (distributors, queriers, ingesters, (table-managers?)) are deployed separately (which can be done by adding the "-target" option to the loki command)

Unless you're running at huge scale.

[thirdeyenick]

Thanks for the information. So the only additional system one needs is a consul cluster then.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.