Loki: Grafana always report "Error connecting to datasource: Loki: Bad Gateway. 502"

[xlloveyun]

Describe the bug

Loki start but Grafana always report Error 502.

I tried to build and run from source like below:
$ go get github.com/grafana/loki
$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.
$ go build ./cmd/loki
$ ./loki -config.file=./cmd/loki/loki-local-config.yaml

And the loki logs like below

level=info ts=2019-05-21T10:47:00.558441195Z caller=loki.go:122 msg=initialising module=server
level=info ts=2019-05-21T10:47:00.558530158Z caller=server.go:120 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-05-21T10:47:00.558610078Z caller=loki.go:122 msg=initialising module=overrides
level=info ts=2019-05-21T10:47:00.55861863Z caller=override.go:48 msg="per-tenant overides disabled"
level=info ts=2019-05-21T10:47:00.55862615Z caller=loki.go:122 msg=initialising module=store
level=info ts=2019-05-21T10:47:00.558707817Z caller=loki.go:122 msg=initialising module=ring
level=info ts=2019-05-21T10:47:00.558732354Z caller=loki.go:122 msg=initialising module=querier
level=info ts=2019-05-21T10:47:00.559169799Z caller=loki.go:122 msg=initialising module=ingester
level=info ts=2019-05-21T10:47:00.55925738Z caller=loki.go:122 msg=initialising module=distributor
level=info ts=2019-05-21T10:47:00.559289168Z caller=loki.go:122 msg=initialising module=all
level=info ts=2019-05-21T10:47:00.559351413Z caller=main.go:66 msg="Starting Loki" version="(version=, branch=, revision=)"
level=error ts=2019-05-21T10:47:00.559049891Z caller=consul_client.go:182 msg="error getting path" key=ring err=null
level=info ts=2019-05-21T10:47:00.559428549Z caller=lifecycler.go:373 msg="entry not found in ring, adding with no tokens"
level=info ts=2019-05-21T10:47:00.559723319Z caller=lifecycler.go:303 msg="auto-joining cluster after timeout"

Then I tried to add http://localhost:3100 to grafana, then got

Error connecting to datasource: Loki: Bad Gateway. 502

Environment:

Mac, debian and centos, Grafana in docker, and loki is build from source. I don't know what happend, I've checked the 3100 port has been opened, and iptables is stopped.

Expected behavior

Grafana return success when click the Save&Test button.

[tmikaeld]

@xlloveyun Did you find out what the problem was? I have the same issue...

[cfstyle]

I have the same issue...

[tmikaeld]

@cfstyle You can connect to the IP of the host, like http://155.16.4.68:3100

[cfstyle]

@cfstyle You can connect to the IP of the host, like http://155.16.4.68:3100

thanks. now it's working fine ( grafana can connect the loki and I can explore logs).
But the error "404 page not found" is still there.

[tmikaeld]

@cfstyle Well, the promtail endpoint will only return 404 if you're not doing a query or posting data to it. For example, posting to http://155.16.4.68:3100/api/prom/push would work fine.

[kush2100]

Error connecting to data source: Loki: Bad Gateway. 502
Grafana and Loki are working on same server. Now i want to configure Loki Datasource on public IP like http://1.1.1.1:3100, it will give error Loki: Bad Gateway. 502.
But when connect with local host it will work like http://localhost:3100 or http://127.0.0.1:3100.
Even when i try with http://localhost:3100 get error 404 page not found
Can you please help me on this.

[Gooooodman]

Did you find out what the problem was? I have the same issue
why the issue is closed????
where is the solution ???

[cyriltovena]

Solution is make sure loki is healthy and properly configured in Grafana

[Gooooodman]

a matter of time

[hamiltont]

Grafana in docker

That's your problem. To your docker container, localhost means "inside the same container". Loki is not running inside the container, hence the 502 error.

If you've named the loki container, you can take advantage of Docker's DNS and use this as your datasource URL http://loki:3100 - docker will map loki (or whatever your container name is) to the proper IP for the container running loki. This works best if you're using docker-compose

If loki is running not in a container, but on your host, you can use Dockers host.docker.internal domain as so http://host.docker.internal:3100. This will auto-map to the IP address of the host machine (as of docker 18.03+)

[paaragon]

Grafana in docker

That's your problem. To your docker container, localhost means "inside the same container". Loki is not running inside the container, hence the 502 error.

If you've named the loki container, you can take advantage of Docker's DNS and use this as your datasource URL http://loki:3100 - docker will map loki (or whatever your container name is) to the proper IP for the container running loki. This works best if you're using docker-compose

If loki is running not in a container, but on your host, you can use Dockers host.docker.internal domain as so http://host.docker.internal:3100. This will auto-map to the IP address of the host machine (as of docker 18.03+)

This works perfect for me