Loki: Unable to find any logs to tail makes pods unhelthy

[alikhil]

Describe the bug
Promtail respond with 500 Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc. if there are no logs to scrape which match scrape condition.

To Reproduce
Steps to reproduce the behavior:

1. Started loki (master-f2bec3b0)
2. Started promtail (master-f2bec3b0) to tail pods with label (monitor: spring-app)
3. Start deployment with number of pods less than number of nodes in your cluster
4. Check list of promtail pods and see that in nodes where is no deployment matching label selector promtail pods are failed to start

Expected behavior
All pods in all existing minion nodes will be up and running.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, promtail config, or terminal output
Logs from failing pod:

level=warn ts=2019-07-29T10:17:59.433772066Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511098591Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511380636Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511513947Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511627719Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=info ts=2019-07-29T10:17:59.512196582Z caller=kubernetes.go:192 component=discovery discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-07-29T10:17:59.514231267Z caller=server.go:120 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-07-29T10:17:59.514633991Z caller=main.go:55 msg="Starting Promtail" version="(version=master-f2bec3b0, branch=master, revision=f2bec3b0)"
level=warn ts=2019-07-29T10:18:10.670280612Z caller=logging.go:49 msg="GET /ready (500) 151.423µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "
level=warn ts=2019-07-29T10:18:20.669893429Z caller=logging.go:49 msg="GET /ready (500) 122.642µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "
level=warn ts=2019-07-29T10:18:30.670121286Z caller=logging.go:49 msg="GET /ready (500) 342.837µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "

Logs from running pod:

level=warn ts=2019-07-29T09:45:36.376487811Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.37702662Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377624682Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377787544Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377903092Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=info ts=2019-07-29T09:45:36.378411566Z caller=kubernetes.go:192 component=discovery discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-07-29T09:45:36.466269594Z caller=server.go:120 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-07-29T09:45:36.467184037Z caller=main.go:55 msg="Starting Promtail" version="(version=master-f2bec3b0, branch=master, revision=f2bec3b0)"
level=info ts=2019-07-29T09:45:41.668433219Z caller=filetargetmanager.go:257 msg="Adding target" key="{app=\"my-app\", category=\"backend\", container_name=\"my-app\", env=\"development\", instance=\"my-app-857d7c5d56-6vmsw\", job=\"development/my-app\", monitor=\"spring-app\", namespace=\"development\", pod_template_hash=\"4138371812\"}"
level=info ts=2019-07-29T09:45:41.669612293Z caller=tailer.go:73 msg="start tailing file" path=/var/log/pods/a44eb032-ad23-11e9-b35e-fa163ecb059f/my-app/0.log
2019/07/29 09:45:41 Seeked /var/log/pods/a44eb032-ad23-11e9-b35e-fa163ecb059f/my-app/0.log - &{Offset:0 Whence:0}

$ kubectl get pods 
promtail-4j9vx   1/1     Running   0          1h
promtail-6zfx2   1/1     Running   0          1h
promtail-f968n   1/1     Running   0          57m
promtail-fjk8c   0/1     Running   0          1m
promtail-fp8jc   1/1     Running   0          57m
promtail-h6vg6   1/1     Running   0          1h
promtail-qjjzf   0/1     Running   0          1m
promtail-ztktd   1/1     Running   0          1h

[steven-sheehy]

That's expected. You need to fix your scrape config to be able to scrape at least one pod. At minimum, it should be able to scrape itself.

[alikhil]

@steven-sheehy what if I don't want to scrape logs from all the pods in the cluster and scrape only pods of selected apps (which can be deployed only on some nodes, not all)? Seems like such a usecase is missed now.

I think it can be solved by passing some flag like IGNORE_NO_LOGS.

[pennpeng]

+1

[geekodour]

@alikhil
these two links might be useful to get you scrape_config right, you can select what you want to scrape using relabel configs:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

[alikhil]

@geekodour thank you! Could you tell me what is mean to be right for scrape_config so that promtail will not fail if there is nothing to scrape?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[alikhil]

Do not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[DonnieBwin]

hello it is that been solved?

[HuJake]

I have the same problem, how can I solve it？

NAME                 READY   STATUS    RESTARTS   AGE
test-loki-0           1/1     Running   0          5h49m
test-promtail-bd9h6   0/1     Running   0          3m42s
test-promtail-bpfsc   0/1     Running   0          3m42s
test-promtail-chrcv   0/1     Running   0          3m42s
test-promtail-dq2jw   0/1     Running   0          3m42s
test-promtail-j4sxg   0/1     Running   0          3m42s

caller=logging.go:49 msg="GET /ready (500) 40.895µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.14; "

I found entering the pod for observation,
found positions.yaml:
filename: /run/positions.yaml is empty
Is there a problem?

[slim-bean]

Generally we would suggest at least tailing the promtail log to avoid this issue.

If you don't want to do this I suggest just removing the readiness probe entirely.

[slim-bean]

For anyone who was following this ticket #1920 introduced a flag to essentially always make the /ready endpoint return 200

[imranrazakhan]

@slim-bean In my case docker log driver was set as journald and i changed it to "log-driver": "json-file" and it start working.