Loki: watcher.Add: no such file or directory

[eskornev]

Hello, I'm starting promtail from this template:
https://github.com/moov-io/infra/blob/master/lib/infra/20-promtail.yml

and getting errors for every pod like:
level=error ts=2018-12-19T10:12:57.022052858Z caller=targetmanager.go:145 msg="Failed to create target" key="{instance="weave-net-wcjhh", job="kube-system/weave-net", namespace="kube-system"}" error="watcher.Add: no such file or directory"

file /var/log/positions.yaml in promtail container consists of empty json:
positions: {}

how can i fix it?

[erdii]

same here with Grafana Cloud and

curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh \
  | sh -s 1395 <Your Grafana.com API Key> logs-us-west1.grafana.net default \
  | kubectl apply --namespace=default -f  -

Edit: readability

[adamdecaf]

Did you change any volumes or mount points from the example config?

[adamdecaf]

Oh, does there need to be an os.Stat check before this block?

https://github.com/grafana/loki/blob/9b67d6cb7a6ffacbd2f504598b5107319f071d1c/pkg/promtail/target.go#L56-L59

[eskornev]

No they are the same:
...
volumeMounts:
- mountPath: /etc/promtail
name: promtail
- mountPath: /var/log
name: varlog
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
...
volumes:
- configMap:
name: promtail
name: promtail
- hostPath:
path: /var/log
name: varlog
- hostPath:
path: /var/lib/docker/containers
name: varlibdockercontainers

[woodsaj]

@eskornev what version of kubernetes are you using? If it is 1.9.X or older then you will need to update your scape config so that the __path__ label is set using

       - replacement: /var/log/pods/$1/
         source_labels:
         - __meta_kubernetes_pod_uid
         target_label: __path__

[erdii]

I'm using K8s 1.9.6 and @woodsaj 's fix worked. Thank you!

I replaced

- replacement: /var/log/pods/$1
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid
  - __meta_kubernetes_pod_container_name

with

- replacement: /var/log/pods/$1
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid

twice. Now everything is working as intended!

[eskornev]

oh I knew trouble is in config...
it woks, thanks a lot

[adamdecaf]

Can we close this? Should we document what Kubernetes versions are supported?

[kentore82]

same issue for me on k8s v.1.13.0 and using helm charts as is.
"Failed to create target" key="{app=\"ibm-keepalived-watcher\", controller_revision_hash=\"3280303120\", instance=\"ibm-keepalived-watcher-bws4c\", job=\"kube-system/ibm-keepalived-watcher\", namespace=\"kube-system\", pod_template_generation=\"1\"}" error="watcher.Add: no such file or directory"

[kfox1111]

just deployed a fresh cluster using newest minikube, helm installed the helm chart and then a stable/mysql. I'm getting the same kind of things in the logs.

[kfox1111]

hmm... then it seemed to start working. seems like a race on startup maybe.