Loki: Kubernetes - Promtail link log file

Created on 29 May 2019 · 13Comments · Source: grafana/loki

Hello everybody,

I'm trying to set up promtail in my kubernetes cluster but it's not being easy.
I'm using DaemonSet method, but when I'm checking logs in promtail pod, I can find directories to each pod, but the link to logs file gives me "No such file or directorie".

This is my promtail config:

But when I try to see 0.log file
(this is inside promtail's pod)

What am I doing wrong?

Thanks a lot for your help

Source

Serrvosky

All 13 comments

I'm encountering same issue, why did you close the issue @Serrvosky

chinglinwen on 25 Jul 2019

Hello @chinglinwen.

In my case I forgot to mount host logs path. Promtail creates a link to this folder. Please check if you have the following mounts that are between << >>.

volumeMounts: - mountPath: /etc/promtail name: promtail - mountPath: /var/log name: varlog << - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true >> .... volumes: - configMap: name: promtail-conf name: promtail - hostPath: path: /var/log name: varlog << - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers >>

Serrvosky on 25 Jul 2019

Thanks for your comments.

I do have those volume mounted, is it caused by sym link to different mount place? ( I don't know how to walk around this ).

Do you have any idea about this @Serrvosky

[root@kube-test-81-104 merged]# ll /var/log/pods/t_netshoot-64b954898d-kk6wp_08f0d38a-aeac-11e9-a106-6e97fdc29d10/netshoot/0.log
lrwxrwxrwx 1 root root 162 Jul 25 15:15 /var/log/pods/t_netshoot-64b954898d-kk6wp_08f0d38a-aeac-11e9-a106-6e97fdc29d10/netshoot/0.log -> /data/docker/containers/45507088d1142926c8bea18b927c4d54e9040ea2047e2ba282491b3aeda67866/45507088d1142926c8bea18b927c4d54e9040ea2047e2ba282491b3aeda67866-json.log

the DaemonSet yamls

# Source: loki-stack/charts/promtail/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: loki-promtail
  namespace: loki
  labels:
    app: promtail
    chart: promtail-0.8.1
    release: loki
    heritage: Tiller
  annotations:
    {}

spec:
  selector:
    matchLabels:
      app: promtail
      release: loki
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: promtail
        release: loki          
      annotations:
        checksum/config: 9fcce10088976d3285adc263029a9239dd68d8e56b1ada78771b4d643ccf46cc
        prometheus.io/port: http-metrics
        prometheus.io/scrape: "true"

    spec:
      hostNetwork: true
      serviceAccountName: loki-promtail
      containers:
        - name: promtail
          image: "grafana/promtail:v0.1.0"
          imagePullPolicy: IfNotPresent
          args:
            - "-config.file=/etc/promtail/promtail.yaml"
            # - "-client.url=http://loki:3100/api/prom/push"
            - "-client.url=http://10.101.165.203:3100/api/prom/push"
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: run
              mountPath: /run/promtail
            - mountPath: /var/lib/docker/containers
              name: docker
              readOnly: true
            - mountPath: /var/log/pods
              name: pods
              readOnly: true

          env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          ports:
            - containerPort: 3101
              name: http-metrics
              hostPort: 3101
          securityContext:
            readOnlyRootFilesystem: true
            runAsGroup: 0
            runAsUser: 0

          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

          resources:
            {}

      nodeSelector:
        {}

      affinity:
        {}

      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master

      volumes:
        - name: config
          configMap:
            name: loki-promtail
        - name: run
          hostPath:
            path: /run/promtail
        - hostPath:
            path: /data/docker/containers
          name: docker
        - hostPath:
            path: /var/log/pods
          name: pods

chinglinwen on 25 Jul 2019

Which version of Kubernetes that are you using?

Serrvosky on 25 Jul 2019

kubernetes v1.14
CentOS Linux release 7.4.1708 (Core)

chinglinwen on 25 Jul 2019

Can you please check you RBAC's?

apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: promtail
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: promtail
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: promtail
subjects:
- kind: ServiceAccount
  name: promtail
  namespace: logging

Serrvosky on 25 Jul 2019

I checked, it's the same with your clips, only namespaces is different.

# Source: loki-stack/charts/promtail/templates/clusterrole.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    app: promtail
    chart: promtail-0.8.1
    release: loki
    heritage: Tiller
  name: loki-promtail-clusterrole
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "watch", "list"]

# Source: loki-stack/charts/promtail/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: promtail
    chart: promtail-0.8.1
    heritage: Tiller
    release: loki
  name: loki-promtail
  namespace: loki

for the full promtail yamls

https://pastebin.com/jbv1YVun

chinglinwen on 25 Jul 2019

Humm. i don't understand whats happeing too.
If you wanna try my promtail configs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-conf
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml

    scrape_configs:
      - pipeline_stages:
        - docker:
        job_name: kubernetes-pods-name
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_pod_label_name
          target_label: __service__
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __service__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: job
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
      - pipeline_stages:
        - docker:
        job_name: kubernetes-pods-app
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: drop
          regex: .+
          source_labels:
          - __meta_kubernetes_pod_label_name
        - source_labels:
          - __meta_kubernetes_pod_label_app
          target_label: __service__
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __service__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: job
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
      - pipeline_stages:
        - docker:
        job_name: kubernetes-pods-direct-controllers
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: drop
          regex: .+
          separator: ''
          source_labels:
          - __meta_kubernetes_pod_label_name
          - __meta_kubernetes_pod_label_app
        - action: drop
          regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$
          source_labels:
          - __meta_kubernetes_pod_controller_name
        - source_labels:
          - __meta_kubernetes_pod_controller_name
          target_label: __service__
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __service__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: job
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
      - pipeline_stages:
        - docker:
        job_name: kubernetes-pods-indirect-controller
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: drop
          regex: .+
          separator: ''
          source_labels:
          - __meta_kubernetes_pod_label_name
          - __meta_kubernetes_pod_label_app
        - action: keep
          regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$
          source_labels:
          - __meta_kubernetes_pod_controller_name
        - action: replace
          regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$
          source_labels:
          - __meta_kubernetes_pod_controller_name
          target_label: __service__
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __service__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: job
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
      - pipeline_stages:
        - docker:
        job_name: kubernetes-pods-static
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: drop
          regex: ^$
          source_labels:
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_label_component
          target_label: __service__
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: drop
          regex: ^$
          source_labels:
          - __service__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: job
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: instance
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container_name
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
          - __meta_kubernetes_pod_container_name
          target_label: __path__

Just to see if it makes any diference.

Can you please also check promtail's pod logs?

Serrvosky on 25 Jul 2019

Hi, I've make it works, I mount the volume ( second time ) as the same path as hosts too, so the sym link works now.

Thank you for your help

            - mountPath: /var/lib/docker/containers
              name: docker
              readOnly: true
            - mountPath: /data/docker/containers
              name: docker
              readOnly: true

Can I asked you another question?

Through this link, It says I can define label

https://github.com/grafana/loki/blob/master/docs/promtail-setup.md

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: netshoot
  namespace: t
  labels:
    app: netshoot
    logFileName: my_app_log

I find out that label logFileName doesn't propagate

By looking at the code, I can only see __path__ decide the final log path.

Do you have any idea how can I scrap logs inside the container ( say path: /logs/app/*.log ).

chinglinwen on 25 Jul 2019

👍1

Sorry, but no.

Can't you change the logs destiny to stdout? If yes, they will be on host logs path, and promtail can scrape them without any problem.

Serrvosky on 25 Jul 2019

I was reading that link. To do that, don't forget to update promtail scrape configs with this source label: __meta_kubernetes_pod_label_logFileName

And I can't find any mention to that in your files (https://pastebin.com/jbv1YVun)

Serrvosky on 25 Jul 2019

Wow, very thanks. It helps. I've missed that part.

chinglinwen on 25 Jul 2019

👍1

Awesome community work here, if you had trouble in the past don't hesitate to contribute to our troubleshooting section to help others that may have the same issue.

✌️