Loki: Kubernetes 1.14 CRI directory change

[steven-sheehy]

Is your feature request related to a problem? Please describe.
Kubernetes 1.14 is changing the CRI log directory to include the namespace and pod name.

Describe the solution you'd like
Loki should support the new format as well as the old format out of the box.

Describe alternatives you've considered
Each user customizes promtail configmap

Additional context
See https://github.com/kubernetes/kubernetes/issues/73503

[akhenakh]

Since minikube ships with 1.14 it's becoming a more urgent issue

[davkal]

Until we patch the helm chart, users running CRI need to adapt their relabel rules in the promtail config.
The old path pattern /var/log/pods/$1/*.log no longer works.

I've seen success in changing all occurrences of

        - replacement: /var/log/pods/$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__

to

       - replacement: /var/log/pods/*$1*/*/*.log
          source_labels:
          - __meta_kubernetes_pod_uid
          target_label: __path__

[daixiang0]

@davkal refer to comment, could it be better to add a new scrape config?

[steven-sheehy]

I've confirmed with a developer from Google that GKE switched to the new log directory format even for older Kubernetes versions before 1.14.