Loki: `--client.external-labels=hostname=host1` does not work

[mizeng]

Describe the bug
looks like ClientConfig and ClientConfigs are mis-used for promtail Config. The Command line configs including ExternalLabels are in ClientConfig, however Promtail only used ClientConfigs.

This will cause something like ./promtail --config.file=./docker-config.yaml --client.external-labels=hostname=host1 does not work. The external-labels is missing.

To Reproduce
Steps to reproduce the behavior:

1. Started loki (SHA or version)
2. Started promtail (SHA or version) to tail ./promtail --config.file=./docker-config.yaml --client.external-labels=hostname=host1
3. Then the label "hostname=host1" is not showing up in Loki.

Expected behavior
The label "hostname=host1" should show up in Loki.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
• Deployment tool: [e.g., helm, jsonnet]

Screenshots, promtail config, or terminal output
If applicable, add any output to help explain your problem.

[mizeng]

@daixiang0 someone just suggest me to use exiternal-labels in https://github.com/grafana/loki/issues/634. So what should I use for that use case (want to quickly add different host labels for multiple hosts)?

[daixiang0]

Just give a quick ask:

scrape_configs:
- job_name: system
  entry_parser: raw 
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      hostname: host1

Do not think a good way to show different hostname in local mode, but in k8s, you can use __meta_kubernetes_node_name refer to doc, so that you can add below scrape config:

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: hostname

But i saw in the latest scrape confg:

      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        separator: ''
        source_labels:
        - __meta_kubernetes_pod_label_name
        - __meta_kubernetes_pod_label_app
      - action: drop
        regex: ^([0-9a-z-.]+)(-[0-9a-f]{8,10})$
        source_labels:
        - __meta_kubernetes_pod_controller_name
      - source_labels:
        - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__

__host__ is you want, right?

[mizeng]

@daixiang0 This requirement is mainly from non k8s env. Say, if I need to install promtail in lots of VMs, and I need to have their hostname as one of the labels or default labels. I checked the code, hostname's already in s.hostname (https://github.com/grafana/loki/blob/master/pkg/promtail/targets/filetargetmanager.go#L213), just not add into labels like filename. So I want to add it into labels.

However slim-bean suggested me to use external_labels to fulfill my original requirements, which I found it's not working at all. Moreover, after debug the code, I found the external-config command line args seems not working at all. It will be put into promtail.config.ClientConfig, and promtail usually only will use promtail.config.ClientConfigs (https://github.com/grafana/loki/blob/master/pkg/promtail/promtail.go#L33).

[daixiang0]

scrape_configs:
- job_name: system
  entry_parser: raw 
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      hostname: host1

This config can work for local mode, please note that since you run at local, for each vm you need to run a promtail. need to dig to find the issue about external_labels.

[mizeng]

I know this can work, but not efficient in deploying for different hostname for different vms. need something like hostname:${HOSTNAME} in promtail config file or promtail --external-labels=hostname=${HOSTNAME} in promtail startup command line.

[cyriltovena]

@mizeng it will work if you use the single client config.

Either by providing a client: in the config or via command line. If you only use the clients: it won't work.

We can't parse a set of label set via command line, so using a single client if your only option.

[mizeng]

@Kuqd could you share one example with both command line start up and the promtail config?

[daixiang0]

@mizeng the clients config is as below:

clients:
  - url: url/to/a
    foo: a
  - url: url/to/b
    foo: b

you can set in config, that works for me.

We can't parse common flag via command line if specify configfile at same time.

[mizeng]

@daixiang0 then how can I fulfill the requirement? paas variable to label? https://github.com/grafana/loki/issues/634

[mizeng]

never mind. Finally I figure it out.
Remove "Clients" in the promtail config, and use below comand to start promtail:

./promtail --client.url=http://xxxx:3100/api/prom/push --client.external-labels=hostname=${hostname} --config.file=./docker-config.yaml

[minhdanh]

__host__ is you want, right?

@daixiang0 I need to query the logs by the hostname (with something like {hostname="gke-pool-1-xxxx"}, but __host__ is an internal label that is not usable in Loki. So how can that be done? Do I need another relabel_config for that?

[daixiang0]

@minhdanh just need add --client.external-labels=hostname=${hostname} then do query. If still question, better to open a new issue to track it.

[sebek72]

@minhdanh just need add --client.external-labels=hostname=${hostname} then do query. If still question, better to open a new issue to track it.

Does not work in 1.5.0 (hostname or HOSTNAME are empty).

[GJaminon]

This worked for me with v1.5:

• name: "Create the PromTail DeamonSet"
  k8s:
  host: "{{ k8s_url }}"
  kubeconfig: "{{ kubeconfig_file }}"
  wait: true
  definition:
  apiVersion: apps/v1
  kind: DaemonSet
  metadata:
  name: promtail
  namespace: monitoring
  spec:
  selector:
  matchLabels:
  name: promtail
  template:
  metadata:
  labels:
  name: promtail
  spec:
  serviceAccount: promtail-serviceaccount
  serviceAccountName: promtail-serviceaccount
  volumes:
  - name: logs
  hostPath:
  path: /var/log
  - name: promtail-config
  configMap:
  name: "{{ config_map_result.result.metadata.name }}"
  containers:
  - name: promtail-container
  image: grafana/promtail:1.5.0
  args:
  - -config.file=/etc/promtail/promtail.yaml
  - -client.url=http://loki:3100/loki/api/v1/push
  - -client.external-labels=hostname=$(NODE_NAME)
  env:
  - name: NODE_NAME
  valueFrom:
  fieldRef:
  fieldPath: spec.nodeName
  volumeMounts:
  - name: logs
  mountPath: /var/log
  - name: promtail-config
  mountPath: /etc/promtail

Config:

• set_fact:
  promtail_config: |
  server:
  http_listen_port: 9080
  grpc_listen_port: 0
  positions:
  filename: /tmp/positions.yaml
  scrape_configs:
  - job_name: system
  entry_parser: raw
  static_configs:
  - targets:
  - localhost
  labels:
  job: varlogs
  __path__: /var/log/*.log