Victoriametrics: vmagent panic with valid RelabelConfig

[ihard]

Describe the bug
vmagent panic with valid RelabelConfig, in log:

nfig/relabel-config.yml": error when parsing `relabel_config` #1: missing `target_label` for `action=replace`
panic: FATAL: cannot load relabel configs from -remoteWrite.urlRelabelConfig="/config/relabel-config.yml": error when parsing `relabel_config` #1: missing `target_label` for `action=replace`

goroutine 1 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logMessage(0xab156b, 0x5, 0xc000540180, 0xb7, 0x4)
    github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:178 +0xa65
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevelSkipframes(0x1, 0xab156b, 0x5, 0xad4b21, 0x4c, 0xc00051dd70, 0x2, 0x2)
    github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:111 +0xe1
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.logLevel(...)
    github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:103
github.com/VictoriaMetrics/VictoriaMetrics/lib/logger.Panicf(...)
    github.com/VictoriaMetrics/VictoriaMetrics/lib/logger/logger.go:99
github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite.newRemoteWriteCtx(0x7ffeefb62f17, 0x15, 0x7ffeefb62f4b, 0x1a, 0xc8, 0xc00012f750, 0xc, 0xc00012f6b8)
    github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:150 +0x91d
github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite.Init()
    github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:72 +0x220
main.main()
    github.com/VictoriaMetrics/VictoriaMetrics/app/vmagent/main.go:56 +0x102

To Reproduce
run vmagent with config:

- job_name: test
  relabel_configs:
    - action: drop
      source_labels: [test1]

Expected behavior
Don't panic ;)

Version
docker victoriametrics/vmagent:v1.34.2-cluster

[valyala]

The provided relabel config isn't suited for passing to -remoteWrite.relabelConfig. It should contain only a list of relabel configs, i.e.:

- action: drop
  source_labels: [test1]

Added stricter syntax checking in 986dba5ab3c22408b7f1874b4f6324d415609106 in order to reduce the confusion in the future.

[ihard]

How use metric_relabel_configs ?

[valyala]

metric_relabel_configs can be used only inside Prometheus config, which is passed to -promscrape.config. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config for details.

Both -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig accept paths to files with relabeling rules only, i.e. the file contents should look like
:

- action: drop
  source_labels: [test1]
...
- action: drop
  source_labels: [testN]

All the rules from -remoteWrite.relabelConfig are applied to all the metrics collected by vmagent before sending them to all the configured -remoteWrite.url destinations.

All the rules from -remoteWrite.urlRelabelConfig are applied to all the metrics before sending them to the corresponding positional -remoteWrite.url destination.

For instance, the following command line flags -remoteWrite.url=http://vm1 -remoteWrite.urlRelabelConfig=relabel_for_vm1.yml -remoteWrite.url=http://vm2 -remoteWrite.urlRelabelConfig=relabel_for_vm2.yml apply rules from relabel_for_vm1.yml to metrics sent to vm1, while relabel_for_vm2.yml are applied to metrics sent to vm2.

[valyala]

The v1.34.3 release has better error handling for this issue.

[ihard]

OK, thanks for your help

[ihard]

I can 't delete a separate label in vmagent?
i send metric from telegraf -> vmagent (influxdb protocol)
and get additional tag db=telegraf - how delete one label from metric for this case?

[valyala]

The label can be deleted with one of the following relabeling rules:

- regex: db
  action: labeldrop

or

- target_label: db
  replacement: ""

See this article for details.

[valyala]

Closing this issue as resolved.

@ihard , feel free re-opening it if you think the issue requires additional actions.