Victoriametrics: Issues with moving collectd from Graphite to VM

[RafhaanShah]

I'm trying to switch over my collectd daemon to store it's data in Victoria Metrics instead of my current Graphite back-end but I'm having some issues.

VM options:

--storageDataPath=/storage"
--graphiteListenAddr=:2003"
--opentsdbListenAddr=:4242"
--httpListenAddr=:8428"

First I tried using the graphiteListenAddr endpoint of VM, and on Grafana the metric names show up in the query editor suggestions but no values are returned, here's the result from the query inspector:

Object
request:Object
url:"https://grafana.server/api/datasources/proxy/2/api/v1/query_range?query=collectd.host.curl_xml-instance.gauge-CPU_clock&start=1587234480&end=1587234780&step=15"
    method:"GET"
    body:undefined
  response:Object
    status:"success"
      data:Object
        resultType:"matrix"
          result:Array[0]

Grafana screenshot
And the collectd config and docs

collectd config

<Plugin write_graphite>
    <Node "node">
        Host "victoriametrics"
        Port "2003"
        Protocol "udp"
        Prefix "collectd."
    </Node>
</Plugin>

Next I tried using the opentsdbListenAddr, which resulted in VM logs having messages about missing tag values.

victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put \"curl_xml.instance.gauge.Memory usage\" 1587233920 1346.996 fqdn=host  test=abc": cannot unmarshal tags in "\"curl_xml.instance.gauge.Memory usage\" 1587233920 1346.996 fqdn=host  test=abc": missing tag value for "1346.996"
victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put \"curl_xml.instance.gauge.GPU usage\" 1587233920 19 fqdn=host  test=abc": cannot unmarshal tags in "\"curl_xml.instance.gauge.GPU usage\" 1587233920 19 fqdn=host  test=abc": missing tag value for "19"
victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put \"curl_xml.instance.gauge.GPU temperature\" 1587233920 33 fqdn=host  test=abc": cannot unmarshal tags in "\"curl_xml.instance.gauge.GPU temperature\" 1587233920 33 fqdn=host  test=abc": missing tag value for "33"
victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put \"curl_xml.instance.gauge.Fan speed\" 1587233920 0 fqdn=host  test=abc": cannot unmarshal tags in "\"curl_xml.instance.gauge.Fan speed\" 1587233920 0 fqdn=host  test=abc": missing tag value for "0"
victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put \"curl_xml.instance.gauge.Fan speed 2\" 1587233920 0 fqdn=host  test=abc": cannot unmarshal tags in "\"curl_xml.instance.gauge.Fan speed 2\" 1587233920 0 fqdn=host  test=abc": missing tag value for "1587233920"
victoriametrics    | 2020-04-18T18:18:39.763Z   error   VictoriaMetrics/lib/protoparser/opentsdb/parser.go:127  cannot unmarshal OpenTSDB line "put curl_xml.instance.gauge.Power 1587233920 23 fqdn=host  test=abc": cannot unmarshal tags in "curl_xml.instance.gauge.Power 1587233920 23 fqdn=host  test=abc": missing tag value for ""

And the collectd config and docs

collectd config

<Plugin write_tsdb>
    ResolveInterval 60
    ResolveJitter 60
    <Node "node">
        Host "victoriametrics"
        Port "4242"
        HostTags "test=abc" (tried without this too)
    </Node>
 </Plugin>

Any ideas?

[valyala]

VictoriaMetrics expects valid PromQL or MetricsQL queries sent to /api/v1/query_range and /api/v1/query endpoints. So it parses the query collectd.host.curl_xml-instance.gauge-CPU_clock into the following expression: collectd.host.curl_xml - instance.gauge - CPU_clock, i.e. it tries subtracting instance.gauge and CPU_clock from collectd.host.curl_xml. There are the following solutions for the issue:

• to escape special chars such as - with backslash, i.e. to use collectd.host.curl_xml\-instance.gauge\-CPU_clock query.
• to put the metric name into double quotes: {__name__="collectd.host.curl_xml-instance.gauge-CPU_clock"}.

See PromQL tutorial for more information about PromQL.

As for the errors when trying to send data over OpenTSDB telnet put protocol, it looks like write_tsdb plugin for collectd violates OpenTSDB telnet put protocol specs by sending quoted metric name with spaces. and by putting multiple spaces in front of test=abc. The specs say that metric name may not contain spaces and that it shouldn't be quoted.

[RafhaanShah]

Still a beginner at PromQL so didn't know about escaping the query but it worked perfectly, thanks so much!