Victoriametrics: Create a tool for migrating Prometheus data from InfluxDB to VictoriaMetrics

[valyala]

The issue

It is non-trivial to migrate Prometheus data from InfluxDB to VictoriaMetrics, when InfluxDB was set up as a remote storage for Prometheus.

The solution

Create a tool similar to outflux for seamless migration of Prometheus data from Influx to VictoriaMetrics.

[JogiInnov]

Totally agree.
I was working on huge data import from Influx to Victoriametrics and found many difficulties. It would be much appreciated if we can have tool like Outflux.

[valyala]

A temporary workaround is to export data from Influx with influx_inspect and then import it into VM.

[hagen1778]

Hi! There is a naive implementation of migrating tool for InluxDB here - https://github.com/VictoriaMetrics/vmctl. Would really appreciate any feedback!

[valyala]

See also this comment containing shell script for data migration from InfluxDB to VictoriaMetrics.

[ahmadalli]

unfortunately vmctl's influx import does not support prometheus data. check out https://github.com/VictoriaMetrics/vmctl/issues/8

[hagen1778]

Hi @ahmadalli! The issue mentions only metric names conversion like removing _value suffix before importing. It is just nice to have feature but it doesn't stop you from migrating from Influx to VM. Is it critical in your case?

[ahmadalli]

@hagen1778 what I understand is prometheus data in influx has different format so if I import them from influx to VM they won't work out of the box compared to importing prometheus tsdb snapshots

[hagen1778]

Not exactly. When we import data from Influx to VM some data mapping happens - see https://github.com/VictoriaMetrics/vmctl#data-mapping
And author of the issue that you referenced propose to change the mapping if we know in advance that metrics in Influx were Prometheus metrics before. This change will only affect metric and tag names.
What happens now if you import data from Influx:

foo,tag1=value1,tag2=value2 field=12 => foo_field{tag1="value1", tag2="value2"} 12

What author of the issue proposes to do if we know in advance that we import Prometheus metrics:

foo,tag1=value1,tag2=value2 field=12 => foo{tag1="value1", tag2="value2"} 12 // suffix field was dropped

Anyway, my advice is to try to import small fraction of your data in Influx.

[ahmadalli]

the problem is that I'll loose backward compatibility with my current metrics and my dashboards.

about https://github.com/VictoriaMetrics/vmctl/issues/8, I prefer the solution that the author suggested: adding a flag for specifying that it's prometheus data database. I wanted to implement it myself but got lost in the code (unfortunately I haven't written any go code)

[hagen1778]

I see. Thanks for details! I'll see if we can prioritise that FR. Would you mind to put a comment/emotion in that FR so we can better understand users interest?

[ahmadalli]

sure. thank you for your efforts ;)