Victoriametrics: Data Corruption after power outage

[sh0rez]

Hi! After a datacenter crash related to a power outage, victoria is not able to come up anymore:

2019-06-11T14:49:39.922+0000    fatal   app/vmstorage/main.go:38
  cannot open a storage at victoria-metrics-data with retention period 120 months:
  cannot open table at "/victoria-metrics-data/data":
  cannot open partitions in the table "/victoria-metrics-data/data":
  cannot open partition "2019_06":
  cannot open small parts from "/victoria-metrics-data/data/small/2019_06":
  cannot open part "/victoria-metrics-data/data/small/2019_06/720_90_20190611054349.446_20190611054459.446_15A68A9DE06C8CA7": 
  cannot open timestamps file:
  open /victoria-metrics-data/data/small/2019_06/720_90_20190611054349.446_20190611054459.446_15A68A9DE06C8CA7/timestamps.bin:
  no such file or directory

Can I do anything about this?

[valyala]

It looks like VictoriaMetrics doesn't sync filenames inside created part directory before moving it from temporary path to the partition. This may result in disappeared files after power outage or system crash.
Will be fixed in the next release.

Temporary workaround: just remove the reported directory with missing file:

rm -rf /victoria-metrics-data/data/small/2019_06/720_90_20190611054349.446_20190611054459.446_15A68A9DE06C8CA7

This will result in the loss of 720 data points stored in this part.

[valyala]

The issue should be fixed in v1.19.1.

[sh0rez]

Thank you for looking into this that quickly :)