Victoriametrics: Object Storage

[perriea]

Is it expected that the project will support object storage natively like Thanos or Cortex (in futur) ?
Object storage like S3 or GCS.

[valyala]

Thanks for the idea, @perriea ! Could you provide more context where this may be necessary?

Probably it is possible to point VictoriaMetrics (via -storageDataPath) to a directory mounted from S3 or GCS via FUSE wrappers such as gcsfuse or s3fs-fuse. Could you try such setup and report results here?

[MaxDiOrio]

Thanos allows storing of metric data directly in S3 or GCS using Minio.

I'd love to see snapshots be able to be written directly to S3 or GCS. One step backup and truncation of data.

[perriea]

Yes, exactly @MaxDiOrio that's what I'm looking for because I'd like to test this project in a Kubernetes or with (high) scalability context without having to do stateful.
However I will test what I recommended @valyala although I know very well that the system fuse is not crazy on the performances.

[valyala]

I'd love to see snapshots be able to be written directly to S3 or GCS. One step backup and truncation of data.

Unfortunately this isn't easy as it sounds. Automatic uploading of snapshots to GCS and S3 may be interrupted due to various failures:

• network connectivity issues
• intermittent object store maintenance / outage

The probability of errors increases with the size of snapshot files. VictoriaMetrics may produce multi-TB files in snapshots, which result in very high probability of errors during automatic snapshot uploading. How to handle such errors? There are various approaches for errors handling and uploading strategies with different pros & cons:

• Just retry file upload from the beginning. This may be huge waste of time for multi-TB files.
• Resume the upload from the failed place if this is supported by the object store.
• Split big files in smaller chunks and upload them separately with per-chunk retries on errors. During the recovery merge these chunks into the original files.
• Archive the snapshot into a single file, then split it into moderately sized chunks and upload them in parallel to object store in order to improve upload performance.
• Perform incremental backup, i.e. upload only new files and delete missing files from the object store. This is possible because of LSM tree used by VictoriaMetrics.
• Your approach.

There are other approaches exist. Each approach has certain users, but there is no "silver bullet" approach for snapshot uploading, which would suit all the users. So it is highly unlikely VictoriaMetrics will automatically upload snapshots to object store in the future.

[valyala]

Yes, exactly @MaxDiOrio that's what I'm looking for because I'd like to test this project in a Kubernetes or with (high) scalability context without having to do stateful.

It is better to use persistent volumes backed by replicated durable disks for Kubernetes such as Google Compute Engine persistent disks - see these docs for details. VictoriaMetrics is optimized for HDD-backed persistent disks, so there is no much sense in using more expensive SSD-backed persistent disks.

[antoniobeyah]

Agreed with this request. Using persistent volumes within Kubernetes in AWS typically defaults to using EBS which is tied to an availability zone which is a challenge when you have a k8s cluster that spans multiple availability zones. You cannot use the same EBS volume across availability zones so it creates issues when your cluster is in an autoscaling group.

[MaxDiOrio]

This is what multipart S3 uploads help solve. I do understand the challenge though. I'd picture doing an automated monthly backup, which should keep the size more manageable.

[MaxDiOrio]

Maybe do a daily snapshot/archive to cloud storage for small file size with a monthly purge? Definitely check how Thanos is doing this. They are storing and querying directly to cloud storage. Clearly it's not going to be as performant as local storage but it would be a good option.

It could also be a good option for those of us with on-prem slow storage that we can use with minio as object storage. Still local and fast but not on our primary, expensive ssd storage.

It would be great to archive off but still be able to query the archived data like Thanos.

If this can be done without the complexity of Thanos that would be amazing. I spent a full week trying to get the Thanos and sidecar pieces set up but had too many issues. That drove me to Victoria Metrics as a probable solution.

[valyala]

It shouldn't be too hard to store VictoriaMetrics' data in object storage if the storage supports the following operations:
1) Streaming writing to a file without limits on file size. Resume streaming writing after network error must be supported.
2) Fast concurrent reading of arbitrary parts of the file.

The main issue with object storage is network errors. They reduce reliability and availability comparing to much more reliable block storage such as GCE disks, EBS or local disks. Currently VictoriaMetrics assumes that it works with reliable storage (aka filesystem), so it reports the first occurred error related to storage and exits instead of trying to recover from such errors (it is unclear how to recover from these errors). When working with object storage, such errors must be handled gracefully, since they are usually expected in object storage world. This is one of the reasons why Thanos has not so good reliability comparing to VictoriaMetrics.

[valyala]

There is yet another approach for adding ability to use Object Storage as a store for VictoriaMetrics data: to write a thin layer between file API and Object Storage API, which will split big files into smaller chunks and upload them to Object Storage as separate files. This should alleviate network connectivity issues outlined above and will improve upload retrying speed.

[perriea]

It seems like a good idea. How do you see this feature in VictoriaMetrics ? Native functionality ? Plugin ?
I ask because there are different protocols (S3, GCS, Azure Blob storage for the main ones).

[valyala]

There should be native functionality implemented for each supported protocol. Probably https://github.com/minio/minio-go could simplify the integration.

[MaxDiOrio]

Yes...native is probably the way to go.

[Serrvosky]

That solution could be awesome and a massive improve to Victoria Metrics. Can bring a must have feature with the simple approach like Victoria Metrics got us used.

It would be nice to have a flag like timeToSnapshot, where Vitoria Metric's storage nodes can read an interval of time to make a snapshots and send it to the cloud, leaving nothing more that a proper logs .
If something happen there are a lot of log's aggregation and processing tools (like datadog, elastiksearch, loki+grafana) where users can check error logs.

[valyala]

FYI, there is vmbackup tool for full and incremental backups of VictoriaMetrics data on S3 and GCS. The tool will be included into the next VictoriaMetrics release. Now it can be built from sources according to these instructions.

It would be nice to have a flag like timeToSnapshot, where Vitoria Metric's storage nodes can read an interval of time to make a snapshots and send it to the cloud

It should be easy to write a small script around vmbackup tool, which would perform this. See this use case.

[perriea]

Very cool ! We will be able to put a cron job in Kubernetes to backup the TSDB 👍

[valyala]

Published an article about vmbackup tool - https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883

[valyala]

https://objectivefs.com/ looks promising for transparently storing VictoriaMetrics data on object storage such as S3, GCS or various s3-compatible on-prem systems.