Victoriametrics: Everyday retention or fractional retention period

[svesh]

When retention period is 1 month, VM stores data up to 2 months.
At the end of month there is too much disk space used.

Everyday retention or fractional retention period can solve the problem.

[valyala]

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17

[tenmozes]

can we update the format and support strings like "1d", "1m", "1y"? Default measurement is month for backward compatibility, or just introduce new field :)

[valyala]

can we update the format and support strings like "1d", "1m", "1y"?

Sure, this looks like good solution. VictoriaMetrics must check whether the provided retention is multiple of months until fractional retention is implemented.

or just introduce new field

I'm against introducing new field, since this will bring more confusion during setup and operation.

[gseddon]

This improvement would be great for non-prod environments, where we want the same software configuration but do not have huge disk capacity or need very long retention periods.

[svesh]

We collect 1.5 TB metrics per month. We need freeze 3 TB drive space now. This improvement would be great for big systems

[FlorinAndrei]

@svesh You may want to monitor this feature request:

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/342

[valyala]

FYI, the following commits add support for -retentionPeriod smaller than one month:

• for single-node VictoriaMetrics - 5bfd4e62188f74606b463aab3135beb17a8456c8
• for cluster VictoriaMetrics - 0db7c2b500940d278c7c71bb7b0dd6ec9b097b33

These commit allow to set:

• fractional -retentionPeriod, i.e. -retentionPeriod=0.5 sets retention to a half of month
• -retentionPeriod with suffixes such as d (days), w (weeks) or y (years). For example, -retentionPeriod=5d would set retention period to 5 days.

See more details about retention in the docs.

These commits can be tested right now by building VictoriaMetrics from sources according to the following docs:

• how to build single-node VictoriaMetrics
• how to build cluster version of VictoriaMetrics.

[vpedosyuk]

@valyala great news, thanks! I just wanted to make sure whether this sentence from the docs is not valid anymore:

In order to keep data according to -retentionPeriod max disk space usage is going to be -retentionPeriod + 1 month

and we can have now -retentionPeriod + 1 day of disks space?

[weisdd]

@valyala Thanks for implementing the feature!
In the documentation, I couldn't find an update for the forced merge endpoint. Will the format of the partition prefix remain the same? (/internal/force_merge?partition_prefix=YYYY_MM) Or it's something like YYYY_MM_DD now?

[valyala]

I just wanted to make sure whether this sentence from the docs is not valid anymore:

In order to keep data according to -retentionPeriod max disk space usage is going to be -retentionPeriod + 1 month

and we can have now -retentionPeriod + 1 day of disks space?

There is no hard limit on the amount of data that can be stored outside the configured retention. The amount of such data depends on the following factors:

• The configured retention. Smaller retention (i.e. less than a few days) usually result in faster data removal outside the configured retention.
• Data ingestion rate. Bigger data ingestion rate usually increases the frequency of background merges, which increases chances of data removal outside the configured retention.

In the documentation, I couldn't find an update for the forced merge endpoint. Will the format of the partition prefix remain the same?

The format for /internal_force_merge?partition_prefix=YYYY_MM remains the same, since VictoriaMetrics continues storing data in per-month partitions. If -retentionPeriod is set to value smaller than one month, then the data with timestamps outside the retention is gradually deleted during background merge process.

FYI, retention smaller than 1 month is supported starting from v1.45.0. Closing this feature request as done.