Influxdb: Feature request: "thinning" data retention policy

Created on 13 Mar 2016 · 6Comments · Source: influxdata/influxdb

Hi!

I would like to propose a data retention policy for thinning older data.

Suppose I have sensor data coming in every 5 seconds. Having this data in this resolution is very practical for now and the last few month (maybe). But for historical data it is unnecessarily detailed. Using a retention policy now it would delete older data, but I would like to see historical data.

So I Propose a data retention policy (perhaps in unison with an aggregation) to thin older data. Perhaps keep sensor data for every 1h, or the mean/sum etc. for 1h and thus make it fewer data points and use less space.

Have a nice weekend!

Source

TheOneValen

Most helpful comment

This is what I need, a type of continuous downsample on the same series, not a separate series.

Something like:
Now until 30 days, 1 second resolution
30 days until 60 days, 30 second resolution
60 days until 90 days, 1 min resolution
etc etc

Would be nice if it can be configurable, the ranges, and the downsample resolution.

linuxbutler on 21 Mar 2016

👍3

All 6 comments

Look at continous querry for your porpuse.
It's already implemented

max3163 on 14 Mar 2016

👍1

Continuous Query is sadly not exactly what is required here; if you use a CQ you then cant do things that you really have to be able to do - graph multiple 'time periods' on one Grafana graph, or simply have the time picker work in a Grafana dashboard.

I'd call the feature we would like a 'Continuous downsample'. This would be similar to a contiguous query, but work in-place with the same series (it may work by writing out a new series, then deleting the old one, but from the POV of a user its in-place)

daviesalex on 16 Mar 2016

👍1

This is what I need, a type of continuous downsample on the same series, not a separate series.

Something like:
Now until 30 days, 1 second resolution
30 days until 60 days, 30 second resolution
60 days until 90 days, 1 min resolution
etc etc

Would be nice if it can be configurable, the ranges, and the downsample resolution.

linuxbutler on 21 Mar 2016

👍3

Example from @linuxbutler is exactly what I meant. Additionally I have massive performance problems using continuous querys on a Raspberry Pi. Continuous querys also mean an additional measurement to manage in my application as well as in Grafana.

TheOneValen on 22 Mar 2016

Hi,
Before starting to implement this feature I would like to know the status of this in ?
Is anybody already working on this issue ?