Timescaledb: Keeping Continuous Aggregation Query results after raw data removal

[Wilzi]

Hi,

I am looking to drop the raw data from a hypertable which I have a continuous aggregation query on.

I want to drop the raw data on the hypertable in order to decrease the database size due to the amount of incoming data and I would need to keep the continuous aggregation data to base results on.
I was wondering if there is a workaround for the problem described here?

Any guidance is appreciated.

[bboule]

Hey @Wilzi thanks for opening the issue, while I am not ready to commit to a time frame just yet for this, this is a priority for us as we plan our next set of releases. I can tell you you will get some relief in the short term when you release compression (1.5) and I can tell you the problem you are describing is something we are looking to tackle in the next few release cycles. We will update the milestone once we have had a chance to do some additional planning.

[xipascu]

@bboule Would you mind explaining what you mean by release compression and how we could do this?

[bboule]

@xipascu with version 1.5 of timescaleDB we will be offering native compression, you will be able to set policy that will apply compression of chunks based on age. This will give you some relief as it pertains to storage of the older data that you need to keep around for your continuous aggregates. We are looking at the feature you have requested (the ability to down sample via continuous aggs and drop the underlying data) I will update this thread once we have a reasonable expection we can set on this.

[karlyeurl]

Hello! Any updates on the expectations that can be set on that feature?

I am looking to store large amounts of high-frequency readings from many sensors, and continuous aggregations seem to be an outstanding way of setting an upper bound on the storage required for one variable's history, provided I can drop the old raw data. I really need fine-grained details of the readings for the current day, but I want to look at how the sensor behaved over the course of (say) the whole year (in which case only daily means are acceptable), or the last week (hourly means), etc.

As I am already using PG, TimescaleDB would be a very attractive option with this feature.

[bboule]

Hey guys, yes so this is a feature that is in active development. Our plan is to make this part of the next release of TimescaleDB (1.6), our current estimates have this release end of December-Early January timeframe. as we get a little further in the dev cycle I will be able to get a little tighter with my estimate.

Hopefully that is helpful!!

[karlyeurl]

Yes, that's very helpful information!

I will factor that in when I'll choose the system used for storing the time series.

[cevian]

Note that this has been implemented in #1589 and will be part of the next (1.6) release.