Timescaledb: Allow INSERTS/UPDATES/DELETES to compressed chunks

[shane-axiom]

Creating this issue to track a feature request for allowing INSERTS, UPDATES, and DELETES to compressed chunks within hypertables (i.e. performing those operations on hypertables with compressed chunks using data with times within the ranges of compressed chunks). Apologies if this is already covered by another issue, but I scanned for a while and didn't find one.

The current limitations are described in the docs and a blog post, which also mentions:

We also plan to weaken/remove this limitation in future releases.

I'm creating this issue mostly in hopes of hearing from the devs about the feasability and/or timeline of implementing this functionality. If you're interested in this as a user, please just +1 instead of comment spamming unless you have a relevant use case to describe.

Our use case is collecting long time series from many devices outputting observations at various frequency. The vast majority of data ingest is "current" (now-ish) data, and access patterns are primarily wide aggregations of current conditions or full time series extraction of a single device; this makes using compression on older tables very desirable.

However, we are also continuously adding new devices with existing historical time series to the data store, and certain devices periodically need older data deleted and replaced with quality controlled data. Currently this would require manual intervention, either by manually decompressing chunks, inserting data, and recompressing (which is complicated and requires temporary usage of larger disk space) or running the backfill script, which I haven't yet tested but seems like it isn't aware of secondary/space partitioning columns (i.e. it would decompress all chunks corresponding to a time range rather than specific chunks matching the hash of the secondary partitioning column).

I'll note that our use case can tolerate somewhat sub-optimal performance for INSERTS/UPDATES/DELETES on compressed chunks. In other words, we can wait a bit longer these operations to complete as long as they work as expected without manual intervention.

[shane-axiom]

One more note, the backfill script seems to rely on the usage of compression policies, which aren't yet supported on distributed hypertables.

https://github.com/timescale/timescaledb/issues/1900

[mfreed]

Thanks for this tracking issue, @shane-axiom. Team indeed plans to mitigate some of these limitations.

As a start, we've recently merged a PR that allows one to ALTER the schema of a compressed hypertable (by supporting ability to add a column). This will likely be released in v2.1. https://github.com/timescale/timescaledb/pull/2736

[gtskaushik]

Eagerly waiting for this functionality

[q00Dree]

That would be nice. Current situation is kinda uncomfortable with backfilling.