Delta: feature request: hidden partitioning / timestamp partitioning

[changpingc]

We store a lot of time series data using delta format. They all have a timestamp column and are partitioned by date(timestamp).

Two common mistakes I see are
(1) forgetting to include a date filter in addition to timestamp filter
(2) partitioning output by timestamp leading to millions of output folders

The problem goes away if delta supports something like hidden partitioning in Iceberg, or specifying timestamp partition granularity (daily vs hourly) in BigQuery.

https://iceberg.apache.org/partitioning/#icebergs-hidden-partitioning
https://cloud.google.com/bigquery/docs/creating-column-partitions#daily_partitioning_vs_hourly_partitioning

[brkyvz]

Yes, this would be a fantastic addition to Delta!

[gwax]

This would be very useful for geospatial data indexing with H3. If it were possible to use a truncated (lower resolution) index as a partition but query against the full index without having to manually truncate we could accelerate geospatial queries while hiding partition details under the hood.

[rubenssoto]

Is there any plans to support this in delta?

[brkyvz]

yes, work is starting this week! Will be a long process though - it's a big project to get right