Duckdb: Roadmap / Wish List

[ak2k]

Hi, Would you think a high-level list of items planned for development might be a good addition to the project?

From what I understood from Mark's CMU talk on DuckDB, these are some of the major milestones/themes underway or planned:

• Pipelined execution: https://github.com/cwida/duckdb/pull/690
• Storage (DataBlocks?) supporting compression, encoding

I would be curious if this is about right, even if you end up thinking a public roadmap isn't interesting.

Thanks!

[Mytherin]

Perhaps it would be a good idea to make it more public! I can detail at least my plans on what I'm personally going to work on next:

• Pipelined execution, followed by support for parallel execution of individual queries
• More testing/verifying of the codebase, particularly integrating SQLancer and running SQLSmith non-stop to pre-emptively catch more bugs
• Full outer join support
• HUGEINT/DECIMAL types

After that I am going to start working on optimizing the query execution engine, in particular optimizing the parallel join HT and aggregates, as well as optimizations surrounding string aggregates and subqueries.

But this is more my personal to-do list for now :) A more distant road-map would be:

• Storage rework, together with support for compression and encryption
• Support for incremental checkpoints and checkpointing during runtime
• Better cardinality estimation and improved join ordering, as well as usage of HyperLogLog sketches in optimizer
• Support for user-defined functions in languages like Python/etc
• Persistent index structures

Happy to hear suggestions as well :)

[Alex-Monahan]

Hey Folks!

We just pushed a DuckDB feature into git for our self-service analytics platform today, so I felt like now I could reach out with some questions about your roadmap! First of all, I want to recognize that you have built and are building a great product. We are seeing a 3-25x improvement over SQLite in our workloads - truly a game changer. Second of all, I really love a few items that are already on your roadmap: parallel execution, query optimization, and incremental checkpoints. I talked with my team and we had a few other thoughts!

Would you be open to supporting Postgres's JSON data types and operators? That would open the door to hybrid relational/object oriented use cases. Postgres's JSON aggregations also allow for a very flexible pivoting operation.

Postgres's grouping sets, cube, and rollup functions would be very useful for OLAP style workloads. Ex: "Give me all possible averages for these 5 levels of aggregation". They were added in Postgres 9.5, and searching through your copy of the parser, I believe it already has cases for them. My hope is that they would be a small add - I believe they are just a convenience wrapper around several group by's and union all's. Since our platform is self-service, any simplified syntax helps our user base!

Do you happen to already have plans to export to the Apache Arrow or Parquet format? We aren't ready to use that feature today, but we are headed in that direction.

Similarly, we are exploring options to scale beyond memory for data frame style operations and are considering the Dask and Vaex libraries (in addition to the great DuckDB Python API!). Have you already considered integrating with a larger-than-memory dataframe library in some way? The existing Pandas API could be used in some kind of chunking process, or maybe Parquet/Arrow would facilitate fast copying. The individual libraries might also support a more direct mapping.

Thanks!
-Alex

[hannesmuehleisen]

Would you be open to supporting Postgres's JSON data types and operators? That would open the door to hybrid relational/object oriented use cases. Postgres's JSON aggregations also allow for a very flexible pivoting operation.

It's hard to get JSON operations fast, what would be required is translation to our native LIST/STRUCT types. But that requires that the schema of the JSON values (?) is somewhat consistent within a column. Is this generally the case for you?

Postgres's grouping sets, cube, and rollup functions would be very useful for OLAP style workloads. Ex: "Give me all possible averages for these 5 levels of aggregation". They were added in Postgres 9.5, and searching through your copy of the parser, I believe it already has cases for them. My hope is that they would be a small add - I believe they are just a convenience wrapper around several group by's and union all's. Since our platform is self-service, any simplified syntax helps our user base!

I don't see why not, TPC-DS has them and we are currently using alternative query formulations to work around this.

Do you happen to already have plans to export to the Apache Arrow or Parquet format? We aren't ready to use that feature today, but we are headed in that direction.

Export to Parquet definitely, Arrow is also possible, see #151

Similarly, we are exploring options to scale beyond memory for data frame style operations and are considering the Dask and Vaex libraries (in addition to the great DuckDB Python API!). Have you already considered integrating with a larger-than-memory dataframe library in some way? The existing Pandas API could be used in some kind of chunking process, or maybe Parquet/Arrow would facilitate fast copying. The individual libraries might also support a more direct mapping.

We have not looked into that yet. Its a bit unclear what the benefits there would be.

[tenko]

Maybe off topic, but found this new project ugrep (BSD License) which seems to put a lot of work in to pattern matching, regex, optimizations and parallel operations. Not sure how complete duckdb is in this area, but could be of interest or hint of improvement directions.

[jshearer]

It's hard to get JSON operations fast, what would be required is translation to our native LIST/STRUCT types. But that requires that the schema of the JSON values (?) is somewhat consistent within a column. Is this generally the case for you?

@hannesmuehleisen Out of curiosity, what does your reference to LIST/STRUCT mean here? Is it possible to support JSON/document-style datatypes, as long as they have a schema? Are you imagining a solution where this structure is actually just flattened to a set of columns? How would arrays work? Is LIST related to Postgres' array column type?

[alanpaulkwan]

Feature request (sorry if they exist already), and informed entirely by my selfish desires :):

1. Working with IP addresses (ipv4/ipv6 to ip integer, vice versa), TLD extraction, ex TLD suffix extraction, post domain URL extraction.
2. Partitioned WRITES
3. Avro support (although the nested type issue is a thorn, I am sure)

Recently, I find myself with GCS cloud firing up pre-emptible instances. What makes DuckDB useful in the cloud context is especially is that DuckDB is self-contained and even easier than all the Apache stuff.

However, when firing up a pre-emptible instance, I mount a gcs bucket and am stuck with BigQuery's really stupid output formats, such as csv / json /Avro, so there is some pathway to conversion to Parquet. What makes Drill really useful is I can give it anything (Avro + json.gz + csv.gz) and it does a quick read job.