Steal Impala's Parquet reader
hannesmuehleisen
All 16 comments
We'd like to have some parquet reader after all. Only thing stopping us now is the dependency zoo it brings with it. But we might able to work around that. Best candidate so far is Arrow's reader.
hannesmuehleisen
on 13 Jun 2019
New battle plan: Ship minified thrift runtime libraries and roll our own reader. Makes arrow dep unnecessary #151
hannesmuehleisen
on 31 Jul 2019
@hannesmuehleisen Why build your own reader and not use Arrow? Very curious about that.
xhochy
on 31 Jul 2019
@xhochy binary size and transitive build/runtime dependencies (e.g. thrift, boost, ...). Plus we can directly exploit e.g. string dicts and bit packing when scanning from DuckDB. But happy to discuss further
hannesmuehleisen
on 31 Jul 2019
As one of the main authors of parquet-cpp (and thus cpp/src/parquet in the Arrow project). I would like to see it used. I guess you will always need a basic set of dependencies for building a Parquet reader and one should not underestimate the complexity of the format, there is a lot of work in that reader.
I'm guessing here but you want to have Parquet reading as a static library that is included in the R packages?
xhochy
on 31 Jul 2019
@xhochy I built DuckDB with a parquet reader last weekend and my build times increased horrendously. Arrow + Parquet comes with dependencies to all the compression libraries (Snappy, Brotli, LZ4, Gzip, ...), double-conversion, boost (!), thrift + (rapidjson and flatbuffers if you keep Arrow IPC).
Right now, DuckDB has almost no dependencies and the very few were copied into the source tree. So this has to be done very carefully if want to preserve the simple embedding into arbitrary projects. You easily end up with something like the ThirdpartyToolchain.cmake of arrow otherwise.
(Which is not bad but still significantly more complex than the current CMakeLists)
ankoh
on 31 Jul 2019
@ankoh You will need at least Thrift and all the compression libraries except gzip for Parquet support. But what in general is so bad about dependencies? You can install them via your package manager and then you have them and don't need to write the functionality you get through them by yourself.
xhochy
on 31 Jul 2019
Dependencies are not such a problem when writing a stand-alone application, but when writing a library they become a different matter entirely. Not only do you need to install all your dependencies, everyone that uses your library needs to install all your dependencies. Then everyone that uses another program or library that uses your library also needs to install all your dependencies.
While "installing dependencies" might seem trivial, it is often not trivial at all.
- What if I want to install your project when I don't have root on my box? Now I can't use "apt-get install" to install the dependency and I need to compile it from source. Now what if you have 5 dependencies that each have 5 dependencies themselves? I need to compile 25 projects from source just to get your code to run.
- What if one of the dependencies breaks because a new version has an API change or a behavior change?
- What if one of the dependencies is not supported on a specific machine architecture, e.g. on an ARM box?
- What about Windows? There is no package manager there.
All of these problems are very real, and become much more likely the more dependencies you include. Especially as library authors we need to be extremely careful to include dependencies because they just make everything more difficult not only for all of your users, but for all the users of any project down the line that depends on your project.
For these reasons we have a very strict no external dependencies policy. Any dependencies we include we inline into the project and make sure that are minimal and work everywhere that our project also runs. We also only use dependencies when we absolutely need them and when they themselves are "clean" (e.g. don't have external dependencies or ship a lot of extra code that we will not use).
Some reading material on dependencies:
Mytherin
on 31 Jul 2019
As a developer of Arrow, I have to deal with quite a lot of dependencies and I can understand that the more you get, the more complicated the setup gets. I'm quite happy with having C++ development environments using conda as a non-root users but "For these reasons we have a very strict no external dependencies policy." is sadly such a bold statement that I don't see any reason to continue interacting here.
xhochy
on 31 Jul 2019
What I mean by "no external dependencies" is that we do not want to rely on the user having anything installed on their system. Our entire code base needs to be able to compile by itself on all the supported platforms (OSX, Linux, Windows). I don't see how that is a bold statement. We are more than happy to use other people's code but that code needs to either have no external dependencies or needs to be adopted to have no external dependencies.
Mytherin
on 31 Jul 2019
I actually like the idea of duckdb and thus I’m trying to get this back to the productive side. From what I understand is your goal to build a minimal, embeddable analytical database. To keep it minimal, you also want to keep the requirements minimal. To support Parquet as a format, you will need to either implement it on your own and thus require Thrift, gzip, zstandard, and snappy as dependencies and spent quite some time on implementing the format or depend on the arrow sources and have boost as an additional dependency (actually the time to get rid of boost in the Parquet part of the Arrow is much smaller than reimplementing the whole of Parquet).
Given that the set of dependencies will never be none, what about adding a simple file reader interface to duckdb? Then you could keep duckdb as comprehensive as it is currently but people could provide readers for many file formats (Parquet, ORC, Arrow, Feather, ..). I guess the main interfaces will be CSV, R and Pandas for most users, so these should be in the core.
Otherwise a simple interface f(list of files, predicate) -> ColumnBatches where predicate is used for pre-filtering the data. This would enable things like predicate pushdown if the format allows it, otherwise the filtering will then be done completely in duckdb. Then a Parquet plugin could be built separately and wouldn’t introduce any additional dependencies to duckdb-core. This won’t let you work on the run-length encoded data but still would allow the exploitation of (string) dictionaries in the Parquet format.
As the interface standard, Arrow as a columnar memory format would be a natural fit. As a library for just a data interface, I understand that arrow-cpp is too heavy but here I’m actually also looking for a better solution like making a really minimal Arrow C interface with no dependencies that is just used for interchanging data with other libraries that understand Arrow.
—
I’m happy to work with you on this topic but a no-dependency policy for every feature will lead to a lot of work on your side. Thus it’s better to contain (like containment) these dependencies to where they are need and not expose them to all users. Parquet is quite heavy on those but that is due to the nature of the format. Also be aware that according to your classifications, I would not depend on Thrift, it has proven to be quite a hassle in the past with changing versions and on some systems it still requires boost.
Also I understand that from a performance perspective, it is compelling to make an implementation as close as possible to the data in the format but that also is where most of the work lies. I cannot stop you doing this but highly recommend first doing the plugin approach as this is a lot less work for you.
—
Answering to some of your questions from earlier as I have spent an awful amount on packaging in the last 5 years in various platforms. If the plugin interface sounds ok to you, it’s a bit unrelated but wanted to share this as it might be helpful.
- What if I want to install your project when I don't have root on my box? Now I can't use "apt-get install" to install the dependency and I need to compile it from source. Now what if you have 5 dependencies that each have 5 dependencies themselves? I need to compile 25 projects from source just to get your code to run.
- Use conda as an environment manager, this is what we do as Arrow developers. You get everything as binary with a decent modern compiler on all three major systems. If you want to build everything with custom compiler flags, Gentoo’s emerge can also be used for an environment and doesn’t need to be the system dependency manager.
- What if one of the dependencies breaks because a new version has an API change or a behavior change?
- You update it and either drop support for the old version or add ifdefs, this is quite harsh but sadly the reality.
- What if one of the dependencies is not supported on a specific machine architecture, e.g. on an ARM box?
- Arrow builds on x86, ARM64 and PPC64le; there is no big endian support but that is negligible nowadays.
- What about Windows? There is no package manager there.
- There is a wide selection, as mentioned above conda works quite well but I know that the KDE people use emerge (not the Gentoo one but a custom one made by the KDE people), the R folks a large repository of ming-based packages and there are several others available.
xhochy
on 1 Aug 2019
I think having a generic interface for supporting external formats is indeed a great idea and something we should definitely implement. We are not opposed to having external dependencies outside of the core. We already have some actually, as we have a NumPy dependency in the Python package for example.
We should then only think about how we can cleanly link provide and link in these optional plugins to the users that want them. Likely loading them from an external shared library or so would be my initial idea for that.
Mytherin
on 1 Aug 2019
Is there an alternate path where duckdb learns to parse Apache Arrow formatted memory buffers, and the application can bring its own choice of libraries to do serialization to parquet?
b11z
on 24 Aug 2019
We are building a C interface to Arrow (not only the C++ implementation but all Arrow implementations) to access "Arrow formatteded memory buffers": https://github.com/apache/arrow/pull/5442
@hannesmuehleisen @Mytherin Would be nice if you could have a look whether this would suit DuckDB for a generic interface to the Arrow world.
xhochy
on 9 Oct 2019
@xhochy thanks, will have a look. Great to see the non-dependency case is being looked into
hannesmuehleisen
on 9 Oct 2019
We now support reading parquet files, see #653
hannesmuehleisen
on 5 Jun 2020
Related issues
GuillaumePressiat
·
6Comments
Mytherin
·
5Comments
pdet
·
5Comments
wizzard0
·
6Comments
waynelapierre
·
9Comments
Most helpful comment
@xhochy I built DuckDB with a parquet reader last weekend and my build times increased horrendously. Arrow + Parquet comes with dependencies to all the compression libraries (Snappy, Brotli, LZ4, Gzip, ...), double-conversion, boost (!), thrift + (rapidjson and flatbuffers if you keep Arrow IPC).
Right now, DuckDB has almost no dependencies and the very few were copied into the source tree. So this has to be done very carefully if want to preserve the simple embedding into arbitrary projects. You easily end up with something like the ThirdpartyToolchain.cmake of arrow otherwise.
(Which is not bad but still significantly more complex than the current CMakeLists)