Duckdb: Optional threads

[ankoh]

TL;DR: We need to build DuckDB without threads.

We are compiling DuckDB to WASM as in-browser accelerator for analytics dashboards.
That works fantastic performance-wise but currently suffers from a significant problem.
Threads require SharedArrayBuffers in browsers which were one of the first targets of Spectre and Meltdown mitigations.
Chrome reenabled SharedArrayBuffers a while ago but Firefox, Chrome on Android and Safari on iOS either disable it entirely or constrain it too much.

We are currently expanding our platform support and definitely want to ship DuckDB to Safari on iOS which effectively leaves us with only one option.
We cannot depend on SharedArrayBuffers and thus have to compile DuckDB without threads.

I know that this heavily affects the design around intra-query parallelism but are optional threads anything that you would consider maintaining?

[Mytherin]

I think supporting a version without threads is not much of a problem, DuckDB is not reliant on background threads at all (and by default doesn't use them currently, you need to manually enable them with PRAGMA threads=X). They are only used if present to speed up query execution. I think I can whip up a version without threads relatively fast.

[hannesmuehleisen]

@ankoh this is super exciting!

[ankoh]

Thanks for the quick response.
I'm primarily fighting the wasm-ld at the moment since it keeps complaining about missing symbols originating from libpthread.
E.g. sem_timedwait and such.
It feels rather insane to just ignore those and keep going but it might very well be an issue with our emscripten setup.
I'll check the PR, thank you.

[Mytherin]

I believe that originates from the concurrent queue we are using. I can strip that one out too if it is giving you problems.

[hannesmuehleisen]

We also should integrate the emscripten build toolchain into our CI once it works so we can ship the results as well. Also we then notice when we break it ^^

[ankoh]

@hannesmuehleisen This would be amazing but is related to #310. We are currently mapping duckdb::DataChunks to flatbuffers. We would totally merge that upstream but I'm not entirely sure whether you are a fan of including flatbuffers in here. Especially since its large enough to not do that in-tree.

The alternative would be, that we just maintain duckdb.wasm with the opinionated flatbuffer API in front ourselves.
(Opinionated because you want to design the WASM api slightly differently compared to traditional FFIs since it's much easier to keep the data ownership on the WASM side when your user is javascript)

We planned to release this under MIT with our tool anyway so then you could still decide whether it's worth the hassle.

[ankoh]

@Mytherin wasm-ld is still complaining about sem_timedwait. But our tests actually pass now when silencing this error.
It still feels very hacky but I guess we will revisit this when things start to break.

[ankoh]

The issue really is moodycamel::details::Semaphore::Semaphore.

[Mytherin]

I have just pushed a change stripping the concurrent queue if threads are disabled (it is not required in that scenario anyway). Take a look if that fixes the problem.

@hannesmuehleisen This would be amazing but is related to #310. We are currently mapping duckdb::DataChunks to flatbuffers. We would totally merge that upstream but I'm not entirely sure whether you are a fan of including flatbuffers in here. Especially since its large enough to not do that in-tree.

I think we should totally run it in our CI. There is no doubt this will break and be a headache otherwise. Can't we fetch the flatbuffers code from elsewhere for the CI build (e.g. from npm)? We are only hardcore anti-dependency for the core of DuckDB - tools and extensions can have dependencies no problem :)

Very nice logo btw :)

[Alex-Monahan]

I truly believe that the browser is in desperate need of a performant SQL database and DuckDB will be a huge hit! If this could be a part of the standard build process, that would be awesome!

[ankoh]

I totally agree.
But maybe just give us another 2 months or so to develop some trust in the abstraction.
If the API is somewhat stable on our end and we don't need to touch it so often, I'm totally down to packing that PR.

[ankoh]

@Mytherin thanks! we needed this on top https://github.com/dashql/duckdb/commit/7b4cdd3bebf2384f4a30dedaf609562a3f7e3ad2 but the linker no longer complains about the queue.

[Mytherin]

Whoops, could have sworn that compiled. Thanks, I have pushed these changes there now as well.

[hannesmuehleisen]

@ankoh could you share the build script please so we can put it into CI?