Materialize: perf: dropping a large index causes a loss of interactivity

[rjnn]

I created a large index, and then dropped it. while the drop was in progress, i could not make additional catalog queries to the index. Once the index drop is completed, the catalog queries are responsive again. This effect is viewable through \timing information:

materialize=> \timing
Timing is on.
materialize=> show indexes from principals;
  on_name   |        key_name        | seq_in_index | column_name | expression | nullable 
------------+------------------------+--------------+-------------+------------+----------
 principals | principals_primary_idx |            1 | tconst      |            | f
 principals | principals_primary_idx |            2 | ordering    |            | f
 principals | principals_primary_idx |            3 | nconst      |            | f
 principals | principals_primary_idx |            4 | category    |            | f
 principals | principals_primary_idx |            5 | job         |            | f
 principals | principals_primary_idx |            6 | characters  |            | f
 principals | principals_primary_idx |            7 | mz_line_no  |            | f
(7 rows)

Time: 32.079 ms
materialize=> drop index principals_primary_idx;
DROP INDEX
Time: 4.345 ms
materialize=> show indexes from principals;
 on_name | key_name | seq_in_index | column_name | expression | nullable 
---------+----------+--------------+-------------+------------+----------
(0 rows)

Time: 114170.406 ms (01:54.170)

[rjnn]

Surfacing (but not assigning) this to SQL folks who might know what's up @JLDLaughlin @sploiselle

[benesch]

I can say with 99% certainly that this has nothing to do with SQL. The DROP INDEX query returned instantly, which means SQL got out of the equation quite quickly.

It's impossible to say for sure without a profile, but I suspect that the dataflow layer correctly realized it could clean up the index, and then spent 2m freeing all the individual vecs in the index. That seems like a lot of time in free, but not implausible, and especially plausible on macOS. Speaking of which, was this on macOS?

[benesch]

/cc @frankmcsherry

[rjnn]

Yes, this was in macOS.

[benesch]

Ok, great. For whoever looks into this: the first step is probably to try to reproduce on Linux. I have a feeling things will be much healthier. That said, I'll bet there's still a non-trivial latency impact. Not sure what strategies we have at our disposal to address this.

[frankmcsherry]

We discussed something similar in the office hours today. One remedy courtesy of HN is to find a way to put the deallocation in another thread, which is a bit silly but returns the interactivity. I don't know what level of involvement is needed as everything is reference counted, and we kinda want someone to watch those and then take / move ownership once the count is low enough. All under the theory that deallocation is what is expensive.

Two minutes to deallocate a few GB sounds pretty crazy though, and is much worse than anything I've experienced myself.

[frankmcsherry]

Potentially related, I have a PR over in DD land that can flatten all of our Vec<Row> allocations down to a few Vec<u8> allocations, which should dramatically reduce the drop time at least. It has negative perf implications elsewhere, because it is a new copy that we didn't have to do when using ownership, but that is a potential remedy as well.

[rjnn]

Perhaps this is a stupid question, but independent of how long the drop takes, why does that dropping block catalog queries?

[benesch]

I'm writing up an answer to that elsewhere. Gimme 1s!

[frankmcsherry]

Many of the SQL catalog read queries (all?) are shipped out to the dataflow plane for execution, to make consistency and such easier.

[rjnn]

Then this sounds like the operator responsible for dropping the data is not being "nice", in the UNIX-sense. We have this recurring vulnerability, where because all operators in timely are cooperatively scheduled, any single operator can just gunk up the system by doing stuff for a good minute without yielding.

It appears that all of our allocations/deallocations need to support "niceness" - where they can pause and resume their memory work to deal with cooperative scheduling(?) That's a crazy ask. We'd have to write our own region-based allocator.

[benesch]

I've written up a good deal of context in the description of #1537, and linked this issue there as well. Hopefully that can help frame future issue reports about loss of interactivity.

One remedy courtesy of HN is to find a way to put the deallocation in another thread, which is a bit silly but returns the interactivity

At least in theory jemalloc's background threads should serve a similar purpose with no code changes on our end. We have those enabled on Linux. So we just need to see whether they make enough of a difference by running this experiment on Linux.

We'd have to write our own region-based allocator.

It's also kind of an open question how that would work in Rust. Standard library types don't presently support custom allocators.

[frankmcsherry]

We'd have to write our own region-based allocator.

That's what happened in here: https://github.com/TimelyDataflow/differential-dataflow/pull/281 It's probably something we want to consider eventually anyhow, and it's not all that hard, but at the moment the performance there is not wonderful, because other parts of the system are ownership-based and so the other allocator just ends up being more copying and allocation at the moment (but drops should be really fast, for what that's worth).