Clickhouse: What is the reasoning getMaxSourcePartsSize implementation
Hi, I recently stumbled on a lagging replica failure caused by a "live lock" between threads trying to find suitable merges. I'm still collecting data to be able to replicate the issue.
At the same time I'm trying to build a mental model about how this whole merge selector (work assigning) by master and merge execution by replicas fit together.
What I'm trying to understand at the moment, is why getMaxSourcePartsSize is implemented the way it is.
Why bigger merges are preferred when thread pool is empty is smaller when thread pool is busy?
nvartolomei
All 4 comments
I would also like this better documented. Struggling to keep our active part count under 200 for our raw data tables. Each insert is 60k rows in a batch at ~800B/row uncompressed (text data). Our volume is ~150k rows/s spread across 9 nodes.
abraithwaite
on 17 Jul 2019
@abraithwaite default server settings? 60k sounds like a small batch size, I had great experience with batch size > 1M.
nvartolomei
on 17 Jul 2019
What I'm trying to understand at the moment, is why getMaxSourcePartsSize is implemented the way it is.
Why bigger merges are preferred when thread pool is empty is smaller when thread pool is busy?
We have fixed number of slots in background processing pool and we don't want for long running merges to occupy all the slots. We want to allow smaller merges to run, otherwise the number of small (just inserted) parts will grow up quickly.
For this purpose, we implement simple heuristic. When more than half of slots in background pool is occupied, lower the limit on max source part size by dividing it by some number after every next slot is occupied.
alexey-milovidov
on 15 Apr 2020
FWIW, for those coming to this issue, we also increased our insertion batch size to 150k events, and moved from inserting into a distributed table, to inserting directly into the shards. This relieves a lot of the load since inserting into a distributed table re-shards the data again across your number of shards given the sharding key.
abraithwaite
on 16 Apr 2020
Related issues
jangorecki
路
3Comments
vixa2012
路
3Comments
opavader
路
3Comments
bseng
路
3Comments
goranc
路
3Comments
Most helpful comment
We have fixed number of slots in background processing pool and we don't want for long running merges to occupy all the slots. We want to allow smaller merges to run, otherwise the number of small (just inserted) parts will grow up quickly.
For this purpose, we implement simple heuristic. When more than half of slots in background pool is occupied, lower the limit on max source part size by dividing it by some number after every next slot is occupied.