materialize startup memory usage is higher than we'd like

[ruchirK]

seen it go up to 500 mb (on mac with 8 threads)

[ruchirK]

seems like num threads is a big driver of memory usage and its stable with just one thread

[frankmcsherry]

One thing worth validating is that it does grow without bound, vs monotonically increasing but perhaps asymptoting. The single-threaded behavior grew monotonically for a while, but did eventually enter a semi-stable state (there is churn, because we have views over logging streams that are effectively logging themselves).

[frankmcsherry]

I ran materialized overnight (something like four CPU hours when I checked this morning) and it was at 316MB, which I think means "stabilized".

Our best guess at the moment is that this may be the previously optional bootstrap views, though investigation has not concluded anything specific yet.

[rjnn]

If we make the bootstrap views unmaterialized, shouldn't this go away?

[frankmcsherry]

Recent user concern about the "start-up CPU cost" that multiple cores are grinding away even without any work installed. Almost certainly related.

[frankmcsherry]

Quantitative data from a "start-up and watch" on elapsed time for installed dataflows:

id |name                                                              |dataflow_id|elapsed_ms|
---|------------------------------------------------------------------|-----------|----------|
724|Dataflow: mz_catalog.mz_perf_peek_durations_aggregates_primary_idx|        724|      1984|
567|Dataflow: mz_catalog.mz_perf_peek_durations_bucket_primary_idx    |        567|      4354|
520|Dataflow: mz_catalog.mz_perf_peek_durations_core_primary_idx      |        520|      1700|
485|Dataflow: mz_catalog.mz_perf_arrangement_records_primary_idx      |        485|      7626|
375|Dataflow: mz_catalog.mz_perf_dependency_frontiers_primary_idx     |        375|     18897|
355|Dataflow: mz_catalog.mz_records_per_dataflow_global_primary_idx   |        355|      3487|
316|Dataflow: mz_catalog.mz_records_per_dataflow_primary_idx          |        316|     12276|
287|Dataflow: mz_catalog.mz_records_per_dataflow_operator_primary_idx |        287|      7574|
239|Dataflow: mz_catalog.mz_dataflow_operator_dataflows_primary_idx   |        239|      3332|
197|Dataflow: mz_catalog.mz_dataflow_names_primary_idx                |        197|      3043|
167|Dataflow: mz_catalog.mz_addresses_with_unit_length_primary_idx    |        167|      1685|

Many seconds worth of work being done maintaining these dataflows (over what I was led to believe was a relatively short about of time, corresponding to 300%-400% utilization to do so).

[frankmcsherry]

If we make the bootstrap views unmaterialized, shouldn't this go away?

I think so. That strikes me as a tasteful approach (views are installed by default, but not materialized). We'll need to check on what this does to catalog compat, in that their materializations are named things in the catalog. Also, there is the quirk at the moment that users cannot delete system indexes (and .. may not be able to create system indexes, though that is harder to test without doing the change first).

[benesch]

We've explicitly decided that we can break your cluster for the time being if you create any views that depend on catalog views, so should be fine! You should get a "nice" error message when you upgrade. Nice in that it will tell you what went wrong, but won't help you fix the problem.

[frankmcsherry]

We should double check if this still applies, now that the logging views are not materialized.

[ruchirK]

Started up a new materialized instance on my laptop with 8 threads, will check back on it in half an hour or so initial impression is that its growing slowly (currently at 128 mb) but its not growing nearly as fast as it was when I first filed this issue

[ruchirK]

after about 30 minutes materialize w/ 8 workers was at ~ 180 MB. I think this issue is "better, but not fully resolved"?

[frankmcsherry]

I'm going to nominate for "fixed" by #4158. @ruchirK feel free to re-open, but my read is that we are on the same page about this!