Materialize: Support for persistence / recovery / availability

[frankmcsherry]

~Customers~ Prospects have requested a better story around "persistence", which we've taken to mean "what happens when the system goes down; can it come back up as if it had not gone down, or will it recover in a meaningfully different state?"

At the moment, I think we have access to several "good stories" about recovery, mostly of the form "we produce correct answers". What we don't have are convincing stories around performance and interoperation. We are 100% relying on other source-of-truth systems to provide data to us, with the fallback plan that we re-load from them in the worst case. Right now "any crash" is the worst case.

We should be able to more clearly specify our requirements of source-of-truth systems, as well as our performance burden on these systems (at the moment, I think a full scan to refresh a Debezium session). We could then measure and optimize our burden, to the point of offloading much of the recovery strategy.

This issue is not exactly about introducing support for the above, but spec'ing out the above so that we understand how to communicate the trade-offs and both present and evaluate the alternatives. Should we recommend folks set up a read replica for us to hammer with Debezium? Should we build our own snapshots in-house? Should people keep Kafka logs around forever, just in case? I don't think we have opinions on these questions yet, having not yet fully mapped out the space.

[cuongdo]

@rjnn Do we have a way of improving our understanding what requirements prospects have of persistence?

I wonder if the implicit answer is: do what databases do. "Just" persist stuff and have it all (views, arrangements, etc.) come back when I restart Materialize. I wonder if asking customers to add more infrastructure or Kafka storage to deploy Materialize could be a significant hurdle for adoption, especially if we're going the cloud route. Cloud services typically excel with fast integration & deployment.

[gregberns]

+1
'Prospect' point of view: We'd like to snapshot 'regularly' (I don't know what that means - once a day?), so that a 'microservice' can rebuild itself from Midnight to Now off the Kafka queue, so it only take 'a couple minutes', instead of from the beginning of time.
Is it RockDB or Sled that can take a snapshot and send it to S3? Would like to persist to a location 'daily', so rebuilding was is non-trivial if an instance goes down.

[benesch]

@danhhz has taken this on. There are various Google Docs floating around internally mapping this out, and a proper team that is owning this project, so I'm going to consider this done!