Materialize: change server panics to graceful error messages

[rjnn]

discussed in a meeting, issue created and assigned to frank to fill out

[jamii]

Was Frank panicking in the meeting?

[frankmcsherry]

We currently have limited confidence about the conditions under which materialized falls down due to unmet assumptions about how it and the world around it work. Violent panics are great when we are trying to figure out whether something is going wrong or not, but we should try and reign in as many of these as possible with bail or expect if that isn't appropriate. Clear error messages in panics are great, especially if we aren't sure that folks will have debug information next to the binaries.

[rjnn]

@cuongdo should we reassign this? I originally only assigned it to frank to fill out the issue description.

[rjnn]

I think this has mostly been done. What else do we need to do to close this issue out?

[frankmcsherry]

I'm not sure we've done any of this. For starters, we would assign the issue to someone to review /audit code for panics, and attempt to characterize the errors as recoverable or non. It's a bunch of debt to pay off.

[cuongdo]

@benesch points out that some ungraceful error handling has been addressed through his runtime errors work, and some has been addressed through Natacha's work on making our Kafka code more robust.

@frankmcsherry: are there specific areas of the code that you are still concerned about?

[benesch]

@frankmcsherry: are there specific areas of the code that you are still concerned about?

Bumping this question!