Materialize: View dependencies are computed on optimized plan

[benesch]

Consider the following, which works in Materialize today:

materialize=> create view v as select 1;
CREATE VIEW
materialize=> create view w as select * from v where false;
CREATE VIEW
materialize=> drop view v;
DROP VIEW

The user expressed that w was dependent on v, but after optimization we determined that it was not. Is that... desirable? Correct? I'm really not sure. Probably it should not be allowed, and we need to track dependencies based on the unoptimized plan.

[benesch]

Ah, indeed, that trivially creates a catalog that will fail to boot. Fascinating.

[rjnn]

This looks correct to me! Just because a user expressed a dependency doesn't mean there is one, and we... do the right thing?

[frankmcsherry]

Probably even if the example as presented is "correct", I think @benesch is worried that there are some corner cases around when do we depend on views and when do we not. For example, I believe that for SELECT queries we need to choose a logical timestamp before we perform optimization (or, we used to) and that choice was made on the unoptimized view, which would include v in this case.

I'm up for keeping this open as "clarify dependence requirements" to ensure that catalogs are always bootable, that we produce results at explicable timestamps, and potentially other gotchas we haven't considered yet.

[justinj]

Outside of any corner cases that are worrying Nikhil, I disagree with @rjnn that this is desirable—I think philosophically all the user-visible semantic information of a program/query should be derived pre-optimization. This is why all those obnoxious diagrams of the "steps of a compiler" have "semantic analysis" as a stratified step before "optimization," it's like, should this program typecheck?

fn main() {
    if false {
        println!("{}", 1 + "banana");
    }
}

[benesch]

Agreed on all counts with Justin!