Glow: Graph optimizer should perform constant folding

[opti-mix]

If e.g. both operands of an arithmetic operation like add or sub are splats (i.e. constants) or more generally tensors with payloads statically known at compile-time , the result of the operation can be computed at compile-time. The process of evaluation the constant expressions at compile time is known as constant folding.

Such constant computations may arise e.g. after applying some other optimizations or transformations.

Currently, Glow's graph optimizer does not support constant folding. But it should be fairly easy to extend the graph optimizer to support constant folding. Most likely it should be performed during the execution of simplifyNode in GraphOptimizer.cpp

[jfix71]

Related -- once we have constant folding implemented, it would be great to extend the optimizer to recognize commutative patterns like this (seen in en2gr) which could also use constant folding:

[jackm321]

This looks interesting to work on. I'm curious if anybody has a sense for what the prioritization is for this? Is it more of a wishlist item for now or would it be worth while to look into doing it?

[jfix71]

Is it more of a wishlist item for now or would it be worth while to look into doing it?

@jackm321 I'd lean a bit more towards wishlist for now. I think it will be important for models which are exported via ONNX, as the exporter is sometimes somewhat naive and outputs patterns like the one I posted above from en2gr. We do care about seq2seq networks like en2gr, and it would be helpful for other models exported from ONNX, but as far as I've seen it isn't prevalent in other networks generally and so IMO isn't a high priority for now.

Also, I've seen related issues (as described here https://github.com/pytorch/glow/issues/1580#issuecomment-418606504) where the ONNX exporter decides to export multiple nodes to create a shape vector as an input tensor to a node instead of statically passing the shape in as member arguments (e.g. the shape for a Reshape was constructed via multiple nodes and passed in as a tensor). So it would be nice to somehow integrate this constant folding at graph building time too, allowing us to load such graphs.

[nickgg]

@jfix71 how much is left to do of this?

[jfix71]

We do some simple versions of this, e.g. merging Reshapes/Transposes/Quantize/ConvertTo/etc. into Constants. There's no specific threshold here in the issue to consider it complete, but I wouldn't consider it complete yet. Ideally we would expand upon the current simple versions to include many more patterns -- and perhaps this could be made more generic by performing constant prop based on calling into our Interpreter kernels.

We probably also would want to refactor these opts into their own constant prop pass, as we may need that pass to occur at building time in special cases if we want to handle cases such as https://github.com/pytorch/glow/issues/2837#issuecomment-488786537, which can be commonly encountered when using the ONNX exporter as I also mentioned above.

[ksaurabh-cadence]

I will be upstreaming a general constant folding support improving upon the idea in #3048, which will perform constant folding at the loader and will support constant folding for most of the onnx and caffe2 operators.

[ksaurabh-cadence]

3141 MR created.