Incubator-mxnet: Automatic Batching for Dynamic Graphs

[sbodenstein]

@piiswrong: This seems very relevant to MXNet's efforts to support dynamic graphs #5705:

• https://arxiv.org/pdf/1705.07860.pdf
• https://github.com/clab/dynet/blob/master/examples/python/tutorials/Autobatching.ipynb

@jekbradbury: any idea whether PyTorch is thinking about adding support for something like this?

[piiswrong]

Yeah we have been thinking about doing the same thing. Haven't got to it yet. Never thought you can write a paper about it...

[jekbradbury]

DyNet's autobatching approach is predicated on their lazy execution semantics (build a graph imperatively but don't execute it until later -- the autobatching can happen in between) and this limits the applicability of autobatching to truly dynamic situations (where the graph structure depends on runtime values of tensors e.g. because of discrete samples from a stochastic policy network).

PyTorch and Chainer are currently using eager execution semantics (other than the minimal laziness that comes from CUDA async execution), which means they'd either need to implement lazy eval or apply a very different autobatching approach. The problems with choosing lazy eval are a) the runtime conditionals mentioned above and b) the fact that as GPUs get faster and faster graph-building starts to be more costly than execution.

Ultimately what we need is either a magical static analyzer that can convert imperative-looking Python into an MXNet-like symbolic graph or a combination of autobatching with lazy eval together with annotations that say "this function can be statically compiled." MXNet is pretty well-positioned to do that since it's always had a mixed imperative-symbolic approach.

[yajiedesign]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!