It seems the BucketIterator produces batches in WxN dim order rather than NxW dim order, where N is the batch size. Given that PyTorch tends to want data with the batch dimension first, wouldn't it be better to transpose the output? Is there an advantage to the current output? (I suppose it's nice for RNN inputs, but I'm still trying to figure out why PyTorch wants RNN inputs in that form).
neverfox
All 4 comments
PyTorch wants RNN inputs in that form because cuDNN does, and cuDNN does because batch-first would mean that the tensor that contains the input for one timestep wouldn鈥檛 be contiguous.
If you鈥檙e using a 1D convolution or another kind of layer, you may have to transpose first (but it鈥檚 unlikely to be a major performance bottleneck in your code).
jekbradbury
on 29 May 2017
@jekbradbury Thanks for the explanation about the RNN inputs! That makes sense. As for another layer I had in mind, it's Embedding, which strikes me as a very common first layer in NLP, usually sitting right before an RNN (and thus requiring a reordered in and possibly out). Good to know, in any case, that a transpose isn't a bottleneck.
neverfox
on 29 May 2017
Embedding should work equally well either way, since it treats all dimensions of its input as batch dimensions (there鈥檚 no real difference between how N and W are treated).
jekbradbury
on 29 May 2017
Ah, good to know. I didn't want to take a chance without understanding the internals better than I did.
neverfox
on 29 May 2017
Related issues
PetrochukM
路
3Comments
pasinit
路
3Comments
dilettacal
路
6Comments
Rj7
路
3Comments
Taekyoon
路
3Comments
Most helpful comment
PyTorch wants RNN inputs in that form because cuDNN does, and cuDNN does because batch-first would mean that the tensor that contains the input for one timestep wouldn鈥檛 be contiguous.
If you鈥檙e using a 1D convolution or another kind of layer, you may have to transpose first (but it鈥檚 unlikely to be a major performance bottleneck in your code).