Incubator-mxnet: Stacking 0-order tensors

[JeanKossaifi]

Description

There seem with stacking 0-order tensors.
Stacking arrays of shape (1, ) results in an array of shape (N, 1) instead of (N, ) so cannot easily be used as a substitute.

This is for Python 3.8.3 and mxnet 1.6.0

Error stacking 0-order arrays

Error Message

MXNetError: [15:42:55] src/imperative/./imperative_utils.h:146: Operator stack inferring shapes failed.
input shapes:
None
None
None
None
output shapes:
None
operator attributes:
num_args : 4
axis : 0


Stack trace:
  [bt] (0) 1   libmxnet.so                         0x000000011d2fcbd9 libmxnet.so + 31705
  [bt] (1) 2   libmxnet.so                         0x000000011f06fc9d mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*) + 2365
  [bt] (2) 3   libmxnet.so                         0x000000011f06e4f0 mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&) + 688
  [bt] (3) 4   libmxnet.so                         0x000000011ef8a45c SetNDInputsOutputs(nnvm::Op const*, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> >*, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> >*, int, void* const*, int*, int, int, void***) + 1564
  [bt] (4) 5   libmxnet.so                         0x000000011ef8b4d3 MXImperativeInvokeEx + 99
  [bt] (5) 6   libffi.7.dylib                      0x000000010dee2ead ffi_call_unix64 + 85

To Reproduce

import mxnet.ndarray as nd

l = [nd.array(v) for v in [1, 2, 3, 4]]
nd.stack(*l, axis=0)

Current behaviour

import mxnet.ndarray as nd

l = [nd.array(v).reshape(1) for v in [1, 2, 3, 4]]
res = nd.stack(*l, axis=0)
res.shape

This code runs but outputs a shape of (4, 1) so isn't a real solution. This is related with the way scalars are represented:

On zero-order tensors

As a side question, related to the previous observation, why does contracting an array fully return an array of shape (1, ) instead of (, )?

nd.dot(r, r).shape # (1, )

[szha]

thanks for reporting the issue (and good to see you again Jean!). what's the behavior in numpy?

[JeanKossaifi]

Hi @szha, nice to see you again as well! :)

what's the behavior in numpy?

So in Numpy, concatenating scalars (zero order tensors) works and returns a 1D array:

import numpy as np
l = [np.array(v) for v in [1, 2, 3, 4]]
np.stack(l, axis=0) # array([1, 2, 3, 4])

Stacking arrays of shape (1, ) actually behaves the same as in NumPy and returns an arrary of shape (4, 1) so isn't directly useable for the above.

Contracting a tensor to a scalar returns a zero order tensor in NumPy (instead of an array of shape (1, ) in MXNet):

l = np.arange(4)
np.dot(l, l).shape # ()

In MXNet:

import mxnet.ndarray as nd
l = nd.arange(4)
nd.dot(l, l).shape # (1, )

[szha]

ok so the reason it failed is that our NDArray didn't support 0-order tensors. we made an extension to the NDArray shape semantics to make it numpy-compatible as part of the numpy compatibility change. There's a switch to enable this extension of shape semantics:

import mxnet as mx
mx.npx.set_np()
l = [mx.nd.array(v) for v in [1, 2, 3, 4]]
mx.nd.stack(*l, axis=0)