Incubator-mxnet: integer overflow bug in large NDArray AGAIN

[Pagey]

Description

a bug that has been fixed last year #11742 @apeforest seems to have returned-
when creating large NDArrays indexes seem to overflow

Environment info (Required)

windows 10, python 3.6.8, mxnet 1.6.0

Error Message:

Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\Python36\lib\site-packages\mxnet-1.6.0-py3.6.egg\mxnet\ndarray\ndarray.py", line 194, in __repr__
return '\n%s\n<%s %s @%s>' % (str(self.asnumpy()),
File "C:\Program Files\Python36\lib\site-packages\mxnet-1.6.0-py3.6.egg\mxnet\ndarray\ndarray.py", line 2092, in asnumpy
ctypes.c_size_t(data.size)))
File "C:\Program Files\Python36\lib\site-packages\mxnet-1.6.0-py3.6.egg\mxnet\base.py", line 253, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [13:27:53] C:\Jenkins\workspace\mxnet\mxnet\src\ndarray\ndarray_function.cc:51: Check failed: size == to->Size() (57032704 vs. 4352000000) : copying size mismatch, from: 228130816 bytes, to: 17408000000 bytes.

Minimum reproducible example

(from issues it appeared in before #10807, #9207, #10158, #9304)

1. 
   print(mx.nd.zeros(shape=(34000000,128)))

or

1. 
   import numpy as np
   import mxnet as mx
   X = np.zeros((200000, 32768), dtypes=np.float32)
   mx.nd.array(X)

Steps to reproduce

see MRE

What have you tried to solve it?

1. build from April 11 2019 seems not to have this issue, so was probably added later

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): Bug

[frankfliu]

@mxnet-label-bot add [bug]

[frankfliu]

@apeforest Would you please take a look?

[larroy]

@access2rohit

[apeforest]

@Pagey The was some performance regression after we changed from int32_t to int64_t in MXNet 1.4 release and we had to add a compiler flag and make the default to int32_t (https://github.com/apache/incubator-mxnet/pull/14570). We have been working on the performance regression issue and plan to use int64_t by default in MXNet 1.6.

For now, please build mxnet from source with compiler flag USE_INT64_TENSOR_SIZE on to work for large NDArray.

[apeforest]

Closing this issue as it is not a bug