Incubator-mxnet: ERROR: test_ndarray.test_order fails on ARMv7 (Qemu)

[larroy]

More details to be added once https://github.com/apache/incubator-mxnet/pull/12094 gets merged

Description

ERROR: test_ndarray.test_order fails on ARMv7 (Qemu)

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

Package used (Python/R/Scala/Julia):
(I'm using ...)

For Scala user, please provide:

1. Java version: (java -version)
2. Maven version: (mvn -version)
3. Scala runtime if applicable: (scala -version)

For R user, please provide R sessionInfo():

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)

Build config:
(Paste the content of config.mk, or the build command.)

Error Message:

======================================================================
ERROR: test_ndarray.test_order
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/qemu/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/qemu/mxnet/tests/python/unittest/test_ndarray.py", line 735, in test_order
    nd_ret_topk = mx.nd.topk(a_nd, axis=3, ret_typ="indices", k=2, is_ascend=False, dtype=np.float64).asnumpy()
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 1980, in asnumpy
    ctypes.c_size_t(data.size)))
  File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 253, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [12:30:41] /work/mxnet/src/operator/tensor/./ordering_op-inl.h:376: Check failed: element_num <= mxnet::common::MaxIntegerValue<IDType>() (5 vs. 0) 'IDType' does not have a sufficient precision to represent the indices of the input array. The total element_num is 5, but the selected IDType can only represent 0 elements

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(dmlc::StackTrace()+0x34) [0xb3b667d4]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x30) [0xb3b66ab8]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(void mxnet::op::TopKImpl<mshadow::cpu, int, double>(mxnet::RunContext const&, mxnet::Resource const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, mxnet::TBlob const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, mxnet::op::TopKParam const&)+0x180) [0xb5544138]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(void mxnet::op::TopK<mshadow::cpu>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x504) [0xb55a2e18]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x1d8) [0xb3c99450]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x586dec) [0xb3c17dec]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x320) [0xb3c2024c]
[bt] (7) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>)+0x84) [0xb3c22788]
[bt] (8) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run()+0x44) [0xb3c20750]
[bt] (9) /usr/lib/arm-linux-gnueabihf/libstdc++.so.6(+0x8cc90) [0xb2f74c90]


-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=911594377 to reproduce.
--------------------- >> end captured logging << ---------------------

Minimum reproducible example

(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

Steps to reproduce

Using the qemu branch in my fork, enabling ndarray tests with the patch below and running:

./build.py -p armv7 && ./build.py -p test.arm_qemu ./runtime_functions.py run_unittests_python3_qemu

--- a/ci/docker/runtime_functions.py
+++ b/ci/docker/runtime_functions.py
@@ -89,7 +89,7 @@ def run_unittests_python3_qemu_():
     logging.info("PIP Installing mxnet/tests/requirements.txt")
     check_call(['sudo', 'pip3', 'install', '-r', 'mxnet/tests/requirements.txt'])
     logging.info("Running tests in mxnet/tests/python/unittest/")
-    check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_ndarray.py:test_ndarray_fluent'])
+    check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_ndarray.py'])


 def parsed_args():

What have you tried to solve it?

[piyushghai]

@mxnet-label-bot [Flaky]

[larroy]

@piyushghai is not flaky.

[larroy]

@mxnet-label-bot, please add labels :[ARM, NDArray, Bug]

[larroy]

@mxnet-label-bot remove label: [Flaky]

[larroy]

@mxnet-label-bot remove ['Flaky']