Glow: Assertion failed: (Index < Length && "Invalid index!") when testing glow using onnx

[zrphercule]

When I use onnx test driver to glowm it crashed on AveragePool test because of:
Assertion failed: (Index < Length && "Invalid index!"), function operator[]
This is a llvm related error, so I need to find which command caused this manually.

Finally I found out the crash is caused by this command in ONNX.cpp::loadOperator when dealing with averagepool:
auto *tr = G_.createTranspose(opName, in, NCHW2NHWC);

I wonder what is this G_ for (since I did not see any relative definition in ONNX.cpp), and why we are using NCHW2NHWC here?

[nadavrot]

@zrphercule Could you please upload the problematic ONNX file?

We convert NCHW to NHWC because we have one canonical representation of convolutions (and other ops). The optimizer will later clean things up. G_ is the graph container that keeps the loaded ONNX function.

[zrphercule]

@zrphercule Could you please upload the problematic ONNX file?

We convert NCHW to NHWC because we have one canonical representation of convolutions (and other ops). The optimizer will later clean things up. G_ is the graph container that keeps the loaded ONNX function.

Thanks, I will upload it tomorrow when I come back to office.

[zrphercule]

@nadavrot @rdzhabarov
Steps of repro this error:

1. git clone onnx
   git clone --recursive https://github.com/onnx/onnx.git

2. install glow and, int build repo, copy libonnxifi-glow.dylib to onnx/ (while changing its name):
   cp lib/Onnxifi/libonnxifi-glow.dylib .../onnx/libonnxifi.dylib

3. in onnx/, install onnx while exporting libjit:
   export ONNX_BUILD_TESTS=1;pip install -v -e .

4. in onnx/, copy avg pool data to a seperate repo:
   cp -r onnx/backend/test/data/node/test_averagepool_3d_default ~/test_example/

5. in onnx/, run onnxifi gtest and reproduce this error:
   ./.setuptools-cmake-build/onnxifi_test_driver_gtests ~/test_example

Right now, I am tracking this bug and still dont know if this is an issue of onnxifi test driver or glow. If you have any problem reproducing this bug, feel free to contact me.
The caller function of glow is in onnx/backend/test/cpp/driver_test.cc. Specifically, the buggy calling happens at the lib.onnxInitGraph().

Thank you!

[bertmaher]

Will take a look now, thanks for reporting!

[bertmaher]

A couple notes for myself:

export ONNX_BUILD_TESTS=1
export LD_LIBRARY_PATH=~/src/onnx
mkdir ~/test_example/    # before copying the avgpool data

Also I had some stale build products with the bad ONNX_NAMESPACE, and incremental rebuild didn't seem to fix it. Nuking my build dir and rebuilding from scratch was OK though.