Incubator-mxnet: Custom Operator Seg Fault

Created on 27 Jul 2019 · 7Comments · Source: apache/incubator-mxnet

After this PR: https://github.com/apache/incubator-mxnet/pull/15518, custom operators will seg fault in symbolic mode.
I have created a script to reproduce the issue. If we set export MXNET_SUBGRAPH_BACKEND=NONE, then the script will run ok.

https://gist.github.com/Zha0q1/b89675763f49a63064ab1a37ea0174f2

I built from source (lattest) and ran the script on Deep Learning Base AMI (Ubuntu) Version 18.0 p2.8xlarge. I used cmake -DUSE_CUDA=0 -DUSE_MKLDNN=1 -GNinja .. to build.

Error Message:


ubuntu@ip-172-31-38-36:~/abc/incubator-mxnet$ python ../test.py 
[23:15:23] ../src/executor/graph_executor.cc:2018: Subgraph backend MKLDNN is activated.
[23:15:23] ../src/executor/graph_executor.cc:2018: Subgraph backend MKLDNN is activated.

Segmentation fault: 11

Stack trace:
  [bt] (0) /home/ubuntu/abc/incubator-mxnet/python/mxnet/../../build/libmxnet.so(+0x8e6149) [0x7ffa1d95b149]
  [bt] (1) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7ffa25ec14b0]
  [bt] (2) python() [0x4954ed]
  [bt] (3) python() [0x46a16d]
  [bt] (4) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0xe719) [0x7ffa251b6719]
  [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_closure_unix64_inner+0x1bf) [0x7ffa24fa5c2f]
  [bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_closure_unix64+0x46) [0x7ffa24fa5fa8]
  [bt] (7) /home/ubuntu/abc/incubator-mxnet/python/mxnet/../../build/libmxnet.so(+0xceb8a2) [0x7ffa1dd608a2]
  [bt] (8) /home/ubuntu/abc/incubator-mxnet/python/mxnet/../../build/libmxnet.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x7ffa1d7d22d6]

Bug

Source

Zha0q1

Most helpful comment

@Zha0q1 Thanks for reporting this. PR https://github.com/apache/incubator-mxnet/pull/15671 is created to fix this, your provided case can pass now. Please have a try for your real case to see if it works.

ZhennanQin on 27 Jul 2019

🚀2 👍2

All 7 comments

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Bug