Incubator-mxnet: convert from mxnet to onnx failed, with Unrecognized attribute: spatial for operator BatchNormalization

[dongjunjundong]

Description

Failed to convert pretrained mode cifar_resnet110_v1 from mxnet format to onnx;

Environment info (Required)

using the mxnet zoo pre trained module cifar_resnet110_v1
mxnet version 1.5.0
the convert call is below:
input_shape = [(1,3,32,32)]
export_model("simple_net-symbol.json","simple_net-0000.params",input_shape)

Error Message:

(Paste the complete error message, including stack trace.)

~/ENV/lib/python3.5/site-packages/onnx/checker.py in checker(proto, ctx)
50 proto_type.__name__))
51 return getattr(C, py_func.__name__)(
---> 52 proto.SerializeToString(), ctx)
53 return cast(FuncType, checker)
54 return decorator

ValidationError: Unrecognized attribute: spatial for operator BatchNormalization

==> Context: Bad node spec: input: "cifarresnetv10_conv0_fwd" input: "cifarresnetv10_batchnorm0_gamma" input: "cifarresnetv10_batchnorm0_beta" input: "cifarresnetv10_batchnorm0_running_mean" input: "cifarresnetv10_batchnorm0_running_var" output: "cifarresnetv10_batchnorm0_fwd" name: "cifarresnetv10_batchnorm0_fwd" op_type: "BatchNormalization" attribute { name: "epsilon" f: 1e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT } attribute { name: "spatial" i: 0 type: INT }

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: ONNX, Bug

[dongjunjundong]

onnx version 1.4.1

[vandanavk]

@dongjunjundong
MXNet has support up to ONNX v1.3.0. BatchNormalization (Opset7) had a an attribute "spatial" which is being exported from MXNet to ONNX. Looks like this attribute has been dropped in BatchNormalization (Opset 9).

Please try using ONNX v1.3.0. Exporting with ONNX 1.3.0 worked for me.

[mika-fischer]

Maybe mxnet should target a specific opset then. Or at the very least have a check in there that errors out with a sane error message when onnx is too new...

[TriLoo]

Same problem, you can remove the 'spatial' attribute of batchnorm from the python file (output of error info)!

[nopattern]

I have convert mxnet(1.5.0) to the onnx(1.5.0), the error is:

`INFO:root:Converting idx: 3, op: null, name: first-3x3-conv-batchnorm_gamma
INFO:root:Converting idx: 4, op: null, name: first-3x3-conv-batchnorm_beta
INFO:root:Converting idx: 5, op: null, name: first-3x3-conv-batchnorm_moving_mean
Traceback (most recent call last):
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 484, in
tune_and_evaluate(tuning_option)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 436, in tune_and_evaluate
net, params, input_shape, _ = get_network(network, batch_size=1)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 93, in get_network
return get_network_lpr_mb2(name,batch_size)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 143, in get_network_lpr_mb2
test_onnx()
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 135, in test_onnx
converted_model_path = onnx_mxnet.export_model(mx_sym, args, [input_shape], np.float32, onnx_file, True)
File "/home/deep/workssd/mxnet/incubator-mxnet/python/mxnet/contrib/onnx/mx2onnx/export_model.py", line 87, in export_model
verbose=verbose)
File "/home/deep/workssd/mxnet/incubator-mxnet/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py", line 234, in create_onnx_graph_proto
in_shape=in_shape[graph_input_idx],
IndexError: list index out of range
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/deep/workssd/work/pycharm-community-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 484, in
tune_and_evaluate(tuning_option)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 436, in tune_and_evaluate
net, params, input_shape, _ = get_network(network, batch_size=1)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 93, in get_network
return get_network_lpr_mb2(name,batch_size)
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 143, in get_network_lpr_mb2
test_onnx()
File "/home/deep/workssd/arm/tvm_app/tune_relay_mobile_gpu.py", line 135, in test_onnx
converted_model_path = onnx_mxnet.export_model(mx_sym, args, [input_shape], np.float32, onnx_file, True)
File "/home/deep/workssd/mxnet/incubator-mxnet/python/mxnet/contrib/onnx/mx2onnx/export_model.py", line 87, in export_model
verbose=verbose)
File "/home/deep/workssd/mxnet/incubator-mxnet/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py", line 234, in create_onnx_graph_proto
in_shape=in_shape[graph_input_idx],
IndexError: list index out of range`

I tracked the code and  found the 'batchnorm_moving_mean' failed because it is not in saved params. so the in_shape[graph_input_idx] is out of bound. 
Anyone has suggestion?

@vandanavk

[nopattern]

the moving_mean is in the auxs param. just merge args and auxs.

[Lebron8997]

@mxnet-label-bot
My onnx 1.5.0 (latest)
For batchnorm, I revised script mxnet/contrib/mx2onnx/_op_translations.py as follows:
1, on line 647: kernel = eval(attrs["kernel"]) if attrs.get("kernel") else None
This is needed for global pooling like:
x = mx.symbol.Pooling(data=data,
pool_type='avg',
global_pool=True,
name=name+'pool')

2, delete line 359: spatial=0
This line is not supported for onnx > 1.3.0

It works for me tranlate .params and json to onnx, but this failed me for inference in tensorRT 5.1.5. It should be another problem though I'm not sure about that.

[arsdragonfly]

@mxnet-label-bot
My onnx 1.5.0 (latest)
For batchnorm, I revised script mxnet/contrib/mx2onnx/_op_translations.py as follows:
1, on line 647: kernel = eval(attrs["kernel"]) if attrs.get("kernel") else None
This is needed for global pooling like:
x = mx.symbol.Pooling(data=data,
pool_type='avg',
global_pool=True,
name=name+'pool')

2, delete line 359: spatial=0
This line is not supported for onnx > 1.3.0

It works for me tranlate .params and json to onnx, but this failed me for inference in tensorRT 5.1.5. It should be another problem though I'm not sure about that.

Did you run into this ? Have you solved it?

[klonikar]

The model-symbol.json file I had did not have spatial attribute. Still it failed. I noticed that the mxnet-onnx converter was adding that attribute as below:
Simply comment the line 359 in the file /Library/Python/3.7/site-packages/mxnet/contrib/onnx/mx2onnx/_op_translations.py on MacOSX or /usr/local/lib/python3.6/dist-packages/mxnet/contrib/onnx/mx2onnx/_op_translations.py on Linux
re-run your mxnet2onnx converter code.

[AIGirl10]

@dongjunjundong
MXNet has support up to ONNX v1.3.0. BatchNormalization (Opset7) had a an attribute "spatial" which is being exported from MXNet to ONNX. Looks like this attribute has been dropped in BatchNormalization (Opset 9).

Please try using ONNX v1.3.0. Exporting with ONNX 1.3.0 worked for me.

Can you please tell me how to export with onnx 1.3 since we are using onnx from mxnet ?

[szha]

cc @josephevans who's working on ONNX improvements.