Onnx-tensorrt: No importer registered for op: NonZero

[UndeadBlow]

When I'm trying to import model from ONNX file, I'm getting:

ERROR: TensorRT/parsers/onnx/ModelImporter.cpp:134 In function parseGraph:
[8] No importer registered for op: NonZero

How NonZero can be replaced or workarounded?

[francisco-perez-sorrosal]

Same problem here when trying to run inference using:

sudo docker run --gpus '"device=0"' --rm -p8000:8000 --shm-size=1g --ulimit  memlock=-1 --ulimit stack=67108864 --net inference_network --network-alias=trt_server -v/home/fperez/dev/models/tensorrt:/models nvcr.io/nvidia/tensorrt:20.03-py3 giexec --onnx=/models/bert-onnx/test/model.onnx --device=0 --verbose

Error:

[W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/07/2020-09:32:17] [04/07/2020-09:32:17] [V] [TRT] ImporterContext.hpp:122: Registering layer: (Unnamed Layer* 16) [Constant] for ONNX node:
[04/07/2020-09:32:17] [V] [TRT] ImporterContext.hpp:97: Registering tensor: 218 for ONNX tensor: 218
[04/07/2020-09:32:17] [V] [TRT] ModelImporter.cpp:180:  [ConstantOfShape] outputs: [218 -> (-1)],
[04/07/2020-09:32:17] [V] [TRT] ModelImporter.cpp:107: Parsing node:  [NonZero]
[04/07/2020-09:32:17] [V] [TRT] ModelImporter.cpp:123: Searching for input: 218
While parsing node number 15 [NonZero -> "219"]:
[04/07/2020-09:32:17] [V] [TRT] ModelImporter.cpp:129:  [NonZero] inputs: [218 -> (-1)],
--- Begin node ---
ERROR: ModelImporter.cpp:134 In function parseGraph:
[8] No importer registered for op: NonZero
input: "218"
output: "219"
op_type: "NonZero"

--- End node ---
[E] Failed to parse onnx file
[04/07/2020-09:32:17] [04/07/2020-09:32:17] [04/07/2020-09:32:17] [04/07/2020-09:32:17] &&&& FAILED TensorRT.trtexec # /opt/tensorrt/bin/trtexec --onnx=/models/bert-onnx/test/model.onnx --device=0 --verbose
[E] Parsing model failed
[E] Engine creation failed
[E] Engine set up failed

In the onnx graph:

...
%216 : Long() = onnx::Gather[axis=0](%214, %215) # /Users/fperez/dev/transformers/src/transformers/modeling_bert.py:175:0
  %217 : Tensor = onnx::Unsqueeze[axes=[0]](%216)
  %218 : Tensor = onnx::ConstantOfShape[value={1}](%217)
  %219 : Tensor = onnx::NonZero(%218)
  %220 : Tensor = onnx::Transpose[perm=[1, 0]](%219)
  %221 : Tensor = onnx::Squeeze[axes=[1]](%220)
...

Are there plans to support NonZero?

[yutkin]

+, the same problem. Need NonZero support.

[Oktai15]

Really need this operation! Any progress with this issue?

[UndeadBlow]

Up? For now no one model from TF OD API can be loaded because of this node.

[Raneee]

Really need this operation!

[gabrielibagon]

This is a problem for me as well.

This occurs in TensorFlow when using tf.where with only the condition argument specified (i.e. x=None and y=None). Even though Where is a supported op, onnx replaces tf.where() with NonZero when only the condition argument is used in tf.where.

Here is a reproducible example for building an onnx model:

import tensorflow as tf
import tf2onnx

with tf.Session() as sess:
    # Build model
    x = tf.convert_to_tensor([1, 0, 0, 1], name='input')
    x = tf.where(x, name='output')  # problematic layer
    # Create graph def
    graph_def = tf.get_default_graph().as_graph_def()
    output_graph_def = tf.graph_util.convert_variables_to_constants(sess, graph_def, ["output"])
    # Convert to onnx and export
    with tf.Graph().as_default() as graph:
        tf.import_graph_def(output_graph_def, name="")
        onnx_graph = tf2onnx.tfonnx.process_tf_graph(graph, opset=11, input_names=["input:0"], output_names=["output:0"])
        model_proto = onnx_graph.make_model("sample")
        with open("sample.onnx", "wb") as f:
            f.write(model_proto.SerializeToString())

The error can be found using trtexec or a onnx2trt (i.e. trtexec --onnx=sample.onnx).

[buptstehc]

Really need this operation, too!

[RunningLeon]

Same issue. Wonder are there any practical reasons that this OP should not be included in TensorRT?

[silverlining21]

Same issue. onnx model generate under enviroment config as below:
TRT 7.2.1
onnx 1.6
opset 11
pytroch 1.5
torchvision 0.6

any update would be appericated.

[kevinch-nv]

We currently do not support the NonZero operator, which is why you are seeing this error. We have plans to support this in a future release.