Onnxruntime: How to extract output tensor from any layer of models

[xiuyangleiasp]

hello everyone, I'm newer of onnxruntime.
I want to get the output tensor of middle layer for a certain image during inference.
What should I to do?
Thanks very much.

[Dian-Yi]

......

[linkerzhang]

@xiuyangleiasp if you know which tensor you'd like to fetch. One good way is to put the tensor into your model (graph) outputs field.

[paulbauriegel]

@linkerzhang I was trying that and I'm not sure if that is currently possible.

Let's say I've a model.onnx with some hidden layer with the output name '65'

import onnx
model_onnx = onnx.load("model.onnx")
assert [n for n in model_onnx.graph.node if '65' in n.output][0].output, "65"

when I now try to access that layer:

import onnxruntime as rt
sess = rt.InferenceSession("model.onnx")
input_name = sess.get_inputs()[0].name
label_name = sess.get_outputs()[1].name # Using one of the output layer names works fine
label_name = '65' # trying to access any hidden layer fails 
pred_onx = sess.run([label_name], {input_name: blob})[0]

I'm getting following exception: Method run failed due to: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Output Name:65

[jywu-msft]

currently you can't fetch any arbitrary intermediate output.
you can only fetch outputs defined in the model's outputs. this is why @linkerzhang suggested a workaround might be to add the intermediate output you want to the model (graph) output field.
this assumes you are able to modify the onnx model (protobuf)

[jpa99]

Given an onnx model file with predefined outputs, is it possible to modify the model to add an intermediate tensor to to the model output field? How might I go about doing this?

EDIT: I was able to add any intermediate tensor to the model output via the following, which enables extracting the specified output in onnxruntime.

import onnx
from onnx import helper

model = onnx.load(model_path)
intermediate_layer_value_info = helper.ValueInfoProto()
intermediate_layer_value_info.name = intermediate_tensor_name
model.graph.output.append(intermediate_layer_value_info)
onnx.save(model, model_path)

[hariharans29]

Thanks @jpa99 for the code snippet. That pretty much answers the question. So closing this.

[ranazulay]

This issue is closed but the proposed solution isn't working.
onnx.onnx_cpp2py_export.checker.ValidationError: Field 'type' of value_info is required but missing.

Anyone else manged to extract results of arbitrary tensors for any layer in the model?

[samratdebroy]

This issue is closed but the proposed solution isn't working.
onnx.onnx_cpp2py_export.checker.ValidationError: Field 'type' of value_info is required but missing.

Anyone else manged to extract results of arbitrary tensors for any layer in the model?

Likewise, I'm also having the same issue. Is there work being done to eventually support inspecting data from intermediate layers? From my understanding, all node outputs have unique names even when the outputs only link to another node's inputs, so it would be nice if users could extract data for any uniquely identified outputs and not just graph-level outputs. It's very time consuming to implement this work-around especially because it fails for some models.

[ranazulay]

I've managed to get some progress with this, but it only works on models with inferred shapes (opset version 8, after that shapes inference is broken).
Another unfortunate limitation is that it only works on bottleneck nodes (not on nodes in parallel sub-graphs, such as shortcut/residual/shuffle blocks).

Hope it helps a bit:

    if end_node_names:
        model = onnx.load(model_path)
        end_node_shapes_and_names = dict()
        for node in model.graph.value_info:
            if node.name in end_node_names:
                end_node_shapes_and_names[node.name] = [x.dim_value for x in node.type.tensor_type.shape.dim]

        for name, shape in end_node_shapes_and_names.items():
            out_node = make_tensor_value_info(name=name, elem_type=onnx.TensorProto.FLOAT, shape=shape)
            model.graph.output.append(out_node)
        onnx.save_model(model, model_path)

    onnx_rt_sess = rt.InferenceSession(model_path)
    end_node_names = end_node_names if end_node_names else [onnx_rt_sess.get_outputs()[0].name]
    feed_dict = {onnx_rt_sess.get_inputs()[0].name: onnx_input_data}
    onnx_results = onnx_rt_sess.run(end_node_names, feed_dict)

[samratdebroy]

Sorry I should have mentioned I also made quite a bit of progress. I wrote a pretty generic script here: https://github.com/zetane/expose_onnx_node_outputs/blob/master/expose_node_outputs.py
that should work with most .onnx models in order to expose internal node outputs as graph-level outputs. It uses onnx.shape_inference to infer the shape and data type of different operator outputs.

It also suffers from some limitations. For example, any node for which data type cannot be inferred cannot be exposed as a graph output as that violates the onnx standard. It's just frustrating to have to use "hacks" like this to circumvent such a simple problem.

I hope the script helps others. The script can be used to overwrite an existing model or it can create a copy with the exposed nodes and will return a filepath to the modified model.