Onnxruntime: How to get more than one outputs using onnxruntime?

[Li-chunming]

My confusion
I'm using Maskrcnn model and got the onnx file in model-zoo, it has one input and four outputs, I dont know how to use OrtRun() to get the entire outputs, I only have tried to get one of them.
Here are the codes:

std::array<float, 1 * 1 * 28 * 28> results_{};
std::array<float, 4> results_extra{}; //4 classes
int result_[1 * 28 * 28]{ 0 }
Ort::Value output_tensor_{ nullptr };
std::array<int64_t, 4> output_shape_{ 1,1,28, 28 };
...
const char* input_names[] = { "image" };
const char* output_names[] = { "6887" };
...
OrtValue* input_tensor_1 = input_tensor_;
OrtValue* output_tensor_1 = output_tensor_;
OrtRun(session_, nullptr, input_names, &input_tensor_1, 1, output_names, 1, &output_tensor_1);

And here are nodes I've printed:

Number of inputs = 1
input 0 : name=image
Input 0 : type=1
Input 0 : num_dims=3
Input 0 : dim 0=3
Input 0 : dim 1=-1
Input 0 : dim 2=-1
Number of outputs = 4
output 0 : name=6568
Output 0 : type=1
Output 0 : num_dims=2
Output 0 : dim 0=-1
Output 0 : dim 1=4
output 1 : name=6570
Output 1 : type=7
Output 1 : num_dims=1
Output 1 : dim 0=-1
output 2 : name=6572
Output 2 : type=1
Output 2 : num_dims=1
Output 2 : dim 0=-1
output 3 : name=6887
Output 3 : type=1
Output 3 : num_dims=4
Output 3 : dim 0=-1
Output 3 : dim 1=1
Output 3 : dim 2=28
Output 3 : dim 3=28

Who know how to change it?
System information

• ONNX Runtime version: 1.1.2 release

[Li-chunming]

@Timonsc Hi~ Can you give me some suggestion? Please!

[xgirones]

What about something like this? (I haven't checked it for syntactic correctness)

static const char* output_names[] = { "6568", "6570", "6572", "6887" };
static const size_t NUM_OUTPUTS = sizeof( output_names ) / sizeof( output_names[ 0 ] );

OrtValue* p_output_tensors[ NUM_OUTPUTS ] = {nullptr};

OrtRun(session_, nullptr, input_names, &input_tensor_1, 1, output_names, NUM_OUTPUTS, p_output_tensors );

[Li-chunming]

What about something like this? (I haven't checked it for syntactic correctness)
static const char* output_names[] = { "6568", "6570", "6572", "6887" };
static const size_t NUM_OUTPUTS = sizeof( output_names ) / sizeof( output_names[ 0 ] );

OrtValue* p_output_tensors[ NUM_OUTPUTS ] = {nullptr};

OrtRun(session_, nullptr, input_names, &input_tensor_1, 1, output_names, NUM_OUTPUTS, p_output_tensors );

Thanks you very much!! It works! But I'm confused how to transfor this datatype(OrtValue) to 4 different sizes and dims Array. (Sorry I'm not familiar with ORT API) I need Array to do postprocess. In my former codes I use the following sentences to build connection between OrtValue * output_tensor_1 and std::array <>results_{}, but now how to copy this process four times? Do you think there is a more convenient way?

std::array<float, 1 * 1 * 28 * 28> results_{};
Ort::Value output_tensor_{ nullptr };
std::array<int64_t, 4> output_shape_{ 1,1,28, 28 };
output_tensor_ = Ort::Value::CreateTensor<float>(allocator_info, results_.data(), results_.size(), output_shape_.data(), output_shape_.size());
OrtValue* output_tensor_1 = output_tensor_;  
OrtRun(session_, nullptr, input_names, &input_tensor_1, 1, output_names, 1, &output_tensor_1);

My entire codes are here.Thanks again for your selfless help!!

[hoaquocphan]

Hi @Li-chunming,

Currently, I'm importing TinyYolov3 to Onnxruntime by C++ and have some difficulty
When check the issue list, I see your issue same as mine and the TinyYolov3 model have some same point with Maskrcnn

• Maskrcnn: 1 inputs and 4 outputs
• TinyYolov3 : 2 inputs and 3 outputs

So Please help me some issues below:

1/ in your code, I see below line:
const char* output_names[] = { "6568","6570","6572","6887" };
so where did you get these name, I didn't see them on https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/mask-rcnn

2/ in the preProcessing step, I see below python code:
image = np.transpose(image, [2, 0, 1])
How do you convert it to C++ code?

3/ have you create a C++ with more than 1 input?
If yes, could you share me a sample? or if you know, Please share me some example

Thanks

[Li-chunming]

Hi @hoaquocphan
As for your 1 questions, you can find the name by putting your onnx flies to the Netron manually ,or use the following codes :
`
Ort::Session session(env, model_path, session_options)

//// print model input layer (node names, types, shape etc.)
Ort::AllocatorWithDefaultOptions allocator;
//// print number of model input nodes
size_t num_input_nodes = session.GetInputCount();
size_t num_output_nodes = session.GetOutputCount();
std::vector<const char*> input_node_names(num_input_nodes);
std::vector<const char*> output_node_names(num_output_nodes);
std::vector<int64_t> input_node_dims
printf("Number of outputs = %zu\n", num_output_nodes);
   ////iterate over all input nodes
for (int i = 0; i < num_output_nodes; i++) {
            std::vector<int64_t> output_node_dims;
    ////    // print input node names
    char* output_name = session.GetOutputName(i, allocator);
    printf("output %d : name=%s\n", i, output_name);
    output_node_names[i] = output_name;
 ////// print input node types
    Ort::TypeInfo type_info = session.GetOutputTypeInfo(i);
    auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
    ONNXTensorElementDataType type = tensor_info.GetElementType();
    printf("Output %d : type=%d\n", i, type);
    //// print input shapes/dims
    output_node_dims = tensor_info.GetShape();
    printf("Output %d : num_dims=%zu\n", i, output_node_dims.size());
    for (int j = 0; j < output_node_dims.size(); j++) {
        printf("Output %d : dim %d=%jd\n", i, j, output_node_dims[j]);
    }
     }`

1. ORT does not have the function to realize transpose(), so I use this:

cv::Mat paddimg;

cv::resize(bgrimg_f, paddimg, cv::Size(padd_w, padd_h));
// 3D matrix transpose HWC -> CHW
float* output = input_image_.data();
fill(input_image_.begin(), input_image_.end(), 0.f);
for (int c = 0; c < 3; c++) {
    for (int i = 0; i < padd_h; i++) {
        for (int j = 0; j < padd_w; j++) {
            output[c * padd_w * padd_h + i * padd_w + j] = (paddimg.ptr<float>(i)[j * 3 + c]);
        }
    }
}`

You can find this in my codes.
But I haven't check the correction, because my postprocession doesn't finished.
3.I haven't used the model that has two or more inputs,(only try tinyYOLOv2 before), maybe we can find the solution together. If you have done this please tell me.

[hoaquocphan]

Hi @Li-chunming

Thank you for your support.
1/ I tried your way and I can get the output name.
2/ I will try it later because I'm making the postprocessing step as you.

I will inform you when I finish the porting tinyYolov3 and I will share you my C++ file for example to support you if you want.
I'm new in this field, so do you want to get contact to support together in the future?

[hoaquocphan]

Hi @xgirones

I see your solutions with multi output and it is OK, but Please help me with multi input.
Currently, I'm porting tiny_yolov3 and it have 2 inputs as below:

Input 0 : name=input_1
Input 0 : type=1
Input 0 : num_dims=4
Input 0 : dim 0=1
Input 0 : dim 1=3
Input 0 : dim 2=416
Input 0 : dim 3=416
Input 1 : name=image_shape
Input 1 : type=1
Input 1 : num_dims=2
Input 1 : dim 0=1
Input 1 : dim 1=2

But I can put the input_1 to create tensor as below:
g_ort->CreateTensorWithDataAsOrtValue(memory_info, input_tensor_values.data(), input_tensor_size*sizeof(float), input_node_dims_input.data(), 4, ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, &input_tensor);

So how to put 2 input: input_1 and image_shape to this function?
Please help me.

[hoaquocphan]

Hi All,

I just found the method here: https://github.com/microsoft/onnxruntime/issues/2250

[hoaquocphan]

Hi @Li-chunming

I don't understand your code about the transpose() function.
the Python is OK with below line
image_data = np.transpose(image_data)

but it can not work with below line as the preprocessing step:
image_data = np.transpose(image_data, [2, 0, 1])

I really can not understand what is [2, 0, 1]
So could anyone explain to me.

[Li-chunming]

Hi @Li-chunming
I don't understand your code about the transpose() function.
the Python is OK with below line
image_data = np.transpose(image_data)
but it can not work with below line as the preprocessing step:
image_data = np.transpose(image_data, [2, 0, 1])
I really can not understand what is [2, 0, 1]
So could anyone explain to me.

This is because when you use cv2.imread() to read an image , the structure is HWC , but my model's input needs CHW, so I need to do 3D transpose. I have found many models don't need to do this,maybe yours is one of them.
Thanks for your help! I'll learn it later!

[hoaquocphan]

Hi @Li-chunming

Thanks you, I understood about transpose image.
In my preprocessing step, it have another line:
image_data = np.expand_dims(image_data, 0)

So do you know about purpose of expand_dims?

[Li-chunming]

Hi @Li-chunming
Thanks you, I understood about transpose image.
In my preprocessing step, it have another line:
image_data = np.expand_dims(image_data, 0)
So do you know about purpose of expand_dims?

Your model input needs dim=4, but the image you read is dim=3, so you need to expand one dimension on the axis = 0, np.expand_dims(image_data, 0) will realize this function.
For instance, the original image has the shape(416, 416, 3),but your model input needs shape (1,3,416,416) to feed, so you should do np.transpose(,[2,0,1]) to let your image become to shape(3,416,416) firstly ,and then do np.expand_dims(, axis=0) to become(1,3,416,416).

[hoaquocphan]

Hi @Li-chunming
Thank you very much for your explain, I understand more after read your explain, but I have some concern as below:
see my code:

stbi_uc * img_data = stbi_load(input_file, &img_sizex, &img_sizey, &img_channels, STBI_default);
struct S_Pixel
    {
        unsigned char RGBA[3];
    };
    const S_Pixel * imgPixels(reinterpret_cast<const S_Pixel *>(img_data ));
int offs;
std::vector<float> input_tensor_values(input_tensor_size);
for ( int c = 0; c < 3; c++){
      for ( int y = 0; y < img_sizey; y++){
            for ( int x = 0; x < img_sizex; x++, offs++){
                const int val(imgPixels[y * img_sizex + x].RGBA[c]);
                imge->set((y*img_sizex+x)*3+c, val);
                input_tensor_values[offs] = ((float)val)/255;
            }
      }
}
CheckStatus(g_ort->CreateTensorWithDataAsOrtValue(memory_info, input_tensor_values.data(), input_tensor_size*sizeof(float), input_node_dims_input.data(), 4, ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, &input_tensor[0]));

As you see that, I write all image data to input_tensor_values, and this is a 1 dimension vector
so how the CreateTensorWithDataAsOrtValue function know this is 1 dimension or 4 dimensions

[Li-chunming]

@hoaquocphan Hi, after checking a few of instance , I have found that all input_tensor vector of them has one dimension, in other words, I think this function has default paramater that this vector must be 1 dimension, we cannot choose the other. Maybe you can set it to 2 dims and check if the function still work.

[hoaquocphan]

Hi @pranavsharma

could you support me about the concern of dimension in my previous comment.

Thanks

[Li-chunming]

Hi~@hoaquocphan Now I've finished my project and gotten right outputs. Here I upload my final C++ codes and related comparative python file.
In this project, it is worth mentioning that I changed my model(Pytorch version) to a new one(Tensorflow version) which has 3 inputs and 7 outputs! So I think my new model is very similar to yours, and maybe you can get a lot of inspiration from my project.
As for your previous questions, can you describe it more specifically? Like "How to realize np.expand_dim() in C++”，so I can give you specific help.

[hoaquocphan]

Hi My good friend @Li-chunming

Thank you very much for your support,
currently, I'm stuck in this ticket: 3227

So I will refer your code and continues to update my code.
If there are any unclear point need support from you, I will ask you later.
Thank again.

[Li-chunming]

Hi@hoaquocphan
For commercial reasons, I cannot share the model with you Bui I'd like to show the detail information of it:

The 0 in shapes means -1 actually, thai's a dynamic size.
In my postprocess, I only need the detection & mask data, so I use output[0]and[3].
As for your problem, I'm still looking for the reason, I'm not too familiar with ORT 1.0 details, but I think the output vector may not use a fixed shape.

[hoaquocphan]

Hi @Li-chunming

Thanks for your support, I'm new so there are many things I didn't know.

I completed the TinyYolo v2 on both onnx v0.4 and v1.0 and push to github as below:
onnxruntime v0.4: tinyyolo v2 on onnx v0.4
onnxruntime v1.0: tinyyolo v2 on onnx v1.0
You can check to see the difference between onnx 0.4 and onnx 1.0

my problem on tinyYolo v3 is because of it have 2 inputs and 3 outputs.
I have tried to push the tensor pointer to the array or vector but the result are the same as ticket 3227

So I think the issue because of I put/get the data to/from Run function via vector uncorrectly.

you can check the OrtApis::Run and check my code to support me to detect my wrong usage of this function. I cannot find the issue from last week up to now

Thanks

[hoaquocphan]

Hi @Li-chunming

I just got enough 3 outputs, so I will continues to create the post processing step.
So maybe I will need the support from you if there are any difficulty, hope that it doesn't bother you.
Thanks

[hoaquocphan]

Hi @Li-chunming

As your previous post, I see your model on the netron
so when I import my model to netron and see this model on netron,

• how we can create the code base on the model on netron?
• are there any document or referent link for investigating it?
  Thanks

[Li-chunming]

Hi@hoaquocphan
I cannot see your model in your link, that's just an empty netron link.
But it doesn' matter, actually I only use netron to find the wrong nodes,this only happens at the stage of exporting the onnx model. For instance,if the onnx model cannot be loaded by onnxruntime, I will find the specific node problem based on the error message.
But in the inference phase after successful loading, I only care about the input and output of the model, including their names and dimensions,except for them, I did not use Netron to view other information.
How is your project progressing? I'm glad to hear that you finally get the output result. Since you only do target detection and do not involve segmentation tasks, the post-processing process should be relatively simple. If there is a problem, I will be happy to help.

[hoaquocphan]

Hi @Li-chunming

As my previous post, I got 3 outputs from onnxruntime and stuck at the post-processing step.
the post-processing step in tiny_yolov3 is simple and unclear.
I found other link1; link2 and it is have some difference about the output of onnxruntime such as:
(1x'n_candidates'x4); (1x80x'n_candidates'); ('nbox'x3)
and
[(1, 13, 13, 255), (1, 26, 26, 255), (1, 52, 52, 255)]

I push all my code and model here
You can try it on Linux PC with onnxruntime branch rel-1.0.0
So hope that you can check and give me some advice about the post-processing step.

Thanks you very much

[stale[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[stale[bot]]

This issue has been automatically closed due to inactivity. Please reactivate if further support is needed.

[hoaquocphan]

Hi @Li-chunming
I have some difficulties need consult you,
so how can I contact you?
could you give me your email, facebook or anything that I can contact you?
thanks very much