Onnxruntime: MKLDNN subgraph Error

[Godricly]

Describe the bug

import torch
import onnxruntime
from torch import nn
from torch.nn import functional as F

class Demo(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(16,16,3,2,1)
        self.conv2 = nn.Conv2d(16,16,3,2,1)
        self.conv3 = nn.Conv2d(16,16,3,2,1)


    def forward(self, x):
        y = self.conv1(x)
        y = F.relu(y)
        out1 = self.conv2(y)
        out2 = self.conv3(y)
        return out1, out2

if __name__ == "__main__":
    input_tensor = torch.zeros((1,16,100,100))
    demo = Demo()
    out = demo(input_tensor)
    torch.onnx.export(demo, input_tensor, "debug.onnx", #verbose=True,
                        input_names=['data'],
                        opset_version=10,
                        do_constant_folding=True,
                        dynamic_axes={'data':{0:'batch'},
                                    })

    ses_option = onnxruntime.SessionOptions()
    ort_session = onnxruntime.InferenceSession("debug.onnx", ses_option)

The model exported fails during intialization.
Error:

RuntimeError: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization:  onnxruntime_mkl/include/onnxruntime/core/graph/graph.h:926 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index.

Urgency
If there are particular important use cases blocked by this or strict project-related timelines, please share more information and dates. If there are no hard deadlines, please specify none.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 18.04
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: 561f2c4a9aa6e2c71ad817ba40891fe364b4545a
• Python version: 3.6.7
• Visual Studio version (if applicable): NA
• GCC/Compiler version (if compiling from source): 7.4.0
• CUDA/cuDNN version: NA
• GPU model and memory: NA

To Reproduce
Describe steps/code to reproduce the behavior:

1. compiling onnxrunime with MKLDNN support
2. pip install torch(I'm using daily build 1.3.0.dev20190906)
   2, run the script above

[Godricly]

@sreekanth-yalachigere @jywu-msft

[Godricly]

FYI.

[Godricly]

  /** Gets the maximum NodeIndex value used in the Graph. */
  int MaxNodeIndex() const noexcept { return static_cast<int>(nodes_.size()); }  //assume the casting won't overflow

  // the definition below
  std::vector<std::unique_ptr<Node>> nodes_;

Is this function somehow misleading? It actually represents the number of nodes before optimization.

[Godricly]

https://github.com/microsoft/onnxruntime/blob/561f2c4a9aa6e2c71ad817ba40891fe364b4545a/onnxruntime/core/providers/mkldnn/mkldnn_execution_provider.cc#L302

according to last one. The condition here should be great equal.

[jywu-msft]

@sreekanth-yalachigere , can you fix this as part of your upcoming PR?

[sreekanth-yalachigere]

@jywu-msft yes.

[Godricly]

fixing that condition wont fix the issue. Please double check the graph optimization parts. thx

[Godricly]

the #2117 didn't fix the demo script for me.

commit id tested 041a1cab54493d3d32e6a6a8b4981add8c1419ed.

onnxruntime/include/onnxruntime/core/graph/graph.h:938 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:4 Max:4

[sreekanth-yalachigere]

@Godricly I fixed this issue. Can you please try this branch?

[Godricly]

Sry for the late reply. It works. Many thx.