One: [onert] Discussion : Does ShapeInferer belong to `core` or `backend`?

[dayo09]

Issue

I have confronted the case that shape inferer depends on backend.
To support the kernel's dynamicShapeInference, I must change not only backend but also shapeInferer.

• Related Issue: https://github.sec.samsung.net/STAR/nnfw/issues/11793#issuecomment-850303

ShapeInferer is dependent on backend(kernel)

• When I want to implement a kernel, I have to change shape inferior to support dynamic shape inference.
  
  
  
  • shapeInferer is kernel-dependent from this point of view
  
  
• However, shapeInferer currently belongs to core, not backend.
  
  
  
  • StaticShapeInferer is used while 'compile', it is needed to calculate the tensor size and plan memory, so its function seems to be core.
  
  

So, what can be a solution is

• To move ShapeInferer into backend
• To load ShapeInferer functions while running StaticShapeInferer(in core)

How do you think about that?
And how will it be to call backend's function from 'compile'(by dl loading)?

@wateret @hyunsik-yoon @lemmaa + @ (all one developers)

[hseok-oh]

Basically, shape inference should independent with backend because shape inference is based on operation's definition.

[wateret]

Agree with @hseok-oh .

[wateret]

@dayo09

When I want to implement a kernel, I have to change shape inferior to support dynamic shape inference.

Are shape inferior and "shapeInferer" same thing?

shapeInferer is kernel-dependent from this point of view

Could you please elaborate why? I looked at https://github.sec.samsung.net/STAR/nnfw/issues/11793#issuecomment-850303 but still don't see why BCQ FC shape inference is dependent on kernel implementation.

[hyunsik-yoon]

Agree with @hseok-oh.

Also some possible future scenario could be the following:

• IR in compile time: Conv - relu
• Backend A (after passes dependent on A): CR(Convfused with Relu)
• Backend B : Conv - Relu

So for A, we need dynamic shape inference for CR; whereas for Backend B, we need dynamic shape inference for Conv and Relu.

If CR is specific to company who creates backend A, dynamic shape inference for CR should be in backend layer and not known to public git (just like bcq?).

Another issue with backend A is that backend A cannot be supported currently. One reason is that FunctionSequence::run() and dynamic shape inference assume that Operand should be matched to Function (1:1 match). But in the above example, there are 2 Operation and 1 Function.

Best solution could be removing all dependency of ir::Operation during execution time.

[wateret]

@hyunsik-yoon

So for A, we need dynamic shape inference for CR; whereas for Backend B, we need dynamic shape inference for Conv and Relu.

If CR is specific to company who creates backend A, dynamic shape inference for CR should be in backend layer and not known to public git (just like bcq?).

What I basically think is, for that fusing case, we can get the shape by calling "Conv" and "Relu" inference functions respectively. And if each operation is not confidential, the fused operation is not confidential. Unless we have an amazing shape inference for some fused operation that must be confidential.

Another issue with backend A is that backend A cannot be supported currently. One reason is that FunctionSequence::run() and dynamic shape inference assume that Operand should be matched to Function (1:1 match). But in the above example, there are 2 Operation and 1 Function.

You're right. We should resolve this for fusing operations, but IMHO it is more of operation fusing optimization task itself, not directly relevant to this topic.

[hyunsik-yoon]

Some more thought on @wateret's https://github.com/Samsung/ONE/issues/4481#issuecomment-704175172
_This comment is not a scope of this issue but leave a memo just for food for future. We may not need such complex case. :-)_

Some imaginary situation could be

• compile time: ADD-MUL-DIV-RELU6-...-CONV-CONV-SOFTMAX

A company FOO has their strange backend which want to convert the above into the following function seq:

• compile time: ADD-SuperFOO-SuperBAR-SOFTMAX

However, this is not possible currently.
1) we don't open _pass_ to 3rd party
2) we don't map a function to multiple ops (could be a subgraph with some branches).
3) we may need to open DynamicShapeInferer (at backend side) for SuperFOO. Also some shape/type validators.

• We know the SuperBAR's output shape (can be calculated from compile time IR). But what's going to be the shape of output of SuperFOO?

[dayo09]

@wateret

Could you please elaborate why? I looked at https://github.sec.samsung.net/STAR/nnfw/issues/11793#issuecomment-850303 but still don't see why BCQ FC shape inference is dependent on kernel implementation.

Well, you are seeing this as a dependency on BCQFC Operator, instead of bcq backend then? (It sounds pretty to me too.)
The problem now I have is that I have to shape validation of BCQ operators and some of the code cannot be exposed outside.
For example, I want to do some shape validation on BCQFC, but I am afraid some of the lines cannot be placed at onert/core/src/compiler/ShapeValidator.h.

[wateret]

@dayo09 Thank you for the explanation. Let me recap.

• There are some code(shape validation or shape inference or something else) that cannot be exposed in this public repo
• That means we need some way to delegate those things to another module(maybe defining an interface)

And I would like to distinguish this problem and where to put its solution. I mean the "another module" should not belong to backend since it is still dependent on the operation definition itself, not backend. In other words, even if there were many backends that support such operations, shape validation would be identical. And that would mean we should introduce another layer for those things but it would be an overkill.

So I would suggest building better "custom operation infra" rather than what I said above. Either way, it is going to require non-trivial amount of work 😢 .

[krayzemli]

Once we say shape inference result must not depend on the backend, as it is now, it would be consistent for shape inference to stay independent, as now, so that an instance of backend would not be necessary for static inferring. If we allow shape inference result to be backend-specific, then it is logical to move inference to backend.

When user wants to register a custom operation, he should be able to register custom shape inferer for that operation as well (I can't figure out how it works now). So, private operations should either be implemented through a custom kernel, or as proxies that depend on an external private implementation code through some way of dynamic binding.