One: Compiler FE: Model partitioning

[seanshpark]

As we support several backends and we can think of this in the frontend,

• one of the backend maybe need off-line code generation
• we can assign specific operator to specific backend
• we can literally split the model into several sub models for the backends

Currently I'm not clear of this task. Let's do some prototyping and see...

[seanshpark]

Let's begin with a simple Model

• ~Source Model: Input - Sin - Cos - Output~
• ~Target Model: (1) Input - Sin - Output , (2) Input - Cos - Output~

With this, I could prepare/build-up necessary methods for basic tasks.
--> it may not be possible to verify with _luci-interpreter_ as sin/cos is not supported...
--> Sqrt, Rsqrt are implemented

• Source Model: Input - Sqrt- Rsqrt- Output
• Target Models: (1) Input - Sqrt- Output , (2) Input - Rsqrt - Output

[seanshpark]

For the first step, lets assume model has only one subgraph with above Input - Sqrt- Rsqrt- Output network.
Then we can use IF or WHILE for multiple subgraphs.

[seanshpark]

https://github.com/Samsung/ONE/pull/6231#issuecomment-799037972 commit stores to two circle files like

[seanshpark]

Let's now try to group graphs with same kind of group(backend).

Cases of source circle model graph

• graph input: single or multiple
• graph output: single or multiple
• node output: used or unused

[seanshpark]

Things to do from 10,000 feet

• [x] check with a model that may produce recursive situation
• [x] produce connection information
• [x] save connection information to JSON file -> #6297
• [x] test framework with running the source model vs produced models -> #6315
• [x] do some refactoring
• [x] check for repetitive things (per Op) and find alternative way
  
  
  
  • [x] ConnectNode
  
  
  • [x] QueryOpCode
  
  
  • [x] CloneNode
  
  
• [x] check with well known public models #6707
  
  
  
  • [x] IV3, IV4
  
  
  • [x] mobilenet
  
  
• [x] support for all node types #6596
• [ ] support multiple subgraph model with IF/WHILE/... #6708
  
  
  
  • [ ] clone for IF/WHILE
  
  
  • [ ] connect for IF/WHILE
  
  
• [ ] TBA
• ADD unit test for new codes

that may produce recursive situation: where Sqrt is assigned to different group(backend)

Things to do from 100 feet

• [ ] how to do with CircleConst ?
  
  
  
  • [x] related: Net_InstanceNorm_003
  
  
  • [ ] partition by name shared constant
  
  
• [x] model validation
  
  
  
  • [x] all nodes must have a name
  
  
  • [x] all names must be unique
  
  
• [ ] Check with multiple virtual outputs model

[seanshpark]

check with a model that may produce recursive situation

• Rsqrt - Rsqrt sequnce should have one input and two outputs: one from first Rsqrt and another from second Rsqrt
• need to fix this

Let's try with this one: Net_Sqrt_Rsqrt_Add_001.circle

Problem(s) to solve

• Clone node input should exist: (1) sort in topological order (2) divide create and connect step --> go with (2)
• Remove unused outputs: but should not remove unused virtual output (or cases in multiple sub graphs)

With first one solved, https://github.com/Samsung/ONE/pull/6231#issuecomment-800719125 commit result

./circle-partitioner \
    --partition Net_Sqrt_Rsqrt_Add_001.part \
    Net_Sqrt_Rsqrt_Add_001.circle Net_Sqrt_Rsqrt_Add_001.part.circle

• todo : Remove unused outputs : need to remove rsqrt2 from outputs

[seanshpark]

todo : Remove unused outputs

Let's think this with save connection information file together...

[seanshpark]

@hseok-oh wrote https://github.com/Samsung/ONE/pull/6231#issuecomment-800762497

My draft based on partitioned model: MANIFEST and Net_Sqrt_Rsqrt_001.part.config(json)

MANIFEST

{ 
  "major-version" : "1",
  "minor-version" : "1",
  "patch-version" : "0",
  "models"      : [ "Net_Sqrt_Rsqrt_001.part.00001_acl_cl.circle" , "Net_Sqrt_Rsqrt_001.part.00002_cpu.circle" ],
  "partition"   : "Net_Sqrt_Rsqrt_001.part.config",
  "model-types" : [ "circle", "circle" ]
}

Net_Sqrt_Rsqrt_001.part.config

{ 
  "source" : {
    "file" : "Net_Sqrt_Rsqrt_001.circle",
    "inputs" : [ "ifm" ],
    "outputs" : [ "sqrt", "rsqrt", "ofm" ]
  },
  "parts" : [
    {  
      "file" : "Net_Sqrt_Rsqrt_001.part.00001_acl_cl.circle",
      "inputs" : [ "ifm" ],
      "outputs" : [ "sqrt2", "sqrt" ],
    },
    {
      "file" : "Net_Sqrt_Rsqrt_001.part.00002_cpu.circle",
      "inputs" : [ "sqrt2" ],
      "outputs" : [ "ofm", "rsqrt"]
    }
  ]
}

[seanshpark]

This (Part_Sqrt_Rsqrt_Add_002) fails with SQRT assigned to different group(backend)

--> seems fixed

[seanshpark]

This (Part_Sqrt_Rsqrt_Add_003) (all cpu) should produce only one but three

[seanshpark]

This (Part_Add_Sub_000) (Sub in different group) doesn't work

--> https://github.com/Samsung/ONE/pull/6231#issuecomment-801726402

[seanshpark]

Rename Net_ to Part_ for testing partitioning...

[seanshpark]

Need to find a model that needs to cleanup for there are unused input...
--> checked with force adding all inputs and do cleanup to remove unused inputs

But still would be nice to find a model for this case.

[seanshpark]

Started posting PRs that seems stable.

[seanshpark]

• how to do with CircleConst ?
  
  
  
  • related: Net_InstanceNorm_003
  
  

CircleConst

• there is no input
• it would be better to treat this to follow the use node (successor of CircleConst)
  
  
  
  • that is CircleConst is group neutral
  
  
  • what if there are two or more use nodes? --> make a clone? --> seems a bit complicated
    
    
    
    
    
    • make another PNode ? to point to same node? --> node2group conflicts.
    
    
    
  
  
• what if we don't partition CircleConst ?
  
  
  
  • in build_graph, we can check each inputs, check if it's CircleConst, find or clone it.
  
  
  • need to cleanup inputs if it is a CircleConst
  
  
  • --> this seems to work
  
  

[seanshpark]

To verify Net_InstanceNorm_003 network, need to add SquaredDifference in luci_interpreter

[seanshpark]

Let's add model validation: all nodes must have a name, all names must be unique

Current status

• TensorFlow internally manages uniqueness of node names.
• Tensorflow lite flatbuffer format does not restrict duplicate names
• Is there any duplicate check in tflite converter?

tflite converter

• tensorflow/python/lite/toco_python_api_wrapper.cc
• tensorflow/lite/toco/python/toco_python_api.cc
• tensorflow/compiler/mlir/lite/python/saved_model_to_tfl_flatbuffer.cc
• tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc
• tensorflow/compiler/mlir/tensorflow/translate/tf_mlir_translate.cc
• tensorflow/compiler/mlir/tensorflow/translate/import_model.cc

Hmm...

[seanshpark]

About terms...

• partition file: .part : describes how to partition the circle model
• connection file: .conn : describes how to connect partitioned circle models
  
  
  
  • we could say, "partition connection ini file" or "partition connection json file"
  
  
• partition configuration: file to store partition file or connection file based on ini or json file format

crew library is a library to read/write partition configuration files.

CC @hseok-oh

[seanshpark]

circle-part-eval generated connection file seems needs some touch

• source circle path is written in full absolute path: doesn't seem to fit with nnpackage

Let's revise circle-partitioner working path and arguments

• partition file is madatory: let's use positional like circle input file
• use third argument (folder) as work folder where input circle, part file exist and outputs are generated
• this way files in the connection file have only filename, no path
• nnpackage can be freely organize source and partitioned circle files

[hseok-oh]

@seanshpark I've confirmed that model partitioning test results are generated in ${NNCC_WORKSPACE}/compiler/circle-part-eval. I'll use this for runtime loading draft #6274. Thanks.

[seanshpark]

How to manage CircleConst

• previous comment : https://github.com/Samsung/ONE/issues/6230#issuecomment-809802813

Current issue

• https://github.com/Samsung/ONE/issues/7185#issuecomment-876012498

As for now, let's try with this change

• if CircleConst is used by only one node, let that CircleConst follow that node
• if CircleConst is shared by two or more nodes, let's make clones so that one CircleConst is used for one node.
  
  
  
  • we don't need to do this. we can ignore CircleConst in checking same group
  
  

--> https://github.com/Samsung/ONE/issues/7209

[seanshpark]

To be sure this change and any other change is working correclty, we need some validation method in test process...

How to validate?

• number of output circle files ?
• each circle file having input nodes with names bla bla bla and output nodes with names bla bla bla...

For first step, number of output circle files can be used for our CircleConst issue.