One: Compiler FE: Partition model with popular public models

[seanshpark]

Let's check model partition with IV3 and other popular models

From this experience we may get more ideas with model partitioning.

• such as, partition before/after particular op

and find unknown problems from this task

---

Sites of public models

• https://tfhub.dev/

[seanshpark]

For #6230

[seanshpark]

How to test the partition...

• Just a sctipt to do some manual partitioning
• A compiler sub module run as a test

As a first step, let's find out what are needed to do this.

[seanshpark]

Found working on top of one-cmds testing is one of (many??) easy approach.

[seanshpark]

Models check

• https://tfhub.dev/captain-pool/lite-model/esrgan-tf2/1
• https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1

faster_rcnn/openimages_v4/inception_resnet_v2/1 load error?

bin/tf2tfliteV2.py", line 195, in _v2_convert
graph_def.ParseFromString(file_content)
google.protobuf.message.DecodeError: Error parsing message

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 1: invalid continuation byte

--> need to add model_format=saved_model to .cfg file.

now the error is

RuntimeError: MetaGraphDef associated with tags {'serve'} could not be found in SavedModel. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: saved_model_cli
available_tags: [set()]

[seanshpark]

CC @mhs4670go

[mhs4670go]

@seanshpark Saved model related API - from_saved_model has three parameters., i.e. saved_model_dir, signature_keys, tags.

As of now, tf2tfliteV2 supports limited interface when it comes to saved model. It means that users can set saved_model_dir but can't set signature_keys, tags. They are just set to default value.

Most of saved models might have default tags, i.e. SERVING. But above inception_resnet_v2 model doesn't.

All tags in the tag set must be present. (default set(SERVING))

$ saved_model_cli show --dir ${model_folder} --all

MetaGraphDef with tag-set: '' contains the following SignatureDefs:

signature_def['default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['images'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, -1, -1, 3)
        name: hub_input/image_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 4)
        name: hub_input/strided_slice:0
    outputs['detection_class_entities'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: hub_input/index_to_string_Lookup:0
    outputs['detection_class_labels'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 1)
        name: hub_input/strided_slice_2:0
    outputs['detection_class_names'] tensor_info:
        dtype: DT_STRING
        shape: (-1, 1)
        name: hub_input/index_to_string_1_Lookup:0
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: hub_input/strided_slice_1:0
  Method name is: 

As you can see, its tag name is ''. So, rest of parameters should be given.

I tried to solve this with various methods. Here is what I have done for everyone's information:)

Use tflite_convert CLI - fail

Because tensorflow 2.x CLI doesn't have option that can set tag, I tested it with tensorflow 1.x CLI that has many options for convenience.

$ tflite_convert --output_file ${output_folder} --saved_model_dir ${model_folder} \
--saved_model_tag_set '' --saved_model_signature_key default \
--input_arrays hub_input/image_tensor --input_shapes 1,200,200,3

...
2021-04-30 23:07:51.759921: F ./tensorflow/lite/toco/toco_tooling.h:38] Check failed: s.ok() Found Sub as non-selected output from Switch, but only Merge supported. Control flow ops like Switch and Merge are not generally supported. We are working on fixing this, please see the Github issue at https://github.com/tensorflow/tensorflow/issues/28485.
Fatal Python error: Aborted

Current thread 0x00007f64e4763740 (most recent call first):
  File "/home/seongwoo/venv/lib/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 52 in execute
  File "/home/seongwoo/venv/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
  File "/home/seongwoo/venv/lib/python3.6/site-packages/absl/app.py", line 299 in run
  File "/home/seongwoo/venv/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40 in run
  File "/home/seongwoo/venv/lib/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 89 in main
  File "/home/seongwoo/venv/bin/toco_from_protos", line 8 in <module>
Aborted (core dumped)

Control flow ops like Switch and Merge are not generally supported.

I think it failed because tensorflow 1.x is used that doesn't support control flow.

Use python API - fail

As I said above, I gave missing parameters to the API.

converter = tf.lite.TFLiteConverter.from_saved_model(flags.input_path, ['default'], {})

```bash
2021-05-01 00:13:06.912003: E tensorflow/core/grappler/grappler_item_builder.cc:669] Init node hub_input/index_to_string/table_init doesn't exist in graph
Traceback (most recent call last):
File "tf2tfliteV2.py", line 258, in
main()
File "tf2tfliteV2.py", line 254, in main
_convert(flags[0])
File "tf2tfliteV2.py", line 237, in _convert
_v2_convert(flags)
File "tf2tfliteV2.py", line 229, in _v2_convert
tflite_model = converter.convert()
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/lite/python/lite.py", line 1117, in convert
return super(TFLiteConverterV2, self).convert()
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/lite/python/lite.py", line 921, in convert
self._funcs[0], lower_control_flow=False))
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1105, in convert_variables_to_constants_v2_as_graph
aggressive_inlining=aggressive_inlining)
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 801, in __init__
aggressive_inlining)
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 972, in _run_inline_graph_optimization
return tf_optimizer.OptimizeGraph(config, meta_graph)
File "/home/seongwoo/venv_2x/lib/python3.6/site-packages/tensorflow/python/grappler/tf_optimizer.py", line 58, in OptimizeGraph
graph_id, strip_default_attributes)
ValueError: Failed to import metagraph, check error log for more info.

### Re-save the model - success
- reference: https://stackoverflow.com/questions/65650859/converting-pretrained-model-from-tfhub-to-tflite

It turned out that the downloaded model seems pretty much old one. So, it should be re-saved to be fresh one.

```python
MODEL_DIR = 'model_path'
SIGNATURE_KEYS = ['default']
SIGNATURE_TAGS = set()
saved_model = tf.saved_model.load(MODEL_DIR, tags=SIGNATURE_TAGS)
tf.saved_model.save(saved_model, 'new_model_path', signatures=saved_model.signatures)

After running these commands, you can get a new version model.

With this, I ran tf2tfliteV2 again..

$ cd ${model_folder}
$ python tf2tfliteV2.py --v2 --saved_model -i . -o ${output_folder}
2021-05-01 00:44:52.100441: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-01 00:44:52.100492: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
...
2021-05-01 00:45:35.470076: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 6307720 microseconds.
2021-05-01 00:46:07.608598: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:194] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2021-05-01 00:46:11.036877: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
[1]    11651 killed     python tf2tfliteV2.py --v2 --saved_model -i . -o ${output_folder}

Hmm.. I don't know why it was killed. Maybe because of my pc's weakness or of control dependency's problem.

When I tried it with my local pc in my home, it failed. But, with another pc in work place, it is converted successfully:)

[mhs4670go]

The converted file is too big to upload. But I think you can make it by yourself. Or, I'll upload it to other server.

[seanshpark]

import tensorflow as tf

MODEL_DIR = 'faster_rcnn_openimages_v4_inception_resnet_v2_1'
RESAVE_DIR = 'faster_rcnn_openimages_v4_inception_resnet_v2_1-resaved'
SIGNATURE_KEYS = ['default']
SIGNATURE_TAGS = set()
saved_model = tf.saved_model.load(MODEL_DIR, tags = SIGNATURE_TAGS)
tf.saved_model.save(saved_model, RESAVE_DIR, signatures=saved_model.signatures)

@mhs4670go , thank you!!!

I've got the model :)

• this takes several minutes ...

[seanshpark]

partition_faster_rcnn.cfg

[one-build]
one-import-tf=True
one-import-tflite=False
one-import-bcq=False
one-optimize=False
one-quantize=False
one-pack=False
one-codegen=False

[one-import-tf]
input_path=./faster_rcnn_openimages_v4_inception_resnet_v2_1-resaved
output_path=./faster_rcnn_openimages_v4_inception_resnet_v2_1.circle
converter_version=v2
model_format=saved_model

convert to .circle

one-build -C partition_faster_rcnn.cfg

partition_faster_rcnn.part

[partition]
backends=cpu,acl_cl
default=acl_cl
comply=opcode

[OPCODE]
ADD=cpu

pepare partition

mkdir partition_faster_rcnn.work
cp faster_rcnn_openimages_v4_inception_resnet_v2_1.circle partition_faster_rcnn.work/
cp partition_faster_rcnn.part partition_faster_rcnn.work/

run partition

circle_partitioner partition_faster_rcnn.part \
  faster_rcnn_openimages_v4_inception_resnet_v2_1.circle \
  partition_faster_rcnn

-->

ERROR: Error: Unsupported tensor type: STRING

circledump.. there's a STRING tensor -_-;

• it's huge and lots of custom op... and we need to enable IF/WHILE

Operator Codes: [order] OpCodeName (OpCode Enum)
[0] CUSTOM(HashTableV2) (code: 32, version: 1)
[1] MUL (code: 18, version: 1)
[2] SHAPE (code: 77, version: 1)
[3] STRIDED_SLICE (code: 45, version: 1)
[4] CUSTOM(TensorArrayV3) (code: 32, version: 1)
[5] RANGE (code: 96, version: 1)
[6] CUSTOM(TensorArrayScatterV3) (code: 32, version: 1)
[7] WHILE (code: 119, version: 1)
[8] CUSTOM(TensorArraySizeV3) (code: 32, version: 1)
[9] CUSTOM(TensorArrayGatherV3) (code: 32, version: 1)
[10] SUB (code: 41, version: 1)
[11] CONV_2D (code: 3, version: 1)
[12] MAX_POOL_2D (code: 17, version: 1)
[13] AVERAGE_POOL_2D (code: 1, version: 1)
[14] CONCATENATION (code: 2, version: 1)
[15] ADD (code: 0, version: 1)
[16] RESHAPE (code: 22, version: 1)
[17] EXPAND_DIMS (code: 70, version: 1)
[18] TRANSPOSE (code: 39, version: 1)
[19] UNPACK (code: 88, version: 1)
[20] EXP (code: 47, version: 1)
[21] PACK (code: 83, version: 1)
[22] SLICE (code: 65, version: 1)
[23] SOFTMAX (code: 25, version: 1)
[24] ZEROS_LIKE (code: 93, version: 1)
[25] CAST (code: 53, version: 1)
[26] TILE (code: 69, version: 1)
[27] EQUAL (code: 71, version: 1)
[28] SPLIT (code: 49, version: 1)
[29] MINIMUM (code: 57, version: 1)
[30] MAXIMUM (code: 55, version: 1)
[31] GREATER (code: 61, version: 1)
[32] WHERE (code: 109, version: 1)
[33] GATHER (code: 36, version: 1)
[34] FILL (code: 94, version: 1)
[35] CUSTOM(CropAndResize) (code: 32, version: 1)
[36] MEAN (code: 40, version: 1)
[37] FULLY_CONNECTED (code: 9, version: 1)
[38] CUSTOM(LookupTableFindV2) (code: 32, version: 1)
[39] CUSTOM(InitializeTableV2) (code: 32, version: 1)
[40] LESS (code: 58, version: 1)
[41] LOGICAL_AND (code: 86, version: 1)
[42] CUSTOM(TensorArrayReadV3) (code: 32, version: 1)
[43] DIV (code: 42, version: 1)
[44] ROUND (code: 116, version: 1)
[45] REDUCE_MAX (code: 82, version: 1)
[46] IF (code: 118, version: 1)
[47] RESIZE_BILINEAR (code: 23, version: 1)
[48] CUSTOM(TensorArrayWriteV3) (code: 32, version: 1)
[49] SQUEEZE (code: 43, version: 1)
[50] CUSTOM(NonMaxSuppressionV4) (code: 32, version: 1)
[51] CUSTOM(Size) (code: 32, version: 1)
[52] TOPK_V2 (code: 48, version: 1)

...

B(20) (15) 0x1 00 00 00 0xc 00 00 00 0xf 00 00 00 0x55 0x4e 0x4b ...
...
B(36053) (16) 0x31 0x2e 0x31 0x35 0x2e 0x30 00 00 00 00 00 00 00 00 00 00

...

T(0:19) STRING () B(20) std.constant18

$ circledump faster_rcnn_openimages_v4_inception_resnet_v2_1.circle | grep STRING
T(0:19) STRING () B(20) std.constant18
T(0:1312) STRING () B(1313) StatefulPartitionedCall_1:3
T(0:1313) STRING () B(1314) StatefulPartitionedCall_1:1
T(11:0) STRING (604) B(1387) std.constant565
T(11:1) STRING (604) B(1388) std.constant566

[seanshpark]

Check with https://tfhub.dev/google/experts/bit/r50x1/in21k/object/1

RuntimeError: MetaGraphDef associated with tags {'serve'} could not be found in SavedModel. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: saved_model_cli
available_tags: [set()]

import tensorflow as tf

MODEL_DIR = 'experts_bit_r50x1_in21k_object_1'
RESAVE_DIR = 'experts_bit_r50x1_in21k_object_1-resaved'
SIGNATURE_KEYS = ['default']
SIGNATURE_TAGS = set()
saved_model = tf.saved_model.load(MODEL_DIR, tags = SIGNATURE_TAGS)
tf.saved_model.save(saved_model, RESAVE_DIR, signatures=saved_model.signatures)

$ saved_model_cli show --dir experts_bit_r50x1_in21k_object_1
The given SavedModel contains the following tag-sets:
''
'train'

$ saved_model_cli show --dir experts_bit_r50x1_in21k_object_1-resaved
The given SavedModel contains the following tag-sets:
'serve'

Try to import resaved model

ValueError: Only support a single signature key.

Huh?
-->

diff --git a/compiler/tf2tfliteV2/tf2tfliteV2.py b/compiler/tf2tfliteV2/tf2tfliteV2.py
index c6973ff96..acc44ab67 100755
--- a/compiler/tf2tfliteV2/tf2tfliteV2.py
+++ b/compiler/tf2tfliteV2/tf2tfliteV2.py
@@ -214,7 +214,8 @@ def _v2_convert(flags):
         converter = tf.lite.TFLiteConverter.from_concrete_functions([wrap_func])

     if flags.model_format == "saved_model":
-        converter = tf.lite.TFLiteConverter.from_saved_model(flags.input_path)
+        converter = tf.lite.TFLiteConverter.from_saved_model(
+            flags.input_path, signature_keys=['default'])

     if flags.model_format == "keras_model":
         keras_model = tf.keras.models.load_model(flags.input_path)

pepare partition

mkdir partition_experts_bit_object.work
cp experts_bit_r50x1_in21k_object_1.circle partition_experts_bit_object.work/
cp partition_experts_bit_object.part partition_experts_bit_object.work/

run partition

circle_partitioner partition_experts_bit_object.part \
  experts_bit_r50x1_in21k_object_1.circle \
  partition_experts_bit_object.work

[seanshpark]

@mhs4670go , do you think we need to add this change?

-        converter = tf.lite.TFLiteConverter.from_saved_model(flags.input_path)
+        converter = tf.lite.TFLiteConverter.from_saved_model(
+            flags.input_path, signature_keys=['default'])

[mhs4670go]

@seanshpark Umm.. When I tried to convert above model, it works without creating new model. It already has serve tag-set. Could you give it a try one more time?

$ saved_model_cli show --dir . --all
2021-05-04 13:08:14.590200: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, 3)
        name: serving_default_input_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2048)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict

do you think we need to add this change?

I think we need some time. As of now the conversion works well with API's default value. Even if it is needed, it would be better to add 'signauture_keys` option that is given from user later.

[seanshpark]

Model I tested is in TF2.0 Saved Model (v1) paragraph;

wget https://tfhub.dev/tensorflow/efficientnet/lite0/feature-vector/2?tf-hub-format=compressed \
        -O experts_bit_r50x1_in21k_object_1.tar.gz

mkdir -p experts_bit_r50x1_in21k_object_1
tar zxvf experts_bit_r50x1_in21k_object_1.tar.gz -C experts_bit_r50x1_in21k_object_1/

cd experts_bit_r50x1_in21k_object_1

saved_model_cli show --dir . --all
Traceback (most recent call last):
  File "/home/maxwell/bin/saved_model_cli", line 387, in <module>
    Main()
  File "/home/maxwell/bin/saved_model_cli", line 287, in Main
    module_space = FindModuleSpace()
  File "/home/maxwell/bin/saved_model_cli", line 118, in FindModuleSpace
    raise AssertionError('Cannot find .runfiles directory for %s' % sys.argv[0])
AssertionError: Cannot find .runfiles directory for /home/maxwell/bin/saved_model_cli

ADD: FYI: WIP codes: https://github.com/Samsung/ONE/pull/6719/files

[mhs4670go]

@seanshpark It's weird..

$ wget https://tfhub.dev/tensorflow/efficientnet/lite0/feature-vector/2\?tf-hub-format\=compressed \
        -O experts_bit_r50x1_in21k_object_1.tar.gz
$ ls
experts_bit_r50x1_in21k_object_1.tar.gz
$ tar -xf experts_bit_r50x1_in21k_object_1.tar.gz 
$ ls
assets  experts_bit_r50x1_in21k_object_1.tar.gz  saved_model.pb  tfhub_module.pb  variables
$ saved_model_cli show --dir . --all
2021-05-04 13:38:56.453109: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

MetaGraphDef with tag-set: '' contains the following SignatureDefs:

signature_def['default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['images'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, 3)
        name: images:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['default'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1280)
        name: efficientnet-lite0/model/head/global_average_pooling2d/Mean:0
  Method name is: 

signature_def['image_feature_vector']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['images'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, 3)
        name: images:0
...

$ pip list | grep tensorflow
tensorflow             2.4.1
tensorflow-estimator   2.4.0
tensorflow-gpu         2.4.1

[mhs4670go]

@seanshpark Did you build saved_model_cli by yourself? If so, how about just using another one from pip install?

[seanshpark]

Ah... I didn't run saved_model_cli in virtual environment... my bad :)

It seems OK!

[seanshpark]

Anyway, without below change

-        converter = tf.lite.TFLiteConverter.from_saved_model(flags.input_path)
+        converter = tf.lite.TFLiteConverter.from_saved_model(
+            flags.input_path, signature_keys=['default'])

gives me this error

ValueError: Only support a single signature key.

Update: this is not needed for experts_bit_r50x1_in21k_object_1

• but need to check with faster_rcnn_openimages_v4_inception_resnet_v2_1

[seanshpark]

wget https://tfhub.dev/tensorflow/efficientnet/lite0/feature-vector/2?tf-hub-format=compressed \
-O experts_bit_r50x1_in21k_object_1.tar.gz

So sorry; this URL came from unintentional copy-paste... (seems from faster_rcnn_openimages_v4_inception_resnet_v2_1)
anyway, if there is any problems, let's gather them to new issue...

[seanshpark]

Model: https://tfhub.dev/deepmind/i3d-kinetics-400/1

mkdir -p i3d-kinetics-400_1.work
cp partition_i3d-kinetics.part i3d-kinetics-400_1.work/
cp i3d-kinetics-400_1.circle i3d-kinetics-400_1.work/
circle_partitioner \
   partition_i3d-kinetics.part \
   i3d-kinetics-400_1.circle \
   i3d-kinetics-400_1.work

-->

ERROR: circle model has duplicate names

[seanshpark]

Model: https://tfhub.dev/google/tiny_video_net/tvn3/1
-> TF 1.x model...
-> need input/output properties...

• there seems two sub-graphs exist
• input seems Placeholder, [64,160,160,3]
• there seems no output but lots of Assign : don't know what these are...
• try output with dense/BiasAdd, [8,157]

--> we only import v1 frozen model, but this doesn't seem to be it...

--> #6740

[seanshpark]

Model: https://tfhub.dev/tensorflow/movinet/a5/base/kinetics-600/classification/1
-> TF2.x keras model
-> import is ok with saved_model

Error:

bash partition_movinet_a5_kin_600.test
ERROR: circle model has duplicate names

[seanshpark]

Close this task for this sprint as some problems for partitioner was discovered.
Will create new issue for next sprint when necessary.