One: Compiler FE: InstanceNorm from TF1.15.x tflite

[seanshpark]

Let's investivate InstanceNormalization from keras_contrib.layers with TensorFlow 2.0.0 alpha0

Background

• CartoonGan works only with this TF 2.0.0 a0 version
• InstanceNormalization is used
• InstanceNormalization decomposed network of tflite model looks somewhat differenent than 2.3.0 or 2.5.0

Purpose

• Support this pattern to our InstNorm fusion

Investigation result

• Use InstanceNormalization source from keras_contrib
  
  
  
  • keras_contrib will request above TensorFlow 2.2.0, which this is not our targert
  
  
• Use TensorFlow 1.15.3
  
  
  
  • This version will produce somewhat different InstanceNormalization tflite which I am interested in.
  
  

[seanshpark]

Virtual env setup

virtualenv --system-site-packages -p python3 tf20a0venv
source tf20a0venv/bin/activate

python -m pip install tensorflow==2.0.0-alpha0
python -m pip install git+https://www.github.com/keras-team/keras-contrib.git

[seanshpark]

Virtual env setup

virtualenv --system-site-packages -p python3 tf1153venv
source tf1153venv/bin/activate

python -m pip install tensorflow==1.15.3
python -m pip install h5py==2.10.0

[seanshpark]

Test sorce

import sys
import tensorflow as tf

from tensorflow.keras.layers import Layer, InputSpec
from tensorflow.keras import initializers, regularizers, constraints
from tensorflow.keras import backend as K

class InstanceNormalization(Layer):

copy-paste InstanceNormalization class from keras_contrib/instancenormalization.py

# Our test model
inputs = tf.keras.Input(shape=(16, 16, 3))
y = tf.keras.layers.ZeroPadding2D(padding=(3, 3))(inputs)
y = InstanceNormalization(axis=-1, epsilon=1e-9)(y)
y = tf.keras.layers.Activation("relu")(y)
keras_model = tf.keras.Model(inputs=inputs, outputs=y)

# save to h5 model
keras_model.save("InstNorm_keras.h5")

# convert to tflite model
converter = tf.lite.TFLiteConverter.from_keras_model_file("InstNorm_keras.h5",
    custom_objects={"InstanceNormalization": InstanceNormalization})
converter.allow_custom_ops = True
converter.experimental_new_converter = True
tflite_model = converter.convert()
open("InstNorm_keras.tflite", "wb").write(tflite_model)

InstanceNorm_keras.zip

[seanshpark]

keras model

tflite model

[seanshpark]

CC @meejeong

[seanshpark]

Todo in preparing draft

• [x] prepare tflite recipie
• [x] add to test matereials
  
  
  
  • [x] SQUARE in _luci_interpreter_
  
  
• [x] revise fuse instnorm pass
  
  
  
  • [x] refactor by version codes
  
  
  • [ ] ~add fuse SUB + SQUARE -> SQUARED_DIFFERENCE~
  
  
  • [ ] ~add CSE pass for MEAN~
  
  
  • [ ] ~transform POW(x, 0.5) --> SQRT ???~
  
  
  • [ ] Supprt new version: Version_3
  
  
• [x] verification with dredd
  
  
  
  • [ ] ~support multiple options~
  
  

CSE: Common subexpression elimination

[seanshpark]

To extract only InstanceNormalization,

# Our test model
inputs = tf.keras.Input(shape=(16, 16, 3))
y = tf.keras.layers.ZeroPadding2D(padding=(3, 3))(inputs)
y = InstanceNormalization(axis=-1, epsilon=1e-9)(y)
y = tf.keras.layers.Activation("relu")(y)
keras_model = tf.keras.Model(inputs=inputs, outputs=y)

[seanshpark]

There is a case where last MUL with gamma and ADD with beta does not exist.

• assume gamma is 1.0 and beta is 0.0
• assume they are removed as no effect nodes

[seanshpark]

done