Tfjs: Unknown layer: ConvLSTM2D

[ZissisT]

Hello,

I get the following error when trying to import a model in tensorflow.js (v2.0.1)

Uncaught (in promise) Error: Unknown layer: ConvLSTM2D. This may be due to one of the following reasons:

1. The layer is defined in Python, in which case it needs to be ported to TensorFlow.js or your JavaScript code.
2. The custom layer is defined in JavaScript, but is not registered properly with tf.serialization.registerClass().

Is this layer going to be supported in general (it is supported by tensorflow python api already, https://www.tensorflow.org/api_docs/python/tf/keras/layers/ConvLSTM2D ) ?

Thank you

[rthadur]

@ZissisT there is not plan to support this in near future, do you wish to contribute ?

[ZissisT]

@rthadur I wish I could, but I am new in this area :/ I need some more reading before I can contribute to this one. Thank you for your response though :)

[dikatok]

@rthadur may I contribute to implement the layer?

@ZissisT would you mind sharing the model or any examples you are converting from? Just for testing purposes later

[ZissisT]

@dikatok I tried training a model described in this article https://thebinarynotes.com/video-classification-keras-convlstm/ (whole code and model are included there) and my goal was first to predict those videos using tfjs (using my trained model). Then as a second step instead of providing videos as input I would input frames from webcam and try to predict motion sequence (e.g. me executing a Golf Swing). So I trained the model in colab, converted it for tfjs and tried loading it but of course since it is not supported I got the error described in my first post.

[beniroquai]

I also had the this message earlier today. Recently, the tensorflow team ported it to the tflite framework. Would be very neat to have it in the tfjs framework too. The link can be found here.

The model I would like to export to tfjs looks like this:

````
from tensorflow import keras
from tensorflow.keras import layers

import tensorflow as tf

def SOFI_seq(Ntime=1, Nbatch=1, Nx=100, Ny=100, features=1, Nchannel=1, upsample=2):
# create our model here
kernel_size = 3
nfilters_lstm = 4

model = keras.Sequential(
[
 layers.Reshape(target_shape=(Nbatch,Nx//upsample,Ny//upsample,Ntime), name='reshape_1d_2d'),
 layers.ConvLSTM2D(filters=nfilters_lstm , kernel_size=(kernel_size, kernel_size)
          , data_format='channels_last'
          , recurrent_activation='hard_sigmoid'
          , activation='tanh'
          , padding='same'
          , return_sequences=False
          , name='convlstm2d_1'),
 layers.BatchNormalization(name='batchnorm_1'),
 #UpSampling2D(size=(2,2), name='upsampling_1'), #Subpixel_layer(),
 layers.Conv2D(1, 1, strides=(1, 1), padding='same', data_format='channels_last',  activation='relu'), 
 layers.Reshape((Nx*Ny,), name='reshape_2d_1d')
])

model.compile()

return model

Define Parameters

Define Parameters

Ntime = 30
Nbatch = 1
Nx = 128
Ny = 128
features = 1
Nchannel = 1

Training parameters

Nepochs = 1
Niter = 2
N_upsample = 1

create the model

print('Create the model!')
x_ = tf.ones((1,NtimeNbatchNxNyN_upsample)) #(Ntime, Nbatch, Nx, Ny, Nchannel)
model = SOFI_seq(Ntime=Ntime, Nbatch=Nbatch, Nx=Nx, Ny=Ny, features=features, Nchannel=Nchannel, upsample=N_upsample)
y_ = model(x_)
print(model.summary())

import tensorflowjs as tfjs
myfilename = 'converted_model'+str(Nx)+'_'+str(Ntime)+'_keras'
tfjs.converters.save_keras_model(model, myfilename)
tfjs.converters.load_keras_model(myfilename+'/model.json')
```

[dikatok]

@ZissisT @beniroquai thanks!

[beniroquai]

Thanks @dikatok. Is there anything I could help you with?

[dikatok]

@beniroquai not yet, I will certainly update either in the PR or here when I encounter blockers or need insights

[dikatok]

@ZissisT what format did you first save the model to? SavedModel or?

[ZissisT]

@dikatok
I directly saved it to tfjs format from Colab:

.
.
import tensorflowjs as tfjs
.
.
.
tfjs.converters.save_keras_model(model, '/path/to/tfjs_model_folder')
.
.
.

this saved into tfjs_model_folder all group-shardNofY.bin files together with model.json

[dikatok]

Ok, I have tested loading the reference models from you guys @ZissisT @beniroquai + https://keras.io/examples/vision/conv_lstm/, and it seemed okay.

Will test later this weekend using trained models and share the bundled javascript for you guys to test. 🥂

[beniroquai]

That's awesome! I saw some movement on the pill request. Is there anything in any nightly builds available already? Can't await to see that :)

[dikatok]

bundle.zip

Or just use it in html script tag

<script src="https://firebasestorage.googleapis.com/v0/b/tfjs-d4820.appspot.com/o/tf.min.js?alt=media&token=06f225ba-e480-43a5-840f-a70c403ba062">

Keep in mind some layers might not be compatible at the moment, BN for example

@beniroquai not sure if the PR can be merged yet for nightly build

Update 1

bundle.zip

<script src="https://firebasestorage.googleapis.com/v0/b/tfjs-d4820.appspot.com/o/tf.min.js?alt=media&token=a81d60a5-1246-40ef-a696-7b24db5352b9">

[beniroquai]

@dikatok Thanks for your effort! I tried it and get the following message:

Error: Provided weight data has no target variable: convlstm2d_1/recurrent_kernel

Do you have any idea what this could mean?

Versions:

>>> tf.__version__
'2.3.0'
>>> tfjs.__version__
'2.0.0'

and your bundle as the web-runtime.

Complete Model:

from tensorflow import keras
from tensorflow.keras import layers

import tensorflow as tf


def SOFI_seq(Ntime=1, Nbatch=1, Nx=100, Ny=100, features=1, Nchannel=1, upsample=2):
    # create our model here
    kernel_size = 3
    nfilters_lstm = 4

    model = keras.Sequential(
    [
     layers.Reshape(target_shape=(Nbatch,Nx//upsample,Ny//upsample,Ntime), name='reshape_1d_2d'),
     layers.ConvLSTM2D(filters=nfilters_lstm , kernel_size=(kernel_size, kernel_size)
              , data_format='channels_last'
              , recurrent_activation='hard_sigmoid'
              , activation='tanh'
              , padding='same'
              , return_sequences=False
              , name='convlstm2d'),
     layers.Conv2D(1, 1, strides=(1, 1), padding='same', data_format='channels_last',  activation='relu'), 
     layers.Reshape((Nx*Ny,), name='reshape_2d_1d')
    ])

    model.compile()

    return model


# Define Parameters
# Define Parameters
Ntime = 30
Nbatch = 1
Nx = 128
Ny = 128
features = 1
Nchannel = 1

# Training parameters
Nepochs = 1
Niter = 2
N_upsample = 1

# create the model
print('Create the model!')
x_ = tf.ones((1,Ntime*Nbatch*Nx*Ny*N_upsample)) #(Ntime, Nbatch, Nx, Ny, Nchannel)
model = SOFI_seq(Ntime=Ntime, Nbatch=Nbatch, Nx=Nx, Ny=Ny, features=features, Nchannel=Nchannel, upsample=N_upsample)
y_ = model(x_)
print(model.summary())


import tensorflowjs as tfjs
myfilename = 'converted_model'+str(Nx)+'_'+str(Ntime)+'_keras'
tfjs.converters.save_keras_model(model, myfilename)
tfjs.converters.load_keras_model(myfilename+'/model.json')

[dikatok]

@beniroquai Thanks!

That issue occurred because the name of the recurrent kernel in tfjs was recurrentKernel while in Python it is recurrent_kernel.

I missed it because I always use the option strict: false during the load 😉.

I have updated the bundle and link https://github.com/tensorflow/tfjs/issues/3637#issuecomment-671402608.

[beniroquai]

@dikatok Great! I figured out, that the firebase link worked, but the link to the bundle in the update 1 for some reason didn't. Great job! I'll give it a try now! :-)