Hub: How to fine tuning Universal sentence encoder model

[zhanghaoie]

Hi, I am wondering how to fine tuning universal sentence encoder model, I have set the trainable to True, and print all the trainable_varaibles, It seems like no variable in the use model appears.

Best regards,
Hao

[helloeve]

+1
Setting trainable=True doesn't seems to make a difference. It would be nice to have a way that we could fine tune the sentence encoder on a new dataset.

[micahprice]

+1 #36

[vbardiovskyg]

The universal sentence encoder modules currently contain indeed no trainable variables. We are working with the publisher to add new modules that will be trainable.

[helloeve]

Here is a workaround to fine tune the universal sentence encoder model. The general idea is to duplicate all variables/operations in a new graph and then set them trainable. Feel free to check this repo for this workaround approach before they release the trainable version.

[omerarshad]

Have they released trainable version?

[andresusanopinto]

Please checkout the version 2 of these modules for having the variables exposed as trainable and lets us know if you find any problems.

• https://tfhub.dev/google/universal-sentence-encoder/2
• https://tfhub.dev/google/universal-sentence-encoder-large/2
• https://tfhub.dev/google/universal-sentence-encoder-lite/2

[XMK233]

In version 2 of universal sentence encoder, it is said that "Exposed internal variables as Trainable." But in following code, why is there no trainable variables to be printed? I run the code on CoLab.

```import numpy as np
import tensorflow_hub as hub
import tensorflow as tf
from tensorflow.python.framework import graph_util
from tensorflow.examples.tutorials.mnist import input_data
import os
import pandas as pd

module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder/2")
variables_names = [v.name for v in tf.trainable_variables()]
with tf.Session() as sess111:
values = sess111.run(variables_names)
for k, v in zip(variables_names, values):
print ("Variable: ", k)

print(values)
```

[helloeve]

@XMK233 You will need to set trainable=True according to the documentation https://www.tensorflow.org/hub/api_docs/python/hub/Module#__init__

[XMK233]

Yeah, you are right. Thank you @helloeve !!

But I have another question, which is not related to Universal Sentence Encoder.

When I use the following way to set the parameter trainable, it seems that it is not effective.

model = tf.keras.Sequential([
    hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_96/feature_vector/3", trainable=True),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(10, activation='softmax',
                          kernel_regularizer=tf.keras.regularizers.l2(0.0001))
])
model.build((None,)+(96, 96, 3))
model.compile(
optimizer=tf.keras.optimizers.SGD(lr=0.005, momentum=0.9), 
loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1),
metrics=['accuracy'])
print(model.summary())

The output is like this:

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
keras_layer_2 (KerasLayer)   multiple                  2257984   
_________________________________________________________________
dropout_2 (Dropout)          multiple                  0         
_________________________________________________________________
dense_1 (Dense)              multiple                  12810     
=================================================================
Total params: 2,270,794
Trainable params: 12,810
Non-trainable params: 2,257,984
_________________________________________________________________

The trainable parameter is only in the Dense layer. So does it mean that the image modules are actually not trainable?

Thank you.

[anirbankonar123]

Is the Multilingual-large USE trainable in same way, we need to train this on custom domain and generate embeddings.
Thanks