Hub: Cannot feed single element in ELMo TF-HUB embedder

[simonefrancia]

Hello,
I am working in building a Deep Neural Network Classifier starting from an ELMo embedding module that embeds string, using Tensorflow Hub Module. I am using this model definition:

url = "https://tfhub.dev/google/elmo/2"
embed = hub.Module(url, trainable=True)


def make_elmo_embedding(x):
    embeddings = embed(tf.squeeze(tf.cast(x, tf.string)), signature="default", as_dict=True)["elmo"]
    return embeddings

# elmo embedding dimension
elmo_dim = 1024

# Input Layers
elmo_input = Input(shape=(None, ), dtype="string")

# Hidden Layers
elmo_embedding = Lambda(make_elmo_embedding, output_shape=(None, elmo_dim))(elmo_input)


x = Bidirectional(LSTM(128, dropout=0.2, recurrent_dropout=0.2))(elmo_embedding)
x = Dense(32, activation='relu')(x)
predict = Dense(2, activation='sigmoid')(x)
model = Model(inputs=[elmo_input], outputs=predict)
model.compile(loss='mse', optimizer='sgd')

This code works ok if the input of my network is a list with more than two strings, for example ['hello' , 'my name is Simone'].
If my input is a list with only one element (['hello']), this error appears:

tensorflow.python.framework.errors_impl.InvalidArgumentError:
 input must be a vector, got shape: [] .   [[{{node lambda_1/module_apply_default/StringSplit}}]]

This error is reproducible both in training mode ( batch_size = 1) and inference mode ( if I want to make inference on 1 element for example).

Looking at the error node lambda_1/module_apply_default/StringSplit} I think that the problem is in the Lambda layer, for the fact that in function make_elmo_embedding the tf.squeeze operation cancels the dimension = 1 and the input vector results of dimension = []. I tried to delete tf.squeeze but other errors occured.
What you suggest to solve this situation in order to make this model definition compatible both for single and multiple example lists?

Thanks
I am using Tensorflow 1.13.1

[simonefrancia]

@vbardiovskyg is this a possibile issue or maybe a bug on my implementation?

[vbardiovskyg]

Hi Simone, my suspicion is tf.squeeze, that would make ["hello"] into "hello", but string_split expects a 1D tensor. I would therefore go that direction and remove the errors that come along the way that you are mentioning.

[simonefrancia]

Yes I tried removing tf.squeeze but it gives errors the same. I don't know how to keep it working with only one example.

[vbardiovskyg]

Can you please show the minimal inference code example?

[simonefrancia]

This code (CASE 1) produces the bug:

CASE 1
sentence=['hello']
output = model.predict ([sentence], batch_size=16)

but with this code (CASE 2):

CASE 2
sentence=['hello','hi']
output = model.predict ([sentence], batch_size=16)

it works.
I tried a lot of possible combinations of input types (list of list, np.array of list, list of np.array, ecc) for CASE 1 (1 example to infer) and no one worked.
Thanks

[vbardiovskyg]

Unfortunately I cannot reproduce this and won't be able to answer for the next 2 weeks.

My suggestions:

• try to understand what kind of data is coming in, maybe using print statements.
• if you remove tf.squeeze, what kind of errors do you get? Are they the same?
• maybe try instead tf.squeeze using tf.reshape(..., [-1])

[simonefrancia]

@vbardiovskyg Are there any news about that? I couldn't get it.

[vbardiovskyg]

Hi, as I said there are multiple ways to solve this.

1. Understand what goes into the model:
   Is the tf.squeeze needed here?
   embeddings = embed(tf.squeeze(tf.cast(x, tf.string)), signature="default", as_dict=True)["elmo"]

You could simply change it to tf.reshape(tf.cast(x, tf.string), [-1]) and it should already work.

1. But even better, do you actually need it at all? If you want to pass in batches of tokens or batches of sentences, i.e. shape [None], then the input layer should look like this:
   elmo_input = Input(shape=(), dtype="string") or
   elmo_input = Input(batch_shape=(None, ), dtype="string")

Then you can completely remove tf.squeeze.

But your data could be in a different format and that's why I suggested you try to print out and understand the data flow so you can change the shapes accordingly.

[rmothukuru]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

[tapos12]

Hi, I am still having the same issue when cases like sample size = batch_size + 1 either in training and validation set.

I followed the strategy as @vbardiovskyg mentioned but still producing same error.

[raj-shah14]

I am facing the same issue