Hub: U.S.E. and ELMo - Unseen word vector representation

[truas]

Hello,

ELMo uses sub-words in their process, so I can see how vectors for "unseen words" are produced. For example if I try to obtain the vector for "dogETCabc" even without the embedding for the whole token its sub-word (chars) will produce something. However, when using U.S.E I still can obtain vector representations for any "text" I provide. Does this happen because of the Transformer model? - Since it uses Skip-thought vectors, which has a vocabulary expansion method to encode words not seen at training time. Am I correct to assume that in the DAN encoding this wouldn't be possible?

I'm trying to understand more details on how the vectors of "weird" tokens/text are produced in U.S.E. I could not find this in the documentation, the original paper or in this evaluation of U.S.E and ELMo link.

Regards,
T.

[chiragjn]

+1
Can we get a reliable method to determine whether some text is almost UNK for U.S.E. and it's large version? For the lite version, it is still possible to get unk id and check it using sentencepiece lib.

I tried obtaining a mean unk vector by vectorizing a bunch of random one word strings, it tends to be very similar to other one word random strings but significantly different than vectors of strings with different number of unk words.

[pyturn]

+1
I also want to know how U.S.E encodes unseen words. It can be possible that it assigns some kind of noise (random numbers) to the sentence that contains unseen words. And if it sees some another unseen words(sentence) next time, then it will again assign noise to those (this may be different or may not be). And when we evaluate similarity between these 2 sentences then, we get higher similarity score. So I need to know how USE assigns embedding to unseen words (sentences).

@truas Can you please explain how USE uses Skip-thought vectors, because as per my knowledge it uses transformers model to produce word embedding and then sum it up yo produce sentence embedding.

@chiragjn can you explain how to get unk id and what is sentencepiecelib ?

[chiragjn]

@pyturn The lite version https://tfhub.dev/google/universal-sentence-encoder-lite/2 uses https://github.com/google/sentencepiece lib, so you can probe the sentencepiece model by calling EncodeAsIds on gibberish text and see the output to get the unk id

import tensorflow as tf
import tensorflow_hub as hub
import sentencepiece as spm
with tf.Session() as sess:
    module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-lite/2")
    spm_path = sess.run(module(signature="spm_path"))
sp = spm.SentencePieceProcessor()
sp.load(spm_path)
print(sp.unk_id())
print(sp.EncodeAsIds(u"📋"))
print(sp.EncodeAsIds(u"📋 📭"))
print(sp.EncodeAsIds(u"qwerdtfghjkl"))

[30, 0]
[30, 0, 30, 0]
[30, 1111, 2252, 29, 17, 209, 3095, 785, 173, 139]

You can see that the unk_id is 0. SentencePiece breaks down text into subunits which means it can encode gibberish too (see last example) unless you give it text with subtokens it has never seen before, in that case it replaces them with unk_id which is 0

I also explored USE Large's graph in Tensorboard and extracted the vocab of size 200K but it uses a hash table of 400K buckets and loads embedding lookup matrix of size (600K, 320) from 17 shards.
So I think something else is going on when it encodes text. @andresusanopinto Can you please help in figuring out what is UNK for USE Large model ?

[chiragjn]

Just an update, I got my query answered here https://groups.google.com/a/tensorflow.org/forum/#!topic/hub/6ZH6UzKtwgA

The USE Large v3 model, uses an explicit ~200K vocabulary. Any word not in those 200K words gets hashed to one of the 400K buckets of a HashTable. An embedding is learned for each of these 400K buckets. So overall the embedding Tensor's first dimension is ~600K

[rmothukuru]

Closing this issue as it has been answered in this forum