Hub: ELMo text preprocessing (Is lower-casing needed?)

[iliaschalkidis]

Hi,

In your tutorial, you have the following usage example:

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
embeddings = elmo(
["the cat is on the mat", "dogs are in the fog"],
signature="default",
as_dict=True)["elmo"]

Relying on this example I have the following question: Do we need to provide lowercased text (sentences) in the module or not?

I am not sure if you demonstrated lowercased texts randomly or this it what we need to do in order to get the best of results. The module works either-way with or without lowercasing the text.

It is not clear whether the hosted tensorflow module implements the character-level encoder or the token-level encoder, which are both supported in AllenNLP's tensorflow implementation (https://github.com/allenai/bilm-tf).

Thanx!

[timpal0l]

If you try both, which yields the best performance? I do not use lowercase when using this module, however, I am curious as well if this is the right way to do it.

[iliaschalkidis]

If you try both, which yields the best performance? I do not use lowercase when using this module, however, I am curious as well if this is the right way to do it.

@timpal0l I'm going to run experiments for days. So I'm looking for a definite answer from either TensorHub or AllenNLP people, who curated this module, in order to model my inputs correctly. As I say both representations, mixed-cased and lower-cased, work and do not throw errors, which means that at least uppercase letters are handled correctly in the integrated text splitting function.

BUT if the model has not be pre-trained with both uppercase and lowercase letters, then most possibly the module will return embeddings with noise produced by the uppercase letters' embeddings....

[timpal0l]

@iliaschalkidis I see. You could always benchmark on some standard dataset first, and see what yields the best performance to figure out this hyperparameter. I am quite sure it's not case sensitive, since it's not mentioned in their paper, nor in any tutorial when using ELMo. However, am not :100: % sure.

[iliaschalkidis]

@timpal0l Me too, I'm pretty much sure it's not case sensitive, since in the initial code, both the data loader (https://github.com/allenai/bilm-tf/blob/master/bilm/data.py) does not include any function connected to lowering and the test data (https://github.com/allenai/bilm-tf/blob/master/tests/fixtures/data/vocab_test.txt) are not lower-cased... I would also suspect that if we needed lower-casing, there would be a correlated error when we call the module with mixed-cased text.

[vbardiovskyg]

Hi, I got a reply from the original author, saying: "The module does not do any lowercasing, since the original ELMo release did not do it either."

Additionally: the 1 billion word benchmark contains uppercase letters and the vocabulary too. So there is no reason to lowercase.