Flair: Generate text with language model

[djstrong]

Is there any script that can generate text from trained language model?
Also calculating perplexity on testing data would be helpful.

[alanakbik]

Hi @djstrong you can point the following script to a language model that you've trained with Flair:

import torch
from flair.models import LanguageModel

dataset = '/path/to/your/lm/best-lm.pt'

# load the language model
model = LanguageModel.load_language_model(dataset)

idx2item = model.dictionary.idx2item

# initial hidden state
hidden = model.init_hidden(1)
input = torch.rand(1, 1).mul(len(idx2item)).long()

# generate text character by character
characters = []
number_of_characters_to_generate = 2000
for i in range(number_of_characters_to_generate):
    prediction, rnn_output, hidden = model.forward(input, hidden)
    word_weights = prediction.squeeze().data.div(1.0).exp().cpu()
    word_idx = torch.multinomial(word_weights, 1)[0]
    input.data.fill_(word_idx)
    word = idx2item[word_idx].decode('UTF-8')
    characters.append(word)

    if i % 100 == 0:
        print('| Generated {}/{} chars'.format(i, number_of_characters_to_generate))

# print generated text
print(''.join(characters))

This generates 2000 characters. I think in a future release, we will add such code directly to the LanguageModel class to make this easier!

[djstrong]

Thank you, it works. However, after more than 100 characters there is CUDA error: out of memory.
Edit: A solution to this problem is with torch.no_grad(): before the loop.

[alanakbik]

Ah, thanks for this! In fact, looks like we might want to add this in other parts of the framework as well, since autograd is not needed at evaluation or prediction time.

[stefan-it]

@alanakbik I tried an own language model, but the generation method above gives the following error message:

Traceback (most recent call last):
  File "generate.py", line 19, in <module>
    prediction, rnn_output, hidden = model.forward(input, hidden)
  File "/mnt/flair/flair/models/language_model.py", line 66, in forward
    encoded = self.encoder(input)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 110, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/functional.py", line 1110, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument #3 'index'

Using the latest version from master, can you help?

[alanakbik]

Hello @stefan-it it looks like the model is on cuda while the tensors that are being passed are normal non-cuda tensors. There's two ways to address this:

• Put the language model on cpu by calling .cpu() after you've initialized the model, i.e.
  model = model.cpu()
• Put the input tensor to cuda. You can do this by calling .cuda() on the input tensor when you create it, i.e.
  input = torch.rand(1, 1).mul(len(idx2item)).long().cuda()

Does this work for you? When we add this method officially, we'll work out all the cuda stuff :)

[stefan-it]

Thanks Alan, this is working for me:

import torch
from flair.models import LanguageModel

dataset = 'best-lm.pt'

# load the language model
model = LanguageModel.load_language_model(dataset)

model = model.cuda()

idx2item = model.dictionary.idx2item

# initial hidden state
hidden = model.init_hidden(1)
input = torch.rand(1, 1).mul(len(idx2item)).long().cuda()

# generate text character by character
characters = []
number_of_characters_to_generate = 500
for i in range(number_of_characters_to_generate):
    prediction, rnn_output, hidden = model.forward(input, hidden)
    word_weights = prediction.squeeze().data.div(1.0).exp().cpu()
    word_idx = torch.multinomial(word_weights, 1)[0]
    input.data.fill_(word_idx)
    word = idx2item[word_idx].decode('UTF-8')
    characters.append(word)

    if i % 100 == 0:
        print('| Generated {}/{} chars'.format(i, number_of_characters_to_generate))

# print generated text
print(''.join(characters))

[mhham]

Ah, thanks for this! In fact, looks like we might want to add this in other parts of the framework as well, since autograd is not needed at evaluation or prediction time.

The CUDA out of memory issue for generate_text is still present in release 0.3.2, FYI

[alanakbik]

@mhham Thanks for the pointer! I'll open an issue!