Farm: Which head to use for question generation?

[zzj0402]

Question
I am trying to make an ALBERT based model for question generation task. Which downstream head should I use?

An example would be greatly appreciated.
Additional context

import tensorflow as tf
from transformers import AlbertTokenizer, TFAlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = TFAlbertForMaskedLM.from_pretrained('albert-base-v2')
input_ids = tf.constant(tokenizer.encode("This is a test!"))[
    None, :]  # Batch size 1
outputs = model(input_ids)
prediction_scores = outputs[0]
outputTokens = tf.math.argmax(prediction_scores, axis=2)
outputTokens = tf.keras.backend.eval(outputTokens[0])
outputTokens = tokenizer.decode(outputTokens)
print(outputTokens)

I am thinking making a language modelling task and FARM looks easy to work with.

[Timoeller]

Hey thanks for checking out FARM.
I think question generation is a very interesting use case for doing QA (question matching) at scale.
Unfortunately ALBERT is not usable for text generation (of any kind) and FARM does not support tensorflow as of now.
If you are interested in extractive QA at scale check out https://github.com/deepset-ai/haystack

The newly released ELECTRA model can generate text and we are currently working on implementing it in FARM. Maybe you find the question generation functionality or get help setting it up in https://github.com/huggingface/transformers

[zzj0402]

https://www.aclweb.org/anthology/D19-5821.pdf

This paper uses a masked language model to generate questions. They use a dense layer with softmax to predict the vocabulary tokens, recursively feeding tokens as input.

Maybe we could include in FARM? What I have tried is training a MLM head after ALBERT pretrained model using Huggingface.

Question generation can definitely help QA. Thanks for sharing the Haystack.

[Timoeller]

Thanks @ViktorAlm for pointing this out : ) To clarify my previous statement: Electra only has a small generator part for generating tokens for pre-training the whole model. I do not know how well this generator would be suited for actual question generation.

And thanks @zzj0402 for the more thorough explanation and link to the EMNLP paper. Now I better understand what you want to do. This Bert based MLM approach for language modelling might work in such a restricted setting of generating just a question. I know of a bachelor student who successfully used Bert and Farm to fill in gaps for (ancient Greek) text restoration. You find the code here. There is also a very experimental branch on beam search, which Chan and Fan apparently used for some of their models.

[zzj0402]

Hi @Timoeller, I finished the Question Generation prototype and going to publish API by August =) https://www.patreon.com/posts/question-36863875

[Timoeller]

Nice, congrats! What models/frameworks did you use in the end?

Will you also create a little web demo where people can test out your solution?

---

I am closing this issue now.

[zzj0402]

Nice, congrats! What models/frameworks did you use in the end?

Will you also create a little web demo where people can test out your solution?

I am closing this issue now.

Yes, I am working on it. I code up a transformer based seq-to-seq model. Will update the web demo here when it's done.