Parlai: Adding new tokens ( words ) of a new custom dataset to pre-trained Blender model

[pedramsalimi]

Hi, recently I have fine-tuned a 90M parameter Blender on my own dataset but I got poor results on f1 score and accuracy so I think my dataset words were not added in the pre-trained dictionary.

Is there any command to add new words or tokens of the new dataset in the model dictionary?
Thanks.

[stephenroller]

No, not off the shelf. You can do it if you very carefully manipulate the state dict and use HuggingFace’s add_special_token function, but it will be very tedious.

The 2.7B model uses a byte level BPE dictionary that is a bit more robust to out of vocabulary words, so that could be one other way forward.

It’s worth using display_model to actually see what your generations look like: are they really generating a lot of unk tokens? Are you sure these are OOV problems? How is your ppl? I can frequently FT the 90M model into the low teens, and that’s quite good.

For comparison, our 90M model fine tuned in ConvAI2 with very carefully selected hyper parameters only gets .21 F1. Word overlap is a very coarse metric. See the Dodeca paper for details.

[pedramsalimi]

I've got 27 in ppl score in blender at testing time. And about generation when I was training a seq2seq model on my dataset it returned these for example:

[ running eval: test ]
TEXT: Wait, what?
PREDICTION: I ' m not sure how to be able to do this for you.
~
TEXT: I'm all ready to go.
PREDICTION: Yeah, yeah.
~
TEXT: That's where we're headed. Hope you don't mind a little company because I'm coming with you.
PREDICTION: You ' re right. You ' re right. You ' re right.

while the seq2seq has 11062 words it returned no UNK tokens as you see in the above example. so I think blender with 54944 vocab size won't have any problem, am I right?! although I didn't display generation in blender fine-tuning.

But here arises another question for me: when I start to train gpt2 model number of words is 4 !!
[ num words = 4 ]

is it going to be a problem? It should be noted that my training ppl at epoch 2 is around 15-20 at the moment.

[stephenroller]

27ppl is a pretty bad ppl for any of our blender variants. I suspect something is wrong with your training.

You should definitely run display_model and look at generations on the training (display_model.py -dt train:eval_mode) and valid (display_model.py -dt valid) sets.

The num words printed for the gpt2 model is a bug, it prints the incorrect number due to a hack in our implementation. It's actually somewhere around 50k.

[pedramsalimi]

Yes, you were right, thank you so much for your help and your great help. There were many duplications in the dataset (many similar texts had different labels!).
This time I have got much better ppl score (19 instead of 27).

I have another question before closing this issue.
I want to access my questions and also the model answers in order to keep them in a variable for further processing. Actually I mean when we use Interactive.py we can communicate with the model in the terminal and we don't access dialogues. I want to access the model responses and keep them for some further processing on it. Can I do it somehow?

[stephenroller]

Specifically in interactive? You could add a WorldLogger to it, like eval_model has already:

https://github.com/facebookresearch/ParlAI/blob/4e11d8c6a5a50ca9494d45f7a47eb51d3832b036/parlai/scripts/eval_model.py#L37-L52

You can add this line above the creation of the world:
https://github.com/facebookresearch/ParlAI/blob/4e11d8c6a5a50ca9494d45f7a47eb51d3832b036/parlai/scripts/eval_model.py#L113

and then after the parley(), you add this:

https://github.com/facebookresearch/ParlAI/blob/4e11d8c6a5a50ca9494d45f7a47eb51d3832b036/parlai/scripts/eval_model.py#L132-L133

And then a save at the very end:

https://github.com/facebookresearch/ParlAI/blob/4e11d8c6a5a50ca9494d45f7a47eb51d3832b036/parlai/scripts/eval_model.py#L147-L152

I think this would be useful many times, so if you do this, please create a PR so we can have the functionality for everyone.

[pedramsalimi]

ِYes sure, I would be so happy to create a PR!
So I did it, Could you please check it?
If there is any problem please consider that it was my first PR ever! :)

[pedramsalimi]

But what I would like to do actually is something like this in python:
For example following the parlai docs:

parser = ParlaiParser()
opt = parser.parse_args()

agent = RepeatLabelAgent(opt)
world_logger = WorldLogger(opt) if opt['save_world_logs'] else None 
world = create_task(opt, agent)

for _ in range(10):
  world.parley()
  if world_logger is not None: 
    world_logger.log(world) 
  print(world.display())
  if world.epoch_done():
    print('EPOCH DONE')
  if world_logger is not None: 
    # dump world acts to file 
    world_logger.reset()  # add final acts to logs 
    base_outfile = opt['report_filename'].split('.')[0] 
    outfile = base_outfile + f'_{task}_replies.jsonl' 
    world_logger.write(outfile, world, file_format=opt['save_format']) 
    answer = convert_answer_to_string(outfile) # I don't know the parameter should be outfile or what, I just want to show you what I want to do
    do_somthing_with_answer(answer) # Actually this is what I want to do but with my fine-tuned blender and my own dataset

Although it returns an error at the second line ( opt = parser.parse_args() ) : SystemExit: 2 error when calling parse_args()

All I want to do is accessing model answer in each turn, make it string and do some next operation on it.

[stephenroller]

try parse_args([])

[pedramsalimi]

I am trying to test some baseline in order to compare models. when I train my dataset on IR baseline and TFIDF I am not able to calculate ppl, In fact, I use the -vmt ppl command but it doesn't calculate ppl for me. Is there any way to do that?
Thanks.

[pedramsalimi]

And I have one other question:
Is there any way or command to do some t-test or significance tests on models?

[stephenroller]

TF-IDF cannot compute PPL, it's a metric specific to generative models. F1 or Bleu are the best metrics to compare across these different model types, but they are very noisy metrics.

Significance testing in machine learning is difficult. I like Wilcoxon signed rank tests, but sometimes people use t-tests. Generally, the field doesn't do much significance testing. We don't have this built into parlai already.

[stephenroller]

In this case, i meant “low teens” like the model performance it got was like 12-14 perplexity. Not that it mimicked teenagers. Have I misunderstood your request?

[hflserdaniel]

Oops, sorry for my misunderstanding. Thanks for your quick reply :)

[pedramsalimi]

Is there any way to plot the training loss or ppl progress of a trained model? or even plot the loss of two different trained models in order to compare them?
Thanks.

[stephenroller]

The easiest way is to use —tensorboard-log true during training, and then use tensorboard to look at the graphs. Otherwise, there is a “model.trainstats” file that contains all the validation scores over the full training run.

[hflserdaniel]

How many epochs and how long does it normally take to finetune on the 90M model? I've got a bunch of 1080Ti and a tiny dataset around 10K. Also, I wonder is it feasible to FT on the 3B or even 9B model? Thanks for your help.

[stephenroller]

If you have at least two 1080Ti, you should be able to fine tune the 2.7B with —model-parallel true

[hflserdaniel]

Thanks! I've run some trail FT on the 90M model with the finetune example code (on BST, ConvAI and WoW) for 0.15 epoch. However, the replies from the finetuned model seem to change dramatically:

[context]: your persona: i coexist well with my husband.
your persona: i'm a middle aged mother of two high school boys.
Enter Your Message: hello
[TransformerGenerator]: hello !
Enter Your Message: how are you
[TransformerGenerator]: i ' m good , how are you ?
Enter Your Message: i am fine
[TransformerGenerator]: i ' m glad to hear that .
Enter Your Message: what do you do?
[TransformerGenerator]: i ' m a teacher .
Enter Your Message: are you single or married?
[TransformerGenerator]: i am single .
Enter Your Message: what are your hobbies?
[TransformerGenerator]: i am a teacher .
Enter Your Message: What do you do when you are free?
[TransformerGenerator]: i am a teacher .

It looks like the model has become less engaging and tend to repeat some sentences. Is this normal?

[stephenroller]

That's not enough time to fine tune, you really will probably FT for several dozen epochs. For just 0.15 epochs, this is sorta what I expect.

That said, these models are always very repetitive. One of our findings in the paper was the --beam-context-block-ngram 3 --beam-block-ngram 3 --beam-min-length 20 args are very very important

[hflserdaniel]

Thanks for your advice. I see that the FT demo code for 90M sets --init-model zoo:tutorial_transformer_generator/model, which seems like a plain transformer. I've changed it and loaded model from blender_90M and the model is still engaging after FT :)

[stephenroller]

I'm going to close this issue now. Let's start a new issue if there should be further discussion.