Transformers: Updating simple_lm_finetuning.py for FP16 training

[crowegian]

in simple_lm_finetuning the recent updated code doesn't work with the old optimizer specifications.

When not running with --fp16
optimizer = BertAdam(optimizer_grouped_parameters, lr=args.learning_rate, warmup=args.warmup_proportion, t_total=num_train_optimization_steps) // In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this: optimizer = AdamW(model.parameters(), lr=args.learning_rate, correct_bias=False) # To reproduce BertAdam specific behavior set correct_bias=False scheduler = WarmupLi

fixes the problem as suggested.

But when running --fp16
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_train_optimization_steps) # PyTorch scheduler
will error out with FP16_Optimizer saying that

TypeError: FP16_Optimizer is not an Optimizer

Does the FuseAdam object need to be passed into WarmupLinearSchedule instead?

[crowegian]

Hi, has this been fixed? I've tried updating my language modeling script to match but still getting errors.

[Paethon]

Having the same problem at the moment ...

[Paethon]

I guess the preferred way is to use apex.amp like in this example?
https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.