Wav2letter: LibriSpeech Recipe not converging; loss explodes
I've been struggling for the last few weeks with getting the Recipe for LibriSpeech to work.
The loss is roughly 30 for a number of steps but then explodes.
TER always stays around 99%.
Was someone able to reproduce the results?
What could be the issue?
FredericGodin
>All comments
The params mentioned for Librispeech recipe was using 8 gpus to train.
If you are using different number of GPUs, you would have to play around with "-lr" and "-lrcrit" params to make it converge since the "effective" batchsize per iteration is different.
Assuming you make it work, we also typically do learning rate annealing to get the model converge even better.
Hope it helps !
vineelpratap
on 23 Jan 2019
👍4
Was this page helpful?
0 / 5 - 0 ratings
Related issues
JanX2
路
5Comments
bmblr497
路
5Comments
ghost
路
5Comments
mlexplore1122
路
3Comments
smolendawid
路
3Comments
Most helpful comment
The params mentioned for Librispeech recipe was using 8 gpus to train.
If you are using different number of GPUs, you would have to play around with "-lr" and "-lrcrit" params to make it converge since the "effective" batchsize per iteration is different.
Assuming you make it work, we also typically do learning rate annealing to get the model converge even better.
Hope it helps !