Pytorch-lightning: Gpus=1 + precision not working when using only certain layers
🐛 Bug
Making a finetuning model where the backbone isn't training breaks 16-bit.
williamFalcon
All 3 comments
Hi! thanks for your contribution!, great first issue!
![github-actions[bot] picture](https://avatars2.githubusercontent.com/in/15368?v=4&s=40)
github-actions[bot]
on 4 Nov 2020
😄3
Just to keep all the details here, this seems to be a side effect of amp. When we call self.trainer.scaler.step(optimizer) internally the scaler does an inf check on the optimizer's parameters, which is the assertion being thrown. This check needs to coincide with ensuring that the parameters are even updated within this step.
SeanNaren
on 5 Nov 2020
@SeanNaren follow up with pytorch team
edenlightning
on 17 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
versatran01
·
3Comments
edenlightning
·
3Comments
awaelchli
·
3Comments
as754770178
·
3Comments
srush
·
3Comments
Most helpful comment
Hi! thanks for your contribution!, great first issue!