Pytorch: BatchNorm shouldn't use Bessel's correction for a batch size of 1
Right now it makes everything NaN.
apaszke
>All comments
To be clear, for a N x C x H x W tensor, the problem is only when all of N, H, and W are 1. (So BatchNorm2d on batch size of 1 is OK as long as you don't have a 1x1 image).
What's the desired behavior? The only reasonable behavior I can think of is:
- Raise an exception when the dimensions of which you are normalizing are one or
- Output zero (+ optional affine transform)
I'm not sure the outputing zero is a good idea. I can't think of a case where that's what you want.
colesbury
on 27 Apr 2017
👍3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
miguelvr
路
3Comments
negrinho
路
3Comments
kdexd
路
3Comments
szagoruyko
路
3Comments
a1363901216
路
3Comments
Most helpful comment
To be clear, for a
N x C x H x Wtensor, the problem is only when all of N, H, and W are 1. (So BatchNorm2d on batch size of 1 is OK as long as you don't have a 1x1 image).What's the desired behavior? The only reasonable behavior I can think of is:
I'm not sure the outputing zero is a good idea. I can't think of a case where that's what you want.