OpenBLAS generates random result in MXNet forward
Recently in MXNet there's an issue about random model output when given fixed model and input, if computation happens on CPU with OpenBLAS: https://github.com/apache/incubator-mxnet/issues/11853
We found it is reproducible with OpenBLAS 0.3.1, but not reproducible with 0.2.20. It is easier to reproduce on AWS p3.16xlarge instance with Intel(R) Xeon(R) CPU E5-2686 v4.
I don't have enough knowledge to track down to the breaking point. Instead I searched a bit and now guess it may be related with https://github.com/xianyi/OpenBLAS/issues/1666 and https://github.com/JuliaLang/julia/issues/27960.
hetong007
All 5 comments
If you read julia discussion in your own link it is not reproducible with 0.3.0 and devel, i.e can you confirm that only 0.3.1 is broken?
brada4
on 27 Jul 2018
0.3.2 should be released this weekend. You could try the current "develop" branch (or just apply the changes from PR #1667 to 0.3.1 - you can get them in patch format by appending ".diff" to the PR number in the github interface) to see if the fix from #1666 actually solves your problem.
martin-frbg
on 27 Jul 2018
@martin-frbg Thanks for the information. I tested it and seems 0.3.2.dev is good.
hetong007
on 27 Jul 2018
@martin-frbg looking forward to the release
szha
on 30 Jul 2018
Closing as 0.3.2 is released now.
martin-frbg
on 30 Jul 2018
Related issues
boegel
路
5Comments
winpassuser
路
8Comments
gerritholl
路
7Comments
cefengxu
路
7Comments
wyphan
路
5Comments
Most helpful comment
0.3.2 should be released this weekend. You could try the current "develop" branch (or just apply the changes from PR #1667 to 0.3.1 - you can get them in patch format by appending ".diff" to the PR number in the github interface) to see if the fix from #1666 actually solves your problem.