Deeplearning4j: XOR simple nn model fits on 0.5.0 and does not fit on 0.6.0

[AlexanderSerbul]

Good day!

Windows 10, java 7, eclipse mars.

Updated to 0.6.0 and simple example XOR nn model does not works.

Maven configs for 0.5 and 0.6:
https://gist.github.com/AlexanderSerbul/0b44ad2d59e5fc70bd2101bab9b9f1d7
https://gist.github.com/AlexanderSerbul/838a20f7e7007e017d6f9bf0c22387e7

Simple full test code:
https://gist.github.com/AlexanderSerbul/3322efe6b493bea920fdc738aaa1ec60

Output of code, working on 0.5 and not working on 0.6:
https://gist.github.com/AlexanderSerbul/05dd336b5a6dc95ca9489805e8eb62d1

And this code also works OK on 0.5 and does not work on 0.6, too: https://gist.github.com/AlexanderSerbul/5747cb9286b282063b91f05876b0822b

Thank you!

[alexandre-a-denis]

Hi, I observed that when using the "sigmoid" activation function for the output layer (as found in the basic XorExample), it does not reach 100%, while it does when switching to "softmax" function.

[AlexDBlack]

Fixed here: https://github.com/deeplearning4j/dl4j-examples/pull/248

Turns out the good behaviour on 0.5.0 was due to an incorrect implementation of negative log likelihood when using sigmoid activation function. That has since been fixed with the loss function rewrite as of 0.6.0.

The "bad" performance with 0.6.0 is due to the fact that the loss and activation functions are mismatched. With NLL + sigmoid, the muli(labels) means that there is no score/gradient for the label = 0 case. Consequently, both outputs are driven to 1.0 for all inputs/labels - for that loss/activation function combo, that is indeed a local minima. Correct config is softmax + mcxent/nll, or sigmoid + binary xent (former is preferred).

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.