Brain.js: Wrong XOR result with relu activation

[panzad3]

What is wrong?

When using relu activation function to solve XOR, does not matter how many iterations I train through I keep getting randomized results from absolute 0 to correct ( 0 , 0.453 , 0.834 , 0.98 , sometimes even 1.03 when expected should be close to 1 or below as I understand).. changing learning rate or the number of hidden layers does not affect the result ether. Maybe I'm just doing something wrong and it's not a bug but a feature, but I can't seem to figure out what

Where does it happen?

When using node.js for training and switching sigmoid to relu function

How do we replicate the issue?

-npm install brain.js
-copy-paste xor solver from GitHub
-change sigmoind to relu activation
-fire it up

Expected behavior (i.e. solution)

the expected behavior is XOR - (1,0) = 1

Version information

2.0 beta

Nodejs:

node version 10.14.2, I tried updating node to a latest version (updating gl too) but its makes no diff

How important is this (1-5)?

5

Other Comments

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.88. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[AlexanderSotoC]

I think is due to weight initialization, check this paper: Learning XOR: exploring the space of a
classic problem.

[robertleeplummerjr]

Relu has a really tough time with handling _exactly_ 0. I've run into this thinking it was a deficiency in the library, but found out you just need to normalize the value into something the neural network can handle. Try using a value that is _essentially 0_. Like 0.0000001.

[panzad3]

Learning XOR: exploring the space of a
classic problem

It's more like a book than a paper :D not sure I want to read it all as it states in the beginning that perceptrons are unable to solve XOR, where this in not correct as I understand and proven in 1991 paper about Gaussian Activation Function Perceptron (Bell-shaped function that can separate plane in two fields) https://www.researchgate.net/publication/224649299_Gaussian_perceptron_experimental_results

Thanks Robert, I will try it out, you are referring to sort of dying relu problem I guess.. will explore thanks.

[panzad3]

Relu has a really tough time with handling _exactly_ 0. I've run into this thinking it was a deficiency in the library, but found out you just need to normalize the value into something the neural network can handle. Try using a value that is _essentially 0_. Like 0.0000001.

You were totally correct sir, I did a quick proof of concept Arkanoid game https://codepen.io/sanchopanza/pen/eYJRjvm and taught it to play with all possible activation functions.
All of them work as intended on solving this linear problem with the randomly generated training set. will try the same on something non-linear too. Even tho I could not find many references online to Relu having problems with 0, I should probably read more.