All 17 comments
There is no cross validator that exists currently in brain, but I actually am needing one as well. I think the proposal is simple and to the point, it'd be to simply add something like:
network.train(trainingSet, {
crossValidate :
{
testSize: 0.4,
testError: 0.02
}
});
robertleeplummerjr
on 2 Aug 2018
I'm actually dead wrong! https://github.com/BrainJS/brain.js/blob/11324b7dead480577a41420da8cbd32c3cd06684/src/cross-validate.js It does exist. @mubaidr what do you think about evaluating unit tests on it and ensuring it gets properly documented?
robertleeplummerjr
on 2 Aug 2018
Sure, will do.
mubaidr
on 2 Aug 2018
As it exists, we currently have this as the api:
brain.crossValidate(brain.NeuralNetwork, trainingData, opts, trainingOpts);
Because of the 1. amount of arguments, 2. that this isn't a documented feature, and 3. the existing api, what if we 1. shortened arguments to a single one, and 2. provided a means of passing in either a constructor or a instance of the neural network. Proposal:
brain.crossValidate({
constructor: brain.NeuralNetwork, // use this
instance: new brain.NerualNetwork() // or use this, but not both
data,
trainingOpts,
opts,
folds
});
I think we want to have the winning network as a result. Perhaps something like keyNetwork?
robertleeplummerjr
on 3 Aug 2018
I ended up finding a much more dynamic means by using a class and instance rather than using raw json and passing the classifier around. It is up for code review, and is as well documented therein.
robertleeplummerjr
on 20 Sep 2018
Documentation here, we may make a couple tweaks to it in the next few days, but it is now supported and part of the family.
robertleeplummerjr
on 21 Sep 2018
To clarify, this is the existing api as it has landed:
const crossValidate = new brain.CrossValidate(brain.NeuralNetwork);
const stats = crossValidate.train(data, networkOptions, trainingOptions);
const net = crossValidate.toNetwork();
// optionally later
const json = crossValidate.toJSON();
const net = crossValidate.fromJSON(json);
I think one minor thing we want to do with it is this (proposal):
const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, networkOptions);
const stats = crossValidate.train(data, trainingOptions);
const net = crossValidate.toNetwork();
The reason is so we potentially could use cross validate with something like TrainStream (proposal):
const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, networkOptions);
const stream = new brain.TrainStream({ neuralNetwork: crossValidate ...});
This way the options for the net are attached to the instance, can be reused, and now we can handle REAL BIG DATA in the train stream using cross validate.
Also, somewhat related, very little is now preventing us from using any other network in TrainStream, so it follows the flow of using tiny peaces to do amazing things without much work.
robertleeplummerjr
on 21 Sep 2018
Looks good. Optiins are passed with theor relative user.
mubaidr
on 21 Sep 2018
Dear all,
when trying to run the example code on branch v1.x I am constantly getting this error:
(node:2888) UnhandledPromiseRejectionWarning: ReferenceError: options is not defined
at CrossValidate.train (/server/node_modules/brain.js/dist/cross-validate.js:141:41)
It happens when I try to run crossValidate.train(..):
const crossValidate = new brain.CrossValidate(brain.NeuralNetwork, {});
const stats = crossValidate.train(probes, {});
deusama
on 27 Sep 2018
@deusama A fix is on the way! Expect new release soon: https://github.com/BrainJS/brain.js/pull/280
mubaidr
on 27 Sep 2018
The fix works for me, but I get another error when trying to read 'trainingData' from the cross validation example code:
(node:3000) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'input' of undefined
at NeuralNetwork._formatData (/server/node_modules/brain.js/dist/neural-network.js:846:27)
at NeuralNetwork._prepTraining (/server/node_modules/brain.js/dist/neural-network.js:513:19)
at NeuralNetwork.train (/server/node_modules/brain.js/dist/neural-network.js:545:33)
at CrossValidate.testPartition (/server/node_modules/brain.js/dist/cross-validate.js:40:38)
at CrossValidate.train (/server/node_modules/brain.js/dist/cross-validate.js:141:27)
It seems there is something wrong with the data format expected by train().
deusama
on 27 Sep 2018
Looking into this.
robertleeplummerjr
on 27 Sep 2018
@deusama can you put a small jsfiddle or similar together demoing the problem?
robertleeplummerjr
on 27 Sep 2018
deusama
on 28 Sep 2018
@robertleeplummerjr data is an object but an array method is being called on it:
https://github.com/BrainJS/brain.js/blob/5797b875cb7aa1c6e58ea748bd8795019ed1e9d1/src/cross-validate.js#L116
mubaidr
on 28 Sep 2018
Added & Fixed in: https://github.com/BrainJS/brain.js/commit/eebe6c89a27bfeaa4f4302ee6f4f67bd4857399e & https://github.com/BrainJS/brain.js/pull/281
mubaidr
on 3 Oct 2018
Related issues
iSumitBanik
路
4Comments
dan-ryan
路
4Comments
robertleeplummerjr
路
6Comments
lucaspojo
路
5Comments
lynxionxs
路
5Comments
Most helpful comment
To clarify, this is the existing api as it has landed:
I think one minor thing we want to do with it is this (proposal):
The reason is so we potentially could use cross validate with something like TrainStream (proposal):
This way the options for the net are attached to the instance, can be reused, and now we can handle REAL BIG DATA in the train stream using cross validate.
Also, somewhat related, very little is now preventing us from using any other network in TrainStream, so it follows the flow of using tiny peaces to do amazing things without much work.