Brain.js: Resuming training on Neural Network

[knapcio]

Hi, Neural Network gives me weird results. Maybe I miss something. Please help.

What is wrong?

Neural Network result is incorrect.

Where does it happen?

I'm running 2.0.0-alpha.12 on Node 13 and Mac.

How do we replicate the issue?

Firstly I tried:

const brain = require('brain.js');

const net = new brain.NeuralNetwork();

const trainingData = {
    input: { a: 0, b: 1 },
    output: { good: 1 },
};

net.train(trainingData);

const trainingData2 = {
    input: { a: 1, b: 0 },
    output: { bad: 1 },
};

net.train(trainingData2);

const result = net.run({ a: 0, b: 1 });

console.log(result);
// Output: { good: 0.060428500175476074 }
// I expect: { good: 1 }

Then tried serialising as advised but still doesn't work:

const brain = require('brain.js');

// first  training
const trainingData = {
    input: { a: 0, b: 1 },
    output: { good: 1 },
};
const net = new brain.NeuralNetwork();
net.train(trainingData);
const stringifiedNet = JSON.stringify(net.toJSON());

// second training
const trainingData2 = {
    input: { a: 1, b: 0 },
    output: { bad: 1 },
};
const net2 = new brain.NeuralNetwork();
net2.fromJSON(JSON.parse(stringifiedNet));
net2.train(trainingData2);
const stringifiedNet2 = JSON.stringify(net2.toJSON());

// prediction
const net3 = new brain.NeuralNetwork();
net3.fromJSON(JSON.parse(stringifiedNet2));
const result = net3.run({
    a: 0,
    b: 1,
});
console.log(result);
// Output: { good: 0.07364676147699356 }
// I expect: { bad: 0, good: 1 }

const result2 = net3.run({
    a: 1,
    b: 0,
});
console.log(result2);
// Output: { good: 0.07043380290269852 }
// I expect: { bad: 1, good: 0 }

How important is this (1-5)?

4

Expected behavior (i.e. solution)

I expect NN to return other results (described in code comments).

Other Comments

You guys do an awesome job building brain.js!

EDIT: Seen this comment on StackOverflow: The keepNetworkIntact has been renamed to reinforce but can't see this property in INeuralNetworkTrainingOptions

EDIT2: I read somewhere I should pass all data every time I run train. Is there any way around it? I train the NN every day with new daily statistics. Seems insane to have to calculate them every time for all the previous days (i.e. past 2 years).

[robertleeplummerjr]

Your training data is in the wrong format. Supply an array of objects.

const trainingData = [{
    input: { a: 0, b: 1 },
    output: { good: 1 },
}];

net.train(trainingData);

[knapcio]

Thanks for the prompt response. I really appreciate. it.

Sorry, I also tried with arrays and still doesn't work:

const brain = require('brain.js');

const net = new brain.NeuralNetwork();

const trainingData = [{
    input: { a: 0, b: 1 },
    output: { good: 1 },
}];

net.train(trainingData);

const trainingData2 = [{
    input: { a: 1, b: 0 },
    output: { bad: 1 },
}];

net.train(trainingData2);

const result = net.run({ a: 0, b: 1 });

console.log(result);
// Output: { good: 0.0637415423989296 }

and the second way:

const brain = require('brain.js');

// first  training
const trainingData = [{
    input: { a: 0, b: 1 },
    output: { good: 1 },
}];
const net = new brain.NeuralNetwork();
net.train(trainingData);
const stringifiedNet = JSON.stringify(net.toJSON());

// second training
const trainingData2 = [{
    input: { a: 1, b: 0 },
    output: { bad: 1 },
}];
const net2 = new brain.NeuralNetwork();
net2.fromJSON(JSON.parse(stringifiedNet));
net2.train(trainingData2);
const stringifiedNet2 = JSON.stringify(net2.toJSON());

// prediction
const net3 = new brain.NeuralNetwork();
net3.fromJSON(JSON.parse(stringifiedNet2));
const result = net3.run({
    a: 0,
    b: 1,
});
console.log(result);
// Output: { good: 0.060673292726278305 }

const result2 = net3.run({
    a: 1,
    b: 0,
});
console.log(result2);
// Output: { good: 0.070485919713974 }

[robertleeplummerjr]

Reproduced this issue here: https://observablehq.com/@robertleeplummerjr/brain-js-issue-514

[knapcio]

I actually believe there are 2 different issues.

The first one is what you described on observablehq - undefined output value.

The second one is resuming a training. Please consider the following code:

const brain = require('brain.js');

// first  training
const trainingData = [{
    input: { a: 0, b: 1 },
    output: { good: 1, bad: 0.1 },
}];
const net = new brain.NeuralNetwork();
net.train(trainingData);
const stringifiedNet = JSON.stringify(net.toJSON());

// second training
const trainingData2 = [{
    input: { a: 1, b: 0 },
    output: { bad: 1, good: 0.1 },
}];
const net2 = new brain.NeuralNetwork();
net2.fromJSON(JSON.parse(stringifiedNet));
net2.train(trainingData2);
const stringifiedNet2 = JSON.stringify(net2.toJSON());

// prediction
const net3 = new brain.NeuralNetwork();
net3.fromJSON(JSON.parse(stringifiedNet2));
const result = net3.run({
    a: 0,
    b: 1,
});
console.log(result);
// Output: { good: 0.14621490240097046, bad: 0.9085853099822998 }

const result2 = net3.run({
    a: 1,
    b: 0,
});
console.log(result2);
// Output: { good: 0.1430644392967224, bad: 0.9107764959335327 }

Even if I provide the full output in every training (both good and bad) it still gives a wrong result in the first prediction (good should be closer to 1 when bad should be small)

[ehortle]

Hi guys, I have been trying brain.js and am experiencing this as well.

This seems to be falling over at the first hurdle - it seems basically to be an error in the most elementary case, and for my implementation of brain.js it's important that this works. Does this basically mean that for all live installations of brain.js, the neural network is not actually working as specified in the docs? This seems to be an extremely high priority bug to me.

Hoping you can comment on the possibility of a fix soon and thanks for all the work on the project :smile:

[robertleeplummerjr]

It is a very high priority issue, and I plan on looking at it tomorrow morning.

[robertleeplummerjr]

I am not sure I see any issue, though I've tried to reproduce what you mention @knapcio: https://observablehq.com/@robertleeplummerjr/brain-js-issue-514/2

You mention:

Even if I provide the full output in every training (both good and bad) it still gives a wrong result in the first prediction (good should be closer to 1 when bad should be small)

Yet your training data is exactly:

const trainingData2 = [{
    input: { a: 1, b: 0 },
    output: { bad: 1, good: 0.1 },
}];

Which seems to correlate with an a of 1 should produce a high bad and a low good.

When you call:

net2.train(trainingData2);

Are you expecting the previous training data to perhaps _remain intact_ inside the net? If so, you need to simply do this:

net2.train([...trainingData, ...trainingData2]);

[robertleeplummerjr]

@ehortle can you share your use example for more context?

[knapcio]

Ok, I see, so it seems like it's just not possible to resume a training on an already trained net.

Are you expecting the previous training data to perhaps remain intact inside the net?

Not sure what it exactly means, but maybe yes. Here you mention I could achieve it with toJson, but it doesn't seem to work.

As I mentioned in EDIT2: I need to train my network on a daily basis. Every day, I have to pull some data from that single day and modify the current NN. It's impossible to train the network every day from scratch - too much data. I need to simply append the new training result to a previous one.

I don't know the brain.js implementation, but this is how I'd imagine it should work:

1. User trains NN which saves weights and number of samples (maybe it could save some kind of weight of the whole training).
2. Saves NN to JSON.
3. Then user can recreate the NN from JSON any time.
4. Now user can train the NN again and the new training output (weights, sample count etc) is merged with a previous one (possibly using previously mentioned weight of each training)
5. User runs a "prediction" function and it predicts basing on all past trainings.

Hope it all makes sense.

[GELight]

@knapcio
That is exactly the same thing that I need too.
I need a way to make my trained net better and better with time.

@robertleeplummerjr
How @knapcio said: "It's impossible to train the network every day from scratch - too much data. I need to simply append the new training result to a previous one."

If it is possible to expand brain.js with this functionality, that would be a great added value for all users. :)

HOPE SO ... :)

[robertleeplummerjr]

@GELight This is supported, but if new neurons (in this case, aka object keys) are added, this breaks the network's design.

Are all the keys of the object you are feeding in known in advance?

To achieve a network that can train to learn new things:

1. after training, you need to obtain the json of the network, and put it in your storage.
2. when new data is being used to train, append it to the already existing training set.
3. instantiate the network using the json, net.fromJSON(json)
4. then just train the net.

If you find this functionality not working, then there is a bug, and if you could include the scripting used to find that bug, I can get a quick fix in.

[kepelrs]

Hi guys,

Thanks for the great work in this project.

I also seem to be having issues with fromJSON functionality. Here's the code:

const brain = require("brain.js");
const fs = require("fs");

// some Forrest Gump quotes
const trainingData = [
    "Stupid is as stupid does.",
    "My mama always said, ‘Life was like a box of chocolates. You never know what you’re gonna get.'",
    "Run, Forrest! Run!",
    "You have to do the best with what God gave you.",
    "My Mama always said you’ve got to put the past behind you before you can move on.",
    "Mama always said, dying was a part of life. I sure wish it wasn’t.",
    "Listen, you promise me something, OK? Just if you’re ever in trouble, don’t be brave. You just run, OK? Just run away.",
    "What’s normal anyways?",
    "Mama always had a way of explaining things so I could understand them.",
    "Bubba: My given name is Benjamin Buford Blue, but people call me Bubba. Just like one of them ol’ redneck boys. Can you believe that?",
    "My name’s Forrest Gump. People call me Forrest Gump.",
    "Anyway, like I was sayin’, shrimp is the fruit of the sea. You can barbecue it, boil it, broil it, bake it, saute it. There’s shrimp-kabobs, shrimp creole, shrimp gumbo. Pan fried, deep fried, stir-fried. There’s pineapple shrimp, lemon shrimp, coconut shrimp, pepper shrimp, shrimp soup, shrimp stew, shrimp salad, shrimp and potatoes, shrimp burger, shrimp sandwich. That, that’s about it.",
    "Lieutenant Dan got me invested in some kind of fruit company [Apple computer]. So then I got a call from him, saying we don’t have to worry about money no more. And I said, that’s good! One less thing.",
    "That day, for no particular reason, I decided to go for a little run. So I ran to the end of the road. And when I got there, I thought maybe I’d run to the end of town. And when I got there, I thought maybe I’d just run across Greenbow County. And I figured, since I run this far, maybe I’d just run across the great state of Alabama. And that’s what I did. I ran clear across Alabama. For no particular reason I just kept on going. I ran clear to the ocean. And when I got there, I figured, since I’d gone this far, I might as well turn around, just keep on going. When I got to another ocean, I figured, since I’d gone this far, I might as well just turn back, keep right on going.",
    "One day it started raining, and it didn’t quit for four months. We been through every kind of rain there is. Little bitty stingin’ rain… and big ol’ fat rain. Rain that flew in sideways. And sometimes rain even seemed to come straight up from underneath. Shoot, it even rained at night…",
    "Bubba was my best good friend. And even I know that ain’t something you can find just around the corner. Bubba was going to be a shrimping boat captain, but instead, he died right there by that river in Vietnam.",
    "Me and Jenny goes together like peas and carrots.",
    "The best thing about visiting the President is the food! Now, since it was all free, and I wasn’t hungry but thirsty, I must’ve drank me fifteen Dr. Peppers.",
];

const net = new brain.recurrent.LSTM();
// use previously trained model if exists
try {
    net.fromJSON(JSON.parse(fs.readFileSync(__dirname + "/book-model.json")));
} catch (e) {
    console.log("No pre-trained model available");
}

// train net
net.train(trainingData, {
    iterations: 100,
    errorThresh: 0.011,
    log: (stats) => {
        console.log(stats);
        console.log();
        console.log(net.run("Bubba"));
        console.log(net.run("Forrest"));
        console.log();
    },
});

// save trained models
fs.writeFileSync(__dirname + "/book-model.json", JSON.stringify(net.toJSON()));

First run ends with training error: 0.05573762835847086
Second run starts with training error: 1.1483814150826834 (expected to be the same or lower as end of first run)
Third run starts with training error: 0.4861999202810547 (seems like it is always random initial weights)

Maybe I'm doing something wrong?