Brain.js: [help wanted] training via stream

[djabdou50]

hi, how can i train via events like fs.createReadStream ? is it suitable to use it like this ?

let net = new brain.NeuralNetwork();
.on('data', (data) => {
net.train(data);
}

thank you !

[michaelonubogu]

If I understand what you are trying to do, it seems you are trying to stream your data from file into your brainjs neural net (may perhaps because your training data is too large to load all at once into memory).

I don't know if this is the case for you, but I had a similar problem. What I ended up doing is the following:

I stream the data using a readStream - I have this created in my readInputs() function

const readStream = fs.createReadStream(`sample-data.json`);

and create a brainjs.TrainStream

const neuralNet: brainjs.NeuralNetwork = new brainjs.NeuralNetwork();
const trainStream = new brainjs.TrainStream(trainStreamOptions);

const trainStreamOptions: brainjs.ITrainStreamOptions = {
      neuralNetwork: neuralNet,
      neuralNetworkGPU: neuralNet,
      floodCallback: () => {
        this.readInputs(trainStream);
      },
      doneTrainingCallback: (state) => {
        console.log('Finished Training Neural Net!');
      }
    };

this.readInputs(trainStream);

and then on each read event (inside my readInputs() function), stream that data to brainjs - brainjs has a nice functionality which allows you to create a TrainStream. This is what it looks like in my logic:

public readInputs(trainStream: brainjs.TrainStream) {
    // stream data from file
    const readStream = fs.createReadStream(`sample-data.json`);

    oboe(readStream)
      .node('!.*', ((data) => {

        const inputData = this.prepInputData(data);
        trainStream.write(inputData);

        return oboe.drop;
      }).bind(this))
      .done((empty: any) => {
        trainStream.endInputs();
        console.log('Finished streaming JSON from file');
      });
  }

In my case, I'm using oboe.js to stream JSON objects from file - that way I don't need to write that logic to construct a JSON object line by line. You can use the readStream directly as well.

[djabdou50]

this is exactly what i want ! thank you :1st_place_medal: