Tfjs: Fit inside tidy raises exception if validationSplit is provided

[ToonTalk]

TensorFlow.js version

{tfjs-core: "0.13.11", tfjs-layers: "0.8.5", tfjs-converter: "0.6.7", tfjs: "0.13.5"}

Browser version

Version 70.0.3538.110 (Official Build) (64-bit)

Describe the problem or feature request

If either the tidy or the validationSplit (only changes to https://js.tensorflow.org/#getting-started) is missing then it works fine otherwise the following exception.

tfjs:2 Uncaught (in promise) TypeError: Cannot read property 'backend' of undefined
at e.readSync (tfjs:2)
at e.moveData (tfjs:2)
at Object.moveData (tfjs:2)
at e.get (tfjs:2)
at tfjs:2
at Array.map ()
at e.compileAndRun (tfjs:2)
at e.gather (tfjs:2)
at ENV.engine.runKernel.$x (tfjs:2)
at tfjs:2

Code to reproduce the bug / link to feature request

  const model = tf.sequential();
  model.add(tf.layers.dense({units: 1, inputShape: [1]}));

  // Prepare the model for training: Specify the loss and the optimizer.
  model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});

  // Generate some synthetic data for training.
  const xs = tf.tensor2d([1, 2, 3, 4], [4, 1]);
  const ys = tf.tensor2d([1, 3, 5, 7], [4, 1]);

  // Train the model using the data.
  tf.tidy(() => {
      model.fit(xs, ys, {epochs: 10,
                         validationSplit: .2}).then(() => {
        // Use the model to do inference on a data point the model hasn't seen before:
        // Open the browser devtools to see the output
        model.predict(tf.tensor2d([5], [1, 1])).print();
      });
});

[tafsiri]

tf.tidy requires that the function passed to it is synchronous. Model.fit is asynchronous and returns a promise.

Model.fit will also handle its own memory management so you shouldn't need tidy here.

[ToonTalk]

The function passed to tidyis synchronous in my code.

When I added some console.log(tf.memory()) statements and moved the tf.tensor2d([5], [1, 1]) out of the test I saw the number of tensors increase. Perhaps fit does handle its own memory but predict doesn't. But I prefer using thenwhich keeps them together.

[tafsiri]

Sorry I meant to say that you cannot call async functions within the function passed to tidy.

Try moving the tidy within the callback you pass to .then where you wrap just the predict function in tf.tidy.

[ToonTalk]

Thanks. When I move the tidyinside the thenthe exception no longer happens. But the number of outstanding tensors increases. But only if I use validationSplit: .2 - without it the number of tensors doesn't change in the following. With it I see this in the console

{unreliable: false, numBytesInGPU: 0, numTensors: 8, numDataBuffers: 6, numBytes: 48}
tfjs:2 Tensor
[[3.5859156],]
sample.html:32 {unreliable: false, numBytesInGPU: 8, numTensors: 10, numDataBuffers: 10, numBytes: 64}

with this code

      const model = tf.sequential();
      model.add(tf.layers.dense({units: 1, inputShape: [1]}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([1, 2, 3, 4], [4, 1]);
      const ys = tf.tensor2d([1, 3, 5, 7], [4, 1]);
      const test = tf.tensor2d([5], [1, 1]);
      // Train the model using the data.
      console.log(tf.memory());
      model.fit(xs, ys, {epochs: 10,
                         validationSplit: .2
                         }).then(() => {
          tf.tidy(() => {
              model.predict(test).print();
          });
          console.log(tf.memory());
       });

[tafsiri]

cc @caisq do you want to take a look and see if there is a memory leak when using validationSplit.

[bileschi]

Hi @ToonTalk , I'm able to replicate the behavior you describe, that when you call model.fit, the number of outstanding tensors increases from 8 to 10, but only when validation>0. Did you notice that when you call model.fit again, the count stays at 10? I suspect that there is some temporary storage allocated when the validation process happens the first time, and then retrained for future invocations. Is this a problem?

If so, we might be able to add cleanup here: https://github.com/tensorflow/tfjs-layers/blob/c785709a6eecb0515cc1e01853c2edacdac247dc/src/engine/training_tensors.ts#L518

[ToonTalk]

Thanks for looking into this. If it happens only once it is a minor problem that I can easily live with.

[cpury]

Hijacking this since it's the only related issue and top google result:

The docs repeat several times that you should only use the async variants of data and array. However, tf.tidy does not seem to handle anything async. So if you want to use the async variants, you need to handle clean up yourself.

It would be amazing if there'd be a tidyAsync variant or similar that cleans up after the given promise resolves.

Alternatively, some guidance on best practices etc. would also be great