Tfjs: TypeError: Cannot read property 'backend' of undefined - tensor is already disposed when moveData is called.

[sbrl]

I'm trying to setup an LSTM, but Tensorflow.js doesn't seem to like it very much. It keeps crashing with the following error:

Epoch 1 / 50
eta=0.0 --------------------------------------------------------- categoricalCrossentropy=1.04e-5 loss=1.04e-5 /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3280
        var srcBackend = info.backend;
                              ^

TypeError: Cannot read property 'backend' of undefined
    at Engine.moveData (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3280:31)
    at DataStorage.get (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:115:28)
    at NodeJSKernelBackend.getInputTensorIds (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:153:43)
    at NodeJSKernelBackend.executeSingleOutput (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:200:73)
    at NodeJSKernelBackend.reshape (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1055:21)
    at forward (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:7430:24)
    at /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3480:55
    at /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3319:22
    at Engine.scopedRun (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3329:23)
    at Engine.tidy (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3318:21)

Is there any reason for this? I've double-checked all my tensors going in, and they all contain valid data.

System details:

• Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
• Ubuntu 20.10
• Nvidia GeForce RTX 2060 Driver Version: 455.38, CUDA Version: 10.0 (11.1 installed globally, 10.0 installed locally, error occurs regardless of whether I use the GPU or not)
• Packages and versions: @tensorflow/tfjs-node = 2.7.0, @tensorflow/tfjs-node-gpu = 2.7.0

Model summary:

_________________________________________________________________
Layer (type)                 Output shape              Param #   
=================================================================
lstm_LSTM1 (LSTM)            [64,32,1]                 12        
_________________________________________________________________
lstm_LSTM2 (LSTM)            [64,32,32]                4352      
_________________________________________________________________
lstm_LSTM3 (LSTM)            [64,32,1]                 136       
=================================================================
Total params: 4500
Trainable params: 4500
Non-trainable params: 0
_________________________________________________________________

Options objects for the 3 layers:

{
  stateful: true,
  units: 1,
  returnSequences: true,
  inputShape: [ 32, 1 ],
  batchSize: 64
}
{ stateful: true, units: 32, returnSequences: true }
{ stateful: true, units: 1, returnSequences: true }

• Model creation code: https://ybin.me/p/3b2151c72409317f#IXhSzO+VS1abLA9RXjX4JMtQf/oVbGUFe+xTqLoFcNE=
• Dataset creation code: https://ybin.me/p/22a21ae11489ef2c#eZaLy5MgWwi7L3EFo5D+jHsG8xAv7mp+8suvTPbXHqY=

The dataset is created by passing Uint8Array instances to tf.tensor, which are themselves views of a main ArrayBuffer.

Training code:

await this.model.fitDataset(dataset, {
    epochs: 50,
    verbose: 1,
    yieldEvery: "batch",
    shuffle: false,
    callbacks: {
        onEpochEnd: async (epoch, metrics) => {
            // .....
        }
    }
});

...full code available upon request.

[rthadur]

@sbrl please provide reproduction code in codepen , thank you

[sbrl]

@rthadur Thanks for the reply. Unfortunately, reproducing it in codepen will be extremely challenging because this is a bug in the _Node.js_ version of Tensorflow.js - NOT the browser version. I can certainly provide an isolated Node.js test case instead, but I seriously doubt that a browser test case is going to help anyone debug this issue at all.

[tafsiri]

An isolated node.js test case would be great! Thanks

[sbrl]

Sure thing, @tafsiri!

Ok, so it's taken me a while (I couldn't reproduce it for a moment), but here's a link to an isolated test case: https://ybin.me/p/a1270fe449cdf849#ytc4MN1kBD9FJvUHsV3hDr7fYflYxY9ny1+suX75FZg=

Instructions:

1. Download the file, and save it to isolated-test.mjs.
2. Run npm install @tensorflow/tfjs-node
3. Run node ./isolated-test.mjs to execute the code.

A recent version of Node.js is required (I'm using v15.1.0), because I'm using the ES6 module syntax.

[sbrl]

Oops, I did NOT mean to close this! Reopened. Hopefully that doesn't affect anything @tafsiri @rthadur?

[tafsiri]

I think I've traced this down to an issue in tf.data.generator and how you are using it in this instance. The core issue is that you are passing the same tensor references as part of multiple input/label tuples. Internally at some point tf will dispose of the tensor once it has been consumed from the generator. Here is a workaround:

// in process_data()
yield {
    xs: last.clone(),
    ys: next.clone()
};

Adding .clone() to the tensor that you yield to the generator will ensure each gets a new id and object reference and will prevent early garbage collection.

[sbrl]

Ah, thanks so much! That seems to have fixed it :D :D :D

Is there a way to patch Tensorflow.js to generate a more useful error message in circumstances like these (e.g. if a Tensor doesn't exist, then throwing a nice error to tell the developer about this), or should I just close this issue?

[tafsiri]

You can leave it open for now. We can discuss internally what the best solution might be (there are a number of different places we may want to tackle this). I'm going to update the title for when we get a chance to look more closely at this.

[joshuaellis]

I think I've traced this down to an issue in tf.data.generator and how you are using it in this instance. The core issue is that you are passing the same tensor references as part of multiple input/label tuples. Internally at some point tf will dispose of the tensor once it has been consumed from the generator. Here is a workaround:

// in process_data()
yield {
  xs: last.clone(),
  ys: next.clone()
};

Adding .clone() to the tensor that you yield to the generator will ensure each gets a new id and object reference and will prevent early garbage collection.

This is really useful to know if you're using React & Refs.

[sbrl]

Thanks! I look forward to further news about this.