Tfjs: Error: When running in node, pixels must be an HTMLCanvasElement like the one returned by the `canvas` npm package

[loretoparisi]

To get help from the community, we encourage using Stack Overflow and the tensorflow.js tag.

TensorFlow.js version

latest version in package.json

Browser version

node.js

Describe the problem or feature request

I get this error whe calling tf.browser.fromPixels

Code to reproduce the bug / link to feature request

const tf = require('@tensorflow/tfjs');
require('@tensorflow/tfjs-node');
//...
img = tf.browser.fromPixels(img);

Full stacktrace:

(node:48448) UnhandledPromiseRejectionWarning: Error: When running in node, pixels must be an HTMLCanvasElement like the one returned by the `canvas` npm package
    at NodeJSKernelBackend.fromPixels (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1306:19)
    at Engine.fromPixels (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/engine.js:498:29)
    at fromPixels_ (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/ops/browser.js:83:37)
    at Object.fromPixels (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/ops/operation.js:46:29)
    at /Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/lib/index.js:112:34
    at /Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/engine.js:114:22
    at Engine.scopedRun (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/engine.js:124:23)
    at Engine.tidy (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/engine.js:113:21)
    at Object.Environment.tidy (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/node_modules/@tensorflow/tfjs-core/dist/environment.js:170:35)
    at NSFWJS.infer (/Users/loretoparisi/Documents/Projects/AI/nsfwjs-node/lib/index.js:110:19)
(node:48448) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:48448) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

My package.json

{
  "name": "nsfw-node-test",
  "description": "test",
  "version": "1.1.3",
  "author": {
    "name": "Loreto Parisi",
    "email": "[email protected]"
  },
  "main": "lib/index",
  "engines": {
    "node": "*"
  },
  "scripts": {},
  "devDependencies": {},
  "keywords": [],
  "dependencies": {
    "@tensorflow/tfjs-node": "latest",
    "node-fetch": "^2.2.0"
  }
}

[nsthorat]

Hi @loretoparisi, in this case what type of object is img that you're passing to tf.browser.fromPixels?

[loretoparisi]

@nsthorat so I have loaded the image from a file, it is a jpeg, and I have passed it as discussed in the forums with Daniel, using the canvas package like

const img = new Image()
img.onload = () => {
  console.log("image ready")
  nsfwjs.load().then(function (model) {
    model.classify(img).then(function (predictions) {
      // Classify the image
      console.log('Predictions: ', predictions)
    })
  })
}
img.onerror = err => { throw err }
img.src = 'jenna.jpg';

and then later on in the inference I do

if (!(img instanceof tf.Tensor)) {
    img = tf.browser.fromPixels(img);
}

while model shards are loaded from local path with tf.loadLayersModel(path)

In the code I do TFJS import in this order

require('@tensorflow/tfjs-node');
const tf = require('@tensorflow/tfjs');

[nsthorat]

I think the issue is you need to pass a canvas object to tf.browser.fromPixels, but you are passing an Image object. If you look at the node-canvas documentation you can draw the image to the canvas with:

const canvas = createCanvas(width, height);
const ctx = canvas.getContext('2d');
...
ctx.drawImage(image, 0, 0, width, height);
tf.browser.fromPixels(canvas).print();

[loretoparisi]

@nsthorat and ... we are done!!!
model loaded.
image loaded.

Predictions:  [ { className: 'Porn', probability: 0.9853878617286682 },
  { className: 'Sexy', probability: 0.009328912012279034 },
  { className: 'Hentai', probability: 0.00435712980106473 },
  { className: 'Neutral', probability: 0.0007902148645371199 },
  { className: 'Drawing', probability: 0.00013616154319606721 } ]

So code was

const nsfwjs = require('../lib/index');
const { Image, createCanvas } = require('canvas');

// load model once
nsfwjs.load().then(function (model) {
  console.log("model loaded.");

  // feed images
  const width = 300;
  const height = 300;
  const canvas = createCanvas(width, height);
  const ctx = canvas.getContext('2d');
  const img = new Image()
  img.onload = () => {
    console.log("image loaded.");

    ctx.drawImage(img, 0, 0, width, height);

    // classify
    model.classify(canvas).then(function (predictions) {
      // Classify the image
      console.log('Predictions: ', predictions)
    })
  }
  img.onerror = err => { throw err }
  img.src = 'jenna.jpg';


})

@nsthorat thank you!!!

Why don't use embed a tiny version of canvas within TFJS? I mean just the needed to load and convert. Of course there could be issues about images size (but this is more related to the image dataset...). Currently canvas it's a significant package to be imported, I'm looking for something tiny with a basic canvas support.

[nsthorat]

Woohoo! We'll hopefully soon support tf.node.decodeImage() so you won't have to go through those steps:
https://github.com/tensorflow/tfjs/issues/1416

Feel free to close this out if you are satisfied :)

[loretoparisi]

@nsthorat yes having tf.node.decodeImage() would be the best solution. I assume you can work on byte arrays in Node.js rather than Canvas. Node-canvas uses Cairo, that requires node-gyp, so it could be better to have everything within TFJS.

[nsthorat]

+1 we will prioritize it, it should be pretty simple!

[jetjodh]

@nsthorat Is tf.node.decodeImage() live now?

[nsthorat]

@jetjodh it's not yet implemented, here is the tracking bug: https://github.com/tensorflow/tfjs/issues/1416