P5.js: loadPixels() with get() on capture

[mobeets]

Hi, I've found a bug where, if you have a webcam capture, then calling loadPixels() on the object returned by get() does not work. (This is similar to some previous issues (#1274, #1079), but here the problem only happens if you call loadPixels() on something returned by get().)

Nature of issue?

• [x] Found a bug
• [ ] Existing feature enhancement
• [ ] New feature request

Which platform were you using when you encountered this?

• [ ] Mobile/Tablet (touch devices)
• [x] Desktop/Laptop
• [ ] Others (specify if possible)

Most appropriate sub-area of p5.js?

• [ ] Color
• [ ] Core/Environment/Rendering
• [ ] Data
• [ ] Events
• [x] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [ ] Utilities
• [ ] WebGL
• [ ] Other (specify if possible)

Details about the bug:

• p5.js version: 0.6.1
• Web browser and version: Chrome Version 67.0.3396.87 and Firefox Version 60.0.2
• Operating System: MacOSX 10.12.1
• Steps to reproduce this:

var cap;
var curPatch;

function setup() {
   createCanvas(300, 300);
   cap = createCapture(VIDEO);
   cap.hide();
 }

function draw() {
  curPatch = cap.get(0, 0, 32, 32);
  curPatch.loadPixels();
  text(curPatch.pixels.length, 0, 50);
}

The error I get in Chrome looks like this:

Uncaught DOMException: Failed to execute 'getImageData' on 'CanvasRenderingContext2D': The source width is 0.
    at CanvasRenderingContext2D.getImageData (<anonymous>:209:23)
    at d.Image.d.Renderer2D.loadPixels (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:8:11178)
    at d.Image.loadPixels (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:9:23609)
    at loadPatches (http://127.0.0.1:8000/static/test.js:22:16)
    at draw (http://127.0.0.1:8000/static/test.js:12:3)
    at e.redraw (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:8:30093)
    at e.<anonymous> (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:7:20448)
    at e.<anonymous> (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:7:19138)
    at new e (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:7:22615)
    at e (https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.6.1/p5.min.js:7:30943)

And in Firefox:

IndexSizeError: Index or size is negative or greater than the allowed amount
p5.min.js:8
[29]</d.Renderer2D.prototype.loadPixels
http://127.0.0.1:8000/static/js/p5.min.js:8:11158
[43]</d.Image.prototype.loadPixels
http://127.0.0.1:8000/static/js/p5.min.js:9:23575
loadPatches
http://127.0.0.1:8000/static/test.js:22:7
draw
http://127.0.0.1:8000/static/test.js:12:3
[32]</e.prototype.redraw
http://127.0.0.1:8000/static/js/p5.min.js:8:30093
e/this._draw<
http://127.0.0.1:8000/static/js/p5.min.js:7:20443
<anonymous> self-hosted:973:17 e/this._start<
http://127.0.0.1:8000/static/js/p5.min.js:7:19133
<anonymous> self-hosted:973:17 e
http://127.0.0.1:8000/static/js/p5.min.js:7:22610
e
http://127.0.0.1:8000/static/js/p5.min.js:7:30943

[mobeets]

I'm not familiar with the code base at all, but I have realized that this error only happens for the first second or so of reloading the page, and then everything works just fine.

I found that if I wait to call curPatch.loadPixels(); until curPatch.width > 0, everything works fine after that point. Here's some code to show what I'm talking about:

var cap;
var curPatch;
var i;

function setup() {
   createCanvas(300, 300);
   cap = createCapture(VIDEO);
   cap.hide();
   i = 0;
 }

function draw() {
  background(200);
  i += 1;
  curPatch = cap.get(0, 0, 32, 32);
  if (curPatch.width > 0) {
    curPatch.loadPixels();
    text('success: ' + i.toString(), 100, 50);
  } else {
    text('error: ' + i.toString(), 0, 50);
  }
}

[Ajayneethikannan]

Hi @mobeets ! In the present state, you have to call loadPixels() before using the function get() on the video element. You can build the current version from the master branch.
This would work correctly

var cap;
var curPatch;

function setup() {
   createCanvas(300, 300);
   cap = createCapture(VIDEO);
   cap.hide();
 }

function draw() {
  cap.loadPixels();
  curPatch = cap.get(0, 0, 32, 32);
  curPatch.loadPixels();
  text(curPatch.pixels.length, 0, 50);
}

@lmccart @Spongman ,

• Since image.get() and get() on the main canvas work without calling loadPixels(), I think it would be
  nice to not have to call loadPixels manually.
• I think we should add documentation and reference for p5.MediaElement.prototype.get separately .

Would love to hear suggestions,
thanks!

[Spongman]

calling loadPixels allows us to cache the whole pixels array. if you don't call loadPixels and just call get() we'll often just extract _just_ that pixel from the image. it doesn't make sense to fetch the whole image if all you want is a simple pixel.

[Ajayneethikannan]

@Spongman ,
From what I have found out ,

• We can call get on the main canvas get(), and on image img.get() without the need of calling loadPixels (does it get faster for retrieving one pixel , because we are returning it directly from an array ? ). The path followed is p5.prototype.get --> p5.Renderer2D.prototype.get --> p5.Renderer2D.prototype._getPixel
  and a similar path for p5.Image.

Here we have no need to call loadPixels before calling get.

But in p5.MediaElement,

• If I want to get a portion of the video and not a pixel, I have to call loadPixels every time, before getting the image, because the video is not drawn on the canvas during execution of get.
• For getting a pixel, I should be able to get without calling loadPixels, but it seems like you have to call it atleast once (I called in setup) , otherwise the sketch stops with an error

canvas is undefined.

because the canvas instance is created in p5.MediaElement only after calling loadPixels the first time,
and the error occurs in p5.Renderer.prototype.get, where the canvas variable is referenced , which is undefined, this causes error because the user has to first click on allow webcam , before which the sketch starts and stops with the error.

So currently,I have to call loadPixels before get in case of MediaElement irrespective of pixel or image,
and not neccessary in case of image element or main canvas.

This sketch can be used to test ,

//leave comments to see error for getting one pixel
var cap;
var curPatch;

function setup() {
   createCanvas(300, 300);
   cap = createCapture(VIDEO);
   cap.hide();
   //cap.loadPixels();  //-- uncomment only this to test get for one pixel working properly.
 }

function draw() {
  //cap.loadPixels();  //-- uncomment for checking image, leave commented to see error
  curPatch = cap.get(100, 100);  // change to x, y, w, h format to check for image
  console.log(curPatch);
  //curPatch.loadPixels(); //-- uncomment for checking image.
  fill(curPatch);
  ellipse(100, 100, 100, 100);
  //image(curPatch, 0, 0, 300, 300); -- uncomment for checking image
  //text(curPatch.pixels.length, 0, 50);  -- uncomment for checking image.
}

I am sorry if I am wrong, but would definitely like to hear more about it .
Thanks!

[Spongman]

ah, good catch. it looks like it needs to call _ensureCanvas first, like this:

javascript p5.MediaElement.prototype.get = function() { this._ensureCanvas(); return p5.Renderer2D.prototype.get.apply(this, arguments); };

[Ajayneethikannan]

@Spongman can you please provide your view on this ,

  p5.MediaElement.prototype.get = function() {
    this.loadPixels();
    return p5.Renderer2D.prototype.get.apply(this, arguments);
  };

  p5.MediaElement.prototype._getPixel = p5.Renderer2D.prototype._getPixel;

Here I am calling loadPixels inside the get function, and as loadPixels( ) has all the optimisations present inside it, it will not update the video element's canvas unnecessarily.

And since _getPixel( ) is called through get( ) and the pixels are already loaded( if necessary), I think it can directly be replaced by p5.Renderer2D.prototype._getPixel without any bugs.

It also has the benefit that we do not need to call loadPixels before using get( ) in video elements.
(which is currently needed for images, but not individual pixels in the video)

Thank you !

[Spongman]

no. you need the check that _ensureCanvas provides, otherwise get will always mark the element as modified.

[Ajayneethikannan]

Got it , thanks :+1:

[Ajayneethikannan]

@Spongman Thank you for your suggestions, I have made a PR with the same.

In my humble opinion, it feels a little non intuitive to call loadPixels manually only for the case of getting an image from the videoElement, whereas the get( ) function for all other cases, such as getting pixel from a video element, image instance or the main canvas can be called without it

I think calling loadPixels in p5.MediaElement.get( ) similar to p5.MediaElement._getPixel( ) by checking if _pixelsTime is not equal to elt.currentTime, can achieve this.

Sorry if my suggestion is wrong, but do correct me and provide your thoughts on this,
Thanks again !

[Spongman]

i'm sorry, that doesn't make sense to me.
it's not necessary to duplicate the _getPixel code inside of p5.MediaElement.get(). it's also not necessary to call loadPixels every get() call.