Face-api.js: Preventing memory leak with detectAllFaces() ?

[djipco]

I am using detectAllFaces() with TinyFaceDetector. Every 500ms I feed it a new image to analyze. I noticed that I'm getting a memory leak because new canvas elements keep getting created and are never disposed of (I can see them in the Developer Tools' Canvas tab).

What is the proper way to dispose of all these temporary canvas ?

P.S. Thanks for a great library!

[djipco]

I just realized that the memory issue I am having is related to a leak in the CameraPreview cordova module. Sorry for the false report...

[djipco]

Well, it seems that detectAllFaces() may present an issue after all... Or, maybe, I do not understand how the library works.

As far as I can tell, each time detectAllFaces() is called with an HTMLImageElement as the first parameter, it creates a <canvas> element and stores it in an array. This is done in NetInput.js at line 35-36:

var canvas = input instanceof env.getEnv().Canvas ? input : createCanvasFromMedia(input);
_this._canvases[idx] = canvas;

As far as I can tell, these canvases never get garbage collected. At least, I can see them pile up in the Developer Tools' Canvas tab.

Could there be a problem there?

[djipco]

I'm going crazy from getting random crashes... but I think it's something else... I'll close this until I get better data.

[justadudewhohacks]

Hey Jean-Philippe :)

As far as I can tell, each time detectAllFaces() is called with an HTMLImageElement as the first parameter, it creates a element and stores it in an array. This is done in NetInput.js at line 35-36:

Yes, this is correct.

As far as I can tell, these canvases never get garbage collected.

They should get garbage collected, otherwise you will run out of memory pretty quickly. Maybe you can share some code of what you are doing? If these canvases do not get garbage collected, then there is probably some reference dangling around somewhere.

Maybe you are unintenendly storing the NetInputs on some global object? Maybe the Dev Tools you are using itself cause the memory leak, because it is holding references to the canvases for debugging purposes. Same goes for logging the objects to the console, which will prevent them from being garbage collected until you clear your console.

[djipco]

Hey Vincent!

I didn't know if you would remember our previous exchanges... :-)

As you can see, I am (once again) using some of your software for one of my projects. I'm developing an iPhone app using Phonegap. I'm running face-api.js to perform face detection (no recognition, no landmark detection or anything else) from the phone's camera. I get a rate of about 4fps on an iPhone 5S, which is fine for what I need. However, after 1300-1600 detections, the app crashes and I don't know why.

One of the Cordova plugins I was using leaked memory but it has since been fixed. I have no hints as to what's going on... The only thing that makes me think it might somehow be related to face-api.js, is that when I remove the detection code, it takes longer to crash (but it does crash anyway, so it's probably not related to face-api.js).

Here's the code I am using:

async detectFromBase64ImageData(base64ImageData, mimeType = "image/jpeg") {

  return new Promise(resolve => {

    this.image.addEventListener("load", async () => {
      const faces = await this.detect(this.image);
      resolve(faces);
    }, {once: true});

    this.image.src = "data:" + mimeType + ";base64," + base64ImageData;

  });

}

async detect(source) {

  let faces;

  try {
     faces = await faceapi.detectAllFaces(source, this.options);
  } catch (err) {
    console.info(err);
  }

  return faces;

}

The detectFromBase64ImageData() function gets called repeatedly with the image data from the camera. It works fine for about 12 minutes (or 1500 detections) and then the app crashes. My tests seem to indicate that memory usage is going up...

Any ideas?

[justadudewhohacks]

Hmm, I can't spot anything obvious, that would cause a leak. I didn't run in such issues on my mobiles devices before.

You could try out the webcam examples on the face-api demo page with your iphone. If that's working you atleast know for sure, that it's not an issue with face-api.js.

[djipco]

You could try out the webcam examples on the face-api demo page with your iphone. If that's working you atleast know for sure, that it's not an issue with face-api.js.

Yep, that's a good idea. But, like I said, I don't think the problem is with face-api.js. That's why I close the issue myself. From the crash logs, I think the problem is that the application draws too much CPU over too long.

Anyway, thanks for the input!

[josiahbryan]

I can confirm that detectAllFaces is leaking memory. I pass frames to a WebWorker with postMessage and transfer lists, then create an ImageData in the WebWorker as shown below.

If I run the code as shown, the Chrome process memory balloons and grows DURING the frame processing. (E.g. while the webworker is inside detectAllFaces. Within ~3 frames, Chrome is using 25% of the RAM and growing. After ~30 seconds, Chrome has consumed nearly all of my 16 GB of RAM.

As soon as I comment out the line that has detectAllFaces - memory usage flatlines. Does not grow or change. And that is the ONLY code change I make - with detectAllFaces, memory leaks. Without that call, no memory leak. Therefore, something is going wrong with detectAllFaces - because there are no other references to the given imgData other than detectAllFaces.

Any ideas on how to diagnose further or, hopefully, fix?

// Earlier in the code ...
faceapi.nets.ssdMobilenetv1.loadFromUri('/models')
this.faceDetectorOptions = new faceapi.SsdMobilenetv1Options({ minConfidence: 0.5 });

// For every message from the parent thread, we do...
const imgData = new ImageData(
    new Uint8ClampedArray(message.data),
    message.width,
    message.height
);

const img = faceapi.createCanvasFromMedia(imgData);

const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)