Face-api.js: Running FaceAPI.js within Web Worker (slow)

[spencerwaynesmith]

We found that the video is choppy when running the faceAPI on certain devices. Specifically, on IOS devices. Normally, this can be fixed by running the code in a Web Worker. The problem is that in order to run within a Web Worker we need to pass in ImageData / not DOM related objects like Canvas/Video/Image. It also appears to run much, much slower when we run in a Web Worker.

Is there a way we can make this work? What do you suggest? What type of object can we create on the Web Worker side from an ImageData object that will run at around the same speed?

[jeffreytgilbert]

That's because if you run this on a web worker, you're splitting the CPU to another thread but doing nothing to split loads for the GPU. Transfer of assets to a web worker are not free/fast until browsers support a zero copy strategy like offscreen canvas. Chrome does. Firefox does behind a flag. Safari does not. Zero copy is akin to shifting which CPU thread owns the pointer to that memory object (hence no copying). So let's say you do use Chrome and you do use offscreen canvas to transfer; you'd still need to trick the underlying TensorFlow library doing the ML and the faceapi.js library (both) into believing they are in a normal web browser environment (not node.js, and not webworker) so that they will process the data using Canvas. At that point, you'll have CPU work running off the main ui thread, and you'll have the GPU processing the ML quickly, but again the limitation there is that is only possible with Chrome and Firefox, and I don't know if mobile iOS versions have those same APIs supported.

I filed tickets with Tensorflow and worked on faceapi.js to do exactly this, but the use case I had was a kiosk app on the desktop. Mobile is a long shot.