Opencv4nodejs: Performance issues

[djipco]

I'm using opencv4nodejs to perform face detection in a webcam feed. My early tests show rather poor performance. I'm getting 6-7 frames per second.

I was hoping that, by using OpenCV, I would get much better performance than JavaScript-based solutions such as clmtracker. It turns out that clmtracker is much faster (but not as accurate).

I looked at Chrome's performance monitor and I can see that OpenCV is taking anywhere between 100ms to 150ms to perform detection. This does not seem right. As a comparison, when I run FaceOSC on the same computer, I get a steady 60FPS.

Obviously, my code is not optimized but, as the performance monitor tells me, the bottleneck is outside JavaScript's purview.

Should I be expecting better performance? If not, I'll have to turn to another solution.

[justadudewhohacks]

You should definitely expect to get real time performance with opencv. Looking at your code that you posted in another issue I am assuming you are drawing the result into a canvas?

Unfortunately with electron, apparently also with nw.js there is a problem with the mat.getData function (see this issue: https://github.com/justadudewhohacks/opencv-electron/issues/2). Could you verify whether this is your bottleneck via simple console.time or similar?

I tried to figure out whats going on there and also opened an issue at electron-rebuild without success.

[djipco]

Thanks for the quick reply. I am indeed drawing to canvas. However, this does not seem to be the bottleneck. According to console.time(), these are the culprits:

const frame = vCap.read().bgrToGray();
±60ms

const faceResult = this.faceClassifier.detectMultiScale(frame);
±95ms

The whole detection loop takes about 155ms. This means that, besides those 2 lines, the rest is negligible.

If you want me to run more tests, I'll happily oblige.

[justadudewhohacks]

Ok atleast then there is no problem with getData in nw.js, that's good to know. :D

As far as detectMultiScale is concerned, going with default parameters usually does not provide the expected performance, especially with larger images. To achieve realtime performance I would highly suggest adjusting the parameters scaleFactor to limit the step size between each image scaling operation and minSize (minimum size of faces to expect) depending on your usecase.

For example running this with default parameters for a 1120x630 image on my machine takes me roughly 160ms on average. If I simply limit the search space however, e.g:
classifier.detectMultiScale(grayImg, { scaleFactor: 1.2, minSize: new cv.Size(100, 100) })
this brings it down to ~12ms, which I would consider realtime.

The delay you reported for vCap.read() seems strange to me. I will investigate what's the problem here.

[djipco]

Thanks a lot for your help! By tweaking the settings for detectMultiScale(), I'm now getting detection times below 1ms! However, vCap.read() is still a huge bottleneck. For that single function, console.time() reports times in the range of 50-100ms.

[fildaniels]

Just to check if it’s not an issue with your webcam, did you try to open a video stream by giving an url as a parameter to cv.VideoCapture() ? I’ve tested this OK by running mjpg-streamer (https://github.com/jacksonliam/mjpg-streamer) on a Raspberry pi. It works fine.

[djipco]

I tried with a local video file and I'm getting detection times below 1ms. Seeing that, I switched from my Macbook's onboard webcam to an external Logitech webcam and I'm getting better results (somewhere between 15-60ms).

What puzzles me is that, using the same webcam, I get 60FPS when I use FaceOSC for face detection...

[justadudewhohacks]

Hm ok, great that atleast the detectMultiScale problem is fixed. Did you have better results for reading from a webcam with one of the other javascript libraries you used before?

[djipco]

Did you have better results for reading from a webcam with one of the other javascript libraries you used before?

Yeah, that's the thing. For example, clmtracker gives me a much higher frame rate with the exact same webcam. The same goes for native solutions such as FaceOSC.

There has to be something wrong somewhere in the chain but I don't think the webcam is to blame.

[justadudewhohacks]

In this case the webcam should definitely be fine. Maybe this is just a configuration issue. You can by the way set properties for your VideoCapture, maybe this helps.

See the list of properties here:
https://docs.opencv.org/3.3.0/d4/d15/group__videoio__flags__base.html#gaeb8dd9c89c10a5c63c139bf7c4f5704d

You could try setting CAP_PROP_FPS manually. CAP_PROP_BUFFERSIZE sounds also interesting, maybe just buffer more frames in the capture. I never had the issue, thus I never had to touch these properties.

For example you can get and set properties by:
const fps = vCap.get(cv.CAP_PROP_FPS)
vCap.set(cv.CAP_PROP_FPS, 60)

[djipco]

I found the problem!!!

I was using vCap.read() called with requestAnimationFrame(). By using vCap.asyncRead() instead, I get sub-millisecond results. Yes! Just in time for the week-end.

Thanks a lot for your help and for your work on this library!

[justadudewhohacks]

Awesome!

[tscans]

I'm running into a similar issue with my webcam capture. It's very slow and its impacting my would be real time method. I am running the following code on my express server on an electron app inside of an interval running 30 times a second.

//defined
var vCap = new cv.VideoCapture(2);
vCap.set(cv.CAP_PROP_FRAME_HEIGHT,640);
vCap.set(cv.CAP_PROP_FRAME_WIDTH,960);
const fps = vCap.get(cv.CAP_PROP_FPS)

---

//loop
console.time("CAPTURE")
var frame = vCap.read();
console.timeEnd("CAPTURE")
console.time("ALGO")
const resizedImage = frame.resize(320, 480);
const gray = frame.bgrToGray();
const faces = classifier.detectMultiScale(gray, faceClassifierOpts).objects;
console.timeEnd("ALGO")

When I check the console to see the speed of this code, ALGO is only taking on average 11ms to complete. Yet, CAPTURE is outputting an average of 35ms. I've seen it go as high as 75ms. I tested 2 USB webcams and the built in webcam on my mac and the speeds are all roughly the same. Is there something I am doing wrong with my video capture?

I've also tried @djipco 's example with the async call, but all this allows me to do is set up a .then promise chain and the frame rate isn't any better.

[tscans]

@justadudewhohacks bump