Library: Decode performance

[OwenKrueger]

I'm doing an implementation of this library and it's going great so far, but it's not quite as fast to decode as our previous decoder, and it occasionally has issues decoding at all. I've done a few things in my implementation to try and help but It's still not quite where i'd like it.

I'm already using the MultiFormatReader with TRY_HARDER and POSSIBLE_FORMATS hints (CODE_39, CODE_128, AZTEC, DATA_MATRIX) which gave a decent performance improvement to start with.
I'm wondering if giving the decoder a cropped video feed could help as discussed in #39 / #41 but it looks like the changes there were either moved or changed since, I did find a crop function in the LuminanceSource.ts file but i'm not sure how to use that in conjunction with the MultiFormatReader.

Is there anything I could do in my implementation to pass the decoder a cropped feed? or is there an option or different method of decoding that I could use to further help the performance of the decoder.

Thanks in advance.

[odahcam]

I'm a bit in a hurry this week, but I wanna give you a proper answer given the quality of your question. This might take a few minutes to hours to summarize a plausible answer for that, so please understand if I take a few days to answer. Meanwhile I can help you with small questions if you have.

A few insights:

• TRY_HARDER will make the algorithim slower, even if sometimes it helps scan your code faster, it makes the MultiformatReader do a lot a extra stuff and consume memory.
• POSSIBLE_FORMATS is a good way to reduce MultiformatReaders work.

• I'm wondering if giving the decoder a cropped video feed could help

  Yes, it will.

That's the easy stuff, now the cropping stuff:

Is there anything I could do in my implementation to pass the decoder a cropped feed?

Yes, there is, but there's literally lots of ways of doing that and to give you proper example I would have to expend some hours on it.

You could help me telling what environment are you using the library on (node, browser) and how you're passing video feed to it (built-in camera methods, video tag, image, canvas, etc.).

[OwenKrueger]

Firstly, thanks for your help.

• TRY_HARDER will make the algorithim slower, even if sometimes it helps scan your code faster, it makes the MultiformatReader do a lot a extra stuff and consume memory.

Originally I turned this on because we were trying to scan barcodes from a phone held in a "landscape" orientation, this was simply because our old scanner library worked in landscape mode and it's what we were used to, I can turn it off if necessary.

As far as how i'm using the decode, i'm running this in a browser and it's intended to be used as part of a PWA, im using the ZXing.browserMultiFormateReader object and the decodeOnceFromVideoDevice function, with a video element and the default front facing camera being selected with an html video tag with id "video" to display the camera feed to the user.

 codeReader = new ZXing.BrowserMultiFormatReader(hints)
 codeReader
     .decodeOnceFromVideoDevice(undefined, 'video')
     .then(result => loadAddVehicle(result))
     .catch(err => console.error(err));`

Thanks again for your time, this isn't an urgent issue as I have a lot more work to do on other aspects of this PWA before it's ready for launch.

Additionally I did dig around in the core files directory but wasn't really understanding what was going on all too well, i'm assuming it would be possible for me to write my own "browserMultiFormatReader" function that handles all the same DOM manipulations (Grabbing the camera, attaching it to the video element, passing the feed to the decoder) while also cropping the feed at some point. But I got lost and decided it would be best to try and solve it with the prebuilt functions if possible.

[odahcam]

I gotta tell you that it's not possible yet to do video cropping with current prebuilt functions, there's a need to extend some Reader and change the LuminanceSource (crop it) used to create the BitMatrix passed down to the core's decode method.

---

Oh and you're welcome, I try to do my best here, but sometimes it takes me a while to solve things due to other responsabilities, so please forgive me for any incomplete/delayed response.

[OwenKrueger]

At first I tried redefining the createBinaryBitmap method and cropping the mediaElement before getting the luminance source but I don't think I really understand JS objects as well as I thought I did since I had no lucking redefining that method at runtime.

I will try rolling my own code using the underlying core functions without the example prebuilt browser ones. But I think in the future it could be convenient if there was a simpler way to pass that information into the prebuilts (maybe through hints?) But I think I saw some changes that were made a few years back related to that and it developed into a mess of callbacks passing the crop options through like, four separate functions.

In any case, knowing that cropping the feed will both help performance and is possible is a great start!

[odahcam]

I think this could be configured int the BrowserCodeReader itself, so any decoding method there could take profit of cropping functionality. Fixing callback mess was one of my biggest troubles maintaining the codebase and still there's a lot to do, that's why I'm very cautious before adding new stuff. Maybe for croppping I just didn't had the right idea yet. Still, there's so much work to do and so few time that I end up focusing on what people ask the most and forget about this kind of stuff.

For cropping stuff out of your target media you have to extend BrowserCodeReader and override the createBinaryBitmap method. On this lines you should have access to the luminanceSource, like this: https://github.com/zxing-js/library/blob/39dbdc103f42b5869f07d060b4f0b6a8a2cc2b19/src/browser/BrowserCodeReader.ts#L854-L857

The browser layer uses a canvas to perform decoding operations, so you can easily crop the image from the canvas using the HTMLCanvasElementLuminanceSource instance. After that, you can extend that custom BrowserCodeReader and create specific readers for QR Code, Data Matrix, etc.

I know it's like a lot of work, but so is maintaining all this stuff. For now, and I hope a short period of time, that's the way to go. PRs are welcome BTW.

Thanks for bringing up the cropping feature, different from the past, I think now is a good time to have it.

[OwenKrueger]

Thanks for following up!
I tried extending the BrowserCodeReader and I can replace the createBinaryBitmap function with my own that would then crop the luminanceSource like this

const luminanceSource = new HTMLCanvasElementLuminanceSource(canvas);
const croppedLuminanceSource = luminanceSource.crop(100, 100, 500, 500);
const hybridBinarizer = new HybridBinarizer(croppedLuminanceSource);

However, when I try to crop the luminance source I run into this error.

RangeError: Maximum call stack size exceeded at l.crop (library@latest:1)

This is the luminanceSourceCrop function.

https://github.com/zxing-js/library/blob/09444be03d901bcf437173d1f955b63727868a79/src/browser/HTMLCanvasElementLuminanceSource.ts#L81-L84

I'm confused as to what this function does, is this a placeholder for a crop function I would implement myself?

[odahcam]

So, I just took a look in it these days and got myself thinking the same thing. It was so long ago I didn't really remember why it end up this way. So I started digging:

And there's our problem.

I found that this method was never implemented, but this PR had the aim to give users ability to crop canvas image. The idea in here was to crop the canvas image with something like this.

Joining these ideas I would try to implement the crop method to use canvas.getImageData to get a cropped canvas image and recreate the Uint8 buffer array, since I have no idea how to effectively crop the image directly in the buffer array.

So I'd try something like this (not tested):

{
    public crop(left: number /*int*/, top: number /*int*/, width: number /*int*/, height: number /*int*/): LuminanceSource {
        // NOTE: you can always implement the cropping how you want, there's a LOT of ways of cropping a image frame
        const imageData = this.canvas.getContext('2d').getImageData(left, top, width, height);
        this.buffer = HTMLCanvasElementLuminanceSource.toGrayscaleBuffer(imageData.data, width, height);
        return this;
    }
}

This is just some code written on the fly, please use as inspiration, but not expect it to work. Also, I will be glad to see your working results once you figure out a solution for implementing crop.

Good luck!

[OwenKrueger]

Thanks so much again for the help I was messing around with canvas crop functions yesterday with no luck but your snippet seemed to have done the trick.

My solution is working right now just extending straight into the createBinaryBitmap and HTMLCanvasElementLuminanceSource classes. It's probably quite fragile. I'm going to take some time off this issue and dig a little deeper later on to see if there's a more straightforward approach to specify a crop when creating the reader.

[radoslavius]

@OwenKrueger could you please show some final code as you solved it?

[Salketer]

Here's my implementation of cropping, fwiw...

The source code notes that drawImageOnCanvas is there to be extended, so I did. But I needed to change another bit of code for the canvas to actually be of the right size...

const targetSize = Math.ceil(availableWidth < availableHeight ? availableWidth / 2 : availableHeight / 2);
this.codeReader.createCaptureCanvas = function (element)
      {
        const canvasElement = document.createElement("canvas");
        canvasElement.style.width = targetSize + "px";
        canvasElement.style.height = targetSize + "px";
        canvasElement.width = targetSize;
        canvasElement.height = targetSize;
        return canvasElement;
      };
      this.codeReader.drawImageOnCanvas = function (canvasElementContext, srcElement)
      {
        const videoWidth = Math.ceil(srcElement.videoWidth);
        const videoHeight = Math.ceil(srcElement.videoHeight);
        const targetLeft = (videoWidth - targetSize) / 2;
        const targetTop = (videoHeight - targetSize) / 2;
        canvasElementContext.drawImage(srcElement, targetLeft, targetTop, targetSize, targetSize, 0, 0, targetSize, targetSize);
      };

It crops to a square that has half the width/height (which ever is the lowest) of the displayed video (availableHeight and availableWidth).

createCaptureCanvas was simply copied over, changing the values to fit the width/height. I also stripped it from numerous checks as I knew the element was a HTMLVideoElement

This indeed improves the speed! Before adding this, I was getting[Violation] 'setTimeout' handler took <N>ms in the close to 500.

WIth this, it is down to 150ms ish!!! So it's definitly worth it to crop!

The next part of the problem is the NotFoundException... This I could not tackle, hence why I'm here now. I'll post a different issue just for it.

[github-actions[bot]]

Stale issue message

[plague69]

Not sure if webworkers is still relevant, but sone gains to be had. Someone has already made a working zxing-js + comlink version (https://github.com/pocesar/react-use-qrcode).

https://github.com/zxing-js/library/issues/42#issuecomment-689307695

[froggr]

I've been playing around with cropping the last day or so. I tried a number of approaches. @Salketer your method is simple and extremely obvious performance increase. Any opinions on hiding the video and appending the the cropped canvas element? In my use case I want to only show the cropped area to the user. A bit of a hack for sure...