Library: Add crop functionality to BrowserCodeReader

[mpodlasin]

You often don't want to scan whole picture. In our case datamatrix is really small, so we want to crop and scan only the center of the picture.

I propose to add additional configuration option to BrowserCodeReader allowing to pass crop options to drawImage method: what should be the size of cropped picture and what would be its position in larger picture

[odahcam]

So how would the user know what portion of the image his device is capturing will be scanned? Your ideia is to identify and crop just the code or just crop at some point of the image?

[mpodlasin]

We make a crop mask with css. Here is a picture of our app:

Only stuff inside white rectangle is passed to algorithm. Since area is 4 times smaller, algorithm has much better results, than if we would pass whole picture. Scanning apps very often have masks like this.

If programmer could pass crop data to reader, he could use the same data to calculate and create mask of proper size.

The only other way to achieve this would be to write custom BrowserCodeReader, which we obviously would like to avoid.

[mpodlasin]

Crop data is calculated based on width and height of video stream, so it would have to be a function:

type Crop = (videoWidth: number, videoHeight: number) => ({ scanHeight: number, scanWidth: number, xOffset: number, yOffset: number })

Here scanHeight and scanWidth would be the size of white rectangle and xOffset and yOffset would say where within whole video the rectangle should be placed.

It could be passed for example as optional argument to a BrowserCodeReader method.

In our case, where we want to have centered crop, with area 4 times smaller, this function would be:

const crop = (videoWidth: number, videoHeight: number) => {
  const scanHeight = videoHeight / 2;
  const scanWidth = videoWidth / 2;
  const xOffset = (videoHeight - scanHeight) / 2;
  const yOffset = (videoWidth - scanWidth) / 2;

  return { scanHeight, scanWidth, xOffset, yOffset };
}

[odahcam]

For the ngx-scanner we have a custom scanner class that overrides some methods. 😄 That's not that good, but also not that bad.

A important thing about the crop masks is that they aren't always just crops, but sometimes they're based on the camera focus and barcode size that should be read, to have something like a perfect size match when scanning.

So the crop would rely just on BrowserCodeReader?

I think that's not a good idea to add it to the core, but as long as the Browser will take care of the image and then pass it to the core to decode, that sounds nice.

[mpodlasin]

Well, if we would extract discussed part of code into some kind of protected method, so that we could extend it by ourselves, I would be ok with that as well. Then even API would not change.

The only risk is some day somebody forgets why this method is protected and just makes it private again, thus breaking our code.

[odahcam]

Well, I think that just this should be in another method:

https://github.com/mpodlasin/library/blob/49e3df942490c5fb1457575ef3a2ef58a9ac0832/src/browser/BrowserCodeReader.ts#L249-L259

I can accept the rest of the code without problems. Our code is not thaaaat good at all :( ...so we gonna have to change it in the future in a way or another and I just don't want to mess things up when that day comes.

[bestbelief]

@mpodlasin Hello, I'm writing this function now, but I don't know how to write it. Could you give me some of your code for reference? I know this request is very presumptuous, but I really don't know how to write it. I've just been working for a while and I hope you can help me. Anyway, thank you very much.

[odahcam]

Oh, I'm closing this because it was merged, but feel free to keep the conversation as it needs.

[boudabza]

Hi
how can i use the drawing canvas in my app
thank you

[odahcam]

Sorry, I could not understand. Please open a new issue.

[bburns]

I got this working with React - it copies a crop of the video feed onto a canvas, then copies the canvas into an image element, then feeds that to the barcode reader...

Tested on MacOS Chrome so far -

// scanner video feed
// see https://github.com/zxing-js/library

import React from 'react'
import './styles.scss'
import { BrowserBarcodeReader } from '@zxing/library'

const timeout = 1000 // time between frames
const scale = 0.5 // size of crop frame

let barcodeReader
let videoStream

// user clicked on the camera button - bring up scanner window.
export async function openScanner(onFoundBarcode) {

  barcodeReader = new BrowserBarcodeReader()

  showScanner()

  // get html elements
  const video = document.querySelector('#scanner-video video')
  const canvas = document.querySelector('#scanner-canvas')
  const img = document.querySelector('#scanner-image')
  const frame = document.querySelector('#scanner-frame')

  // turn on the video stream
  const constraints = { video: true }
  navigator.mediaDevices.getUserMedia(constraints).then(stream => {
    videoStream = stream

    // handle play callback
    video.addEventListener('play', () => {
      // get video's intrinsic width and height, eg 640x480,
      // and set canvas to it to match.
      canvas.width = video.videoWidth
      canvas.height = video.videoHeight

      // set position of orange frame in video
      frame.style.width = video.clientWidth * scale + 'px'
      frame.style.height = video.clientHeight * scale + 'px'
      frame.style.left =
        (window.innerWidth - video.clientWidth * scale) / 2 + 'px'
      frame.style.top =
        (window.innerHeight - video.clientHeight * scale) / 2 + 'px'

      // start the barcode reader process
      scanFrame()
    })

    video.srcObject = stream
  })

  function scanFrame() {
    if (videoStream) {
      // copy the video stream image onto the canvas
      canvas.getContext('2d').drawImage(
        video,
        // source x, y, w, h:
        (video.videoWidth - video.videoWidth * scale) / 2,
        (video.videoHeight - video.videoHeight * scale) / 2,
        video.videoWidth * scale,
        video.videoHeight * scale,
        // dest x, y, w, h:
        0,
        0,
        canvas.width,
        canvas.height
      )
      // convert the canvas image to an image blob and stick it in an image element
      canvas.toBlob(blob => {
        const url = URL.createObjectURL(blob)
        // when the image is loaded, feed it to the barcode reader
        img.onload = async () => {
          barcodeReader
            // .decodeFromImage(img) // decodes but doesn't show img
            .decodeFromImage(null, url)
            .then(found) // calls onFoundBarcode with the barcode string
            .catch(notfound)
            .finally(releaseMemory)
          img.onload = null
          setTimeout(scanFrame, timeout) // repeat
        }
        img.src = url // load the image blob
      })
    }
  }

  function found(result) {
    onFoundBarcode(result.text)
    closeScanner()
  }

  function notfound(err) {
    if (err.name !== 'NotFoundException') {
      console.error(err)
    }
  }

  function releaseMemory() {
    URL.revokeObjectURL(img.url) // release image blob memory
    img.url = null
  }
}

export function closeScanner() {
  if (videoStream) {
    videoStream.getTracks().forEach(track => track.stop()) // stop webcam feed
    videoStream = null
  }
  hideScanner()
  barcodeReader.reset()
}

function showScanner() {
  document.querySelector('.scanner').classList.add('visible')
}

function hideScanner() {
  document.querySelector('.scanner').classList.remove('visible')
}

export default function Scanner() {
  return (
    <div className="scanner">
      <div id="scanner-video">
        <video autoPlay playsInline></video>
      </div>
      <div id="scanner-frame"></div>
      <canvas id="scanner-canvas"></canvas>
      <img id="scanner-image" src="" alt="" />
      <button id="scanner-close" onClick={closeScanner}>
        Close
      </button>
    </div>
  )
}

.scanner {
  height: 100%;
  display: none;
}
.scanner.visible {
  display: block;
}

#scanner-video {
  position: absolute;
  top: 0;
  left: 0;
  bottom: 0;
  right: 0;
  margin: auto;
  z-index: 1;
  background: rgba(0, 0, 0, 0.7);
  z-index: 1;
}

#scanner-video video {
  object-fit: initial;
  position: absolute;
  top: 0;
  left: 0;
  bottom: 0;
  right: 0;
  margin: auto;
  height: 100%;
  width: auto;
  z-index: 2;
  @media (orientation: portrait) {
    width: 100%;
    height: auto;
  }
}

#scanner-frame {
  position: absolute;
  margin: auto;
  box-shadow: 0 0 0 9999px rgba(0, 0, 0, 0.5);
  border: 2px solid orange;
  z-index: 3;
}

#scanner-canvas {
  display: none;
  width: 100%;
  margin: auto;
  z-index: 4;
}

#scanner-image {
  display: none;
  width: 100%;
  margin: auto;
  z-index: 10;
}

#scanner-close {
  position: absolute;
  top: 0;
  left: 0;
  z-index: 30;
  cursor: pointer;
}
#scanner-close:hover {
  background: #aaa;
}

[odahcam]

Very nice @bburns , thanks for sharing!