Arfoundation-samples: Continuous QR Code scan alongside AR features?

[Sterling-Malory-Archer]

I have recently purchased a QR Code scanner asset from the Unity Store.

What I was wondering if it is possible to have QR Code Scanning in conjunction with AR Image Tracking?

This plugin uses a WebCamTexture, but what I was wondering is if I could somehow use the ARCamera instead of WebCamTexture.

Here's the code for Reading QR

public class Read_qr_code : MonoBehaviour
    {
        public RawImage raw_image_video;

        //camera texture
        private WebCamTexture cam_texture;

        //is reading qr_code
        private bool is_reading = true;

        public BrowserOpener browserOpener;

        void OnEnable()
        {
            try
            {
                is_reading = true;

                //init camera texture
                cam_texture = new WebCamTexture();

                cam_texture.Play();


                if (Application.platform == RuntimePlatform.Android)
                {
                    raw_image_video.rectTransform.sizeDelta = new Vector2(Screen.width * cam_texture.width / (float)cam_texture.height, Screen.width);
                    raw_image_video.rectTransform.rotation = Quaternion.Euler(0, 0, -90);
                }
                else if (Application.platform == RuntimePlatform.IPhonePlayer)
                {
                    raw_image_video.rectTransform.sizeDelta = new Vector2(1080, 1080 * cam_texture.width / (float)cam_texture.height);
                    raw_image_video.rectTransform.localScale = new Vector3(1, 1, 1);
                    raw_image_video.rectTransform.rotation = Quaternion.Euler(0, 0, 90);
                }
                else
                {
                    raw_image_video.rectTransform.sizeDelta = new Vector2(Camera.main.pixelWidth, Camera.main.pixelWidth * cam_texture.height / (float)cam_texture.width);
                    raw_image_video.rectTransform.localScale = new Vector3(1, 1, 1);
                }

                raw_image_video.texture = cam_texture;
                //}
            }
            catch (Exception ex)
            {
                Debug.Log(ex.Message);
                throw;
            }
        }

        private float interval_time = 1f;
        private float time_stamp = 0;
        void Update()
        {
            if (is_reading)
            {
                time_stamp += Time.deltaTime;

                if (time_stamp > interval_time)
                {
                    time_stamp = 0;

                    try
                    {

                        IBarcodeReader barcodeReader = new BarcodeReader();
                        // decode the current frame
                        var result = barcodeReader.Decode(cam_texture.GetPixels32(), cam_texture.width, cam_texture.height);
                        if (result != null)
                        {

                            browserOpener.pageToOpen = result.Text;
                            browserOpener.OpenPage();
                            is_reading = true;
                        }

                        is_reading = false;
                    }
                    catch (Exception ex)
                    {
                        Debug.LogWarning(ex.Message);
                    }
                }
            }
        }
    }

[tdmowrer]

It looks like your barcode reader operates on a CPU pixel buffer:

var result = barcodeReader.Decode(cam_texture.GetPixels32(), cam_texture.width, cam_texture.height);

You can get this in ARFoundation using the XRCameraImage API. There's a sample in this repo that does this.

[Sterling-Malory-Archer]

It looks like your barcode reader operates on a CPU pixel buffer:

var result = barcodeReader.Decode(cam_texture.GetPixels32(), cam_texture.width, cam_texture.height);

You can get this in ARFoundation using the XRCameraImage API. There's a sample in this repo that does this.

Hi Tim,

Thanks for the reply!

If I add just the TestCameraImage and then pass the texture I get from that script to my barcodeReader, would I be able to decode the QR Code?

Thank you in advance!

[tdmowrer]

The XRCameraImage API does not give you a Texture2D; however you can get an array of pixels, which is what your barcode reader accepts.

This line, for example, converts the camera's raw pixel data to an RGB image and stores it in a NativeArray.

[Sterling-Malory-Archer]

The XRCameraImage API does not give you a Texture2D; however you can get an array of pixels, which is what your barcode reader accepts.

This line, for example, converts the camera's raw pixel data to an RGB image and stores it in a NativeArray.

If you don't mind, could you explain to me how I could reference that in my barcode reader?

[tdmowrer]

I think this should be enough to get you started:

if (cameraManager.TryGetLatestImage(out XRCameraImage image))
{
    // 'image' is an acquired resource that must be diposed when we are done.
    using (image)
    {
        // - Convert the image to RGBA. This takes considerable time, so there are also asynchronous
        //   options. It's easier to understand the synchronous code for this example, though.
        //
        // - It's /much/ faster to convert to a single channel format, e.g., TextureFormat.R8, if
        //   your 'BarcodeReader' can accept that. But if it only accepts an RGBA image, then this
        //   should work too.
        var conversionParams = new XRCameraImageConversionParams(image, TextureFormat.RGBA32, CameraImageTransformation.MirrorY);

        // Get the size (number of bytes) of the buffer required to hold the RGBA pixel data.
        var dataSize = image.GetConvertedDataSize(conversionParams);
        var bytesPerPixel = 4; // because we chose an RGBA format.

        // Since the 'BarcodeReader' code you posted accepts a managed
        // array, that's what I've used here. It would be more efficient
        // to use a NativeArray if the BarcodeReader can use that.
        var pixels = new Color32[dataSize / bytesPerPixel];
        fixed (void* ptr = pixels)
        {
            image.Convert(conversionParams, new IntPtr(ptr), dataSize);
        }

        // The 'pixels' array now contains the image data.
        var result = barcodeReader.Decode(pixels, image.width, image.height);
    }
}

If the BarcodeReader can work with something other than a Color32[] managed array, there are ways this could be made more efficient, namely

• A single channel image would be faster to work with than an RGBA image.
• Using a NativeArray would be better than allocating a managed array. If you need to call this every frame, consider allocating the managed array once and then reusing it.

[Sterling-Malory-Archer]

Hi Tim,

Do I put this code inside Update()? I know fixed needs a method that is declared unsafe.

unsafe void Update()
        {
            if (cameraManager.TryGetLatestImage(out XRCameraImage image))
            {
                // 'image' is an acquired resource that must be diposed when we are done.
                using (image)
                {
                    // - Convert the image to RGBA. This takes considerable time, so there are also asynchronous
                    //   options. It's easier to understand the synchronous code for this example, though.
                    //
                    // - It's /much/ faster to convert to a single channel format, e.g., TextureFormat.R8, if
                    //   your 'BarcodeReader' can accept that. But if it only accepts an RGBA image, then this
                    //   should work too.
                    var conversionParams = new XRCameraImageConversionParams(image, TextureFormat.RGBA32, CameraImageTransformation.MirrorY);

                    // Get the size (number of bytes) of the buffer required to hold the RGBA pixel data.
                    var dataSize = image.GetConvertedDataSize(conversionParams);
                    var bytesPerPixel = 4; // because we chose an RGBA format.

                    // Since the 'BarcodeReader' code you posted accepts a managed
                    // array, that's what I've used here. It would be more efficient
                    // to use a NativeArray if the BarcodeReader can use that.
                    var pixels = new Color32[dataSize / bytesPerPixel];
                    fixed (void* ptr = pixels)
                    {
                        image.Convert(conversionParams, new IntPtr(ptr), dataSize);
                    }

                    IBarcodeReader barcodeReader = new BarcodeReader();
                    // The 'pixels' array now contains the image data.
                    var result = barcodeReader.Decode(pixels, image.width, image.height);
                    // If the result is not null, we set the browserOpener's string to match the text from result.
                    if(result != null)
                    {
                        browserOpener.pageToOpen = result.Text;
                        browserOpener.OnButtonClicked();
                    }
                }
            }
        }

EDIT: It works, thanks a lot Tim!

I got one question, due to this being in Update it keeps opening the browser I have every time I close it, would adding a bool check stop this from happening? Can I do that inside the using()?

[tdmowrer]

Sorry, I missed your last question

I got one question, due to this being in Update it keeps opening the browser I have every time I close it, would adding a bool check stop this from happening? Can I do that inside the using()?

Sure, that would work. You probably only want to run the barcode scanner at certain times, and then turn it off entirely at some point. That kind of processing is pretty expensive.

[kjyv]

Instead of Convert, it's probably better to use image.ConvertAsync(conversionParams, (status, config, data) => {}) and only run that once the previous conversion has finished. Otherwise the camera view will have hiccups. The barcode reader should also run on another thread than the main thread.

[Blackclaws]

Since I'm doing something similar right now let me give you some advice:

• Definitely do it with ConvertAsync and block starting the next detection until the previous one is done, depending on the size you have selected for your XRCameraImage (by setting the currentFormat on the Camera) it might take a while.

• Use a high resolution camera format if you want to decode QR codes farther away. The background is rendered at the high resolution while the default camera stream you use is low res. Change it by setting the camera configuration to high res.

• Do the processing on a thread if possible (not sure how the Asset from the store does it. ZXing.Net definitely needs a thread since it takes approx 0.2-1 sec to process an image.

[Sterling-Malory-Archer]

Hey guys,

I have a question, my code currently works for black and white QR codes, but it doesn't detect colored ones. Can anyone help me with what I should add in order to get that feature as well?

Thank you!

Here's the code:

            if (cameraManager.TryGetLatestImage(out XRCameraImage image))
            {
                // 'image' is an acquired resource that must be diposed when we are done.
                using (image)
                {
                    // - Convert the image to RGBA. This takes considerable time, so there are also asynchronous
                    //   options. It's easier to understand the synchronous code for this example, though.
                    //
                    // - It's /much/ faster to convert to a single channel format, e.g., TextureFormat.R8, if
                    //   your 'BarcodeReader' can accept that. But if it only accepts an RGBA image, then this
                    //   should work too.
                    var conversionParams = new XRCameraImageConversionParams(image, TextureFormat.RGBA32, CameraImageTransformation.MirrorY);

                    // Get the size (number of bytes) of the buffer required to hold the RGBA pixel data.
                    var dataSize = image.GetConvertedDataSize(conversionParams);
                    var bytesPerPixel = 4; // because we chose an RGBA format.

                    // Since the 'BarcodeReader' code you posted accepts a managed
                    // array, that's what I've used here. It would be more efficient
                    // to use a NativeArray if the BarcodeReader can use that.
                    var pixels = new Color32[dataSize / bytesPerPixel];
                    fixed (void* ptr = pixels)
                    {
                        image.Convert(conversionParams, new IntPtr(ptr), dataSize);
                    }

                    IBarcodeReader barcodeReader = new BarcodeReader();
                    // The 'pixels' array now contains the image data.
                    var result = barcodeReader.Decode(pixels, image.width, image.height);
                    // If the result is not null, we set the browserOpener's string to match the text from result.
                    if (result != null)
                    {
                        browserOpener.pageToOpen = result.Text;
                        tmpText.text = result.Text;
                    }
                }

Any help is kindly appreciated!

[tdmowrer]

BarcodeReader is the QR Code scanner asset you purchased from the Asset Store, correct? Does it have documentation or an example that describes the supported input or lists limitations? (Maybe it only handles black and white, for example.)

[Blackclaws]

It seems to me that this decoder is basically the Unity release of ZXing.Net or at least the interface is similar.

ZXing.Net has a binarizer that handles what pixels are treated as black vs which ones are white. This is a pretty internal thing to the library however. You can pass in a custom binarizer directly to the Barcode Reader however. Maybe that would help such as treating anything with a high green value as black.

[Sterling-Malory-Archer]

BarcodeReader is the QR Code scanner asset you purchased from the Asset Store, correct? Does it have documentation or an example that describes the supported input or lists limitations? (Maybe it only handles black and white, for example.)

I read through the documentation, but it doesn't describe the limitations or supported input.

[tdmowrer]

Does the color in your barcode contain information (i.e., would a black and white version work just as well)? You could try converting the image to black and white first. I'm not sure if BarcodeReader requires a 32 bit color image or not, but if it can accept a single channel grayscale image, ARFoundation's image converter can give you a single channel grayscale version of the image rather than convert it to color (it's actually much faster to do so).

[Sterling-Malory-Archer]

Does the color in your barcode contain information (i.e., would a black and white version work just as well)? You could try converting the image to black and white first. I'm not sure if BarcodeReader requires a 32 bit color image or not, but if it can accept a single channel grayscale image, ARFoundation's image converter can give you a single channel grayscale version of the image rather than convert it to color (it's actually much faster to do so).

My QR code contains just text which is then used to turn on and off some gameobjects.

Could you help me with this converting to grayscale?

[tdmowrer]

Could you help me with this converting to grayscale?

it's one of the XRCameraImageConversionParams (the texture format, specifically). But I don't know if your BarcodeReader will accept a grayscale image. You'd have to find that out first. Your code snippet suggests it is expecting an RGBA32 image.

[Sterling-Malory-Archer]

I think it will accept as the documentation does not state it expects RGBA32

[tdmowrer]

barcodeReader.Decode(pixels, image.width, image.height);

Seems to take an array of Color32. Is there another overload that takes an array of byte?

[Sterling-Malory-Archer]

barcodeReader.Decode(pixels, image.width, image.height);

Seems to take an array of Color32. Is there another overload that takes an array of byte?

Found 3 overloads

    Result Decode (byte[] rawRGB, int width, int height, RGBLuminanceSource.BitmapFormat format);

    Result Decode (LuminanceSource luminanceSource);

    Result Decode (Color32[] rawColor32, int width, int height);

[tdmowrer]

You'll need to look at what input those methods expect (what are the allowed values for RGBLuminanceSource.BitmapFormat, for instance?).

Here's how to use ARFoundation's camera conversion API to get a single channel (i.e., grayscale) image:
We support a few texture formats. Single channel formats are ones like TextureFormat.R8 which give you a byte array where each byte represents a single pixel. See XRCameraImage.FormatSupported for a list of supported formats.

[Sterling-Malory-Archer]

You'll need to look at what input those methods expect (what are the allowed values for RGBLuminanceSource.BitmapFormat, for instance?).

Here's how to use ARFoundation's camera conversion API to get a single channel (i.e., grayscale) image:
We support a few texture formats. Single channel formats are ones like TextureFormat.R8 which give you a byte array where each byte represents a single pixel. See XRCameraImage.FormatSupported for a list of supported formats.

Here are allowed values for .BitmapFormat:

public enum BitmapFormat
{
    Unknown,
    Gray8,
    RGB24,
    RGB32,
    ARGB32,
    BGR24,
    BGR32,
    BGRA32,
    RGB565,
    RGBA32
}

[tdmowrer]

Well, Gray8 seems like the right one. Have you tried that?

[Sterling-Malory-Archer]

Well, Gray8 seems like the right one. Have you tried that?

I haven't, I am a bit of a noob when it comes to using XRCameraImage conversions.

That said if someone can point me or help me modify the code above, I would be extremely appreciative.

[tdmowrer]

I have no idea if this will work, but based on what you've provided, this might work:

if (cameraManager.TryGetLatestImage(out XRCameraImage image))
{
    // 'image' is an acquired resource that must be disposed when we are done.
    using (image)
    {
        // - It's /much/ faster to convert to a single channel format, e.g., TextureFormat.R8, if
        //   your 'BarcodeReader' can accept that (looks like it can).
        var conversionParams = new XRCameraImageConversionParams(image, TextureFormat.R8, CameraImageTransformation.MirrorY);

        // Get the size (number of bytes) of the buffer required to hold the grayscale pixel data.
        var dataSize = image.GetConvertedDataSize(conversionParams);

        // Since the 'BarcodeReader' code you posted accepts a managed
        // array, that's what I've used here. It would be more efficient
        // to use a NativeArray if the BarcodeReader can use that.
        var grayscalePixels = new byte[dataSize];
        fixed (void* ptr = grayscalePixels)
        {
            image.Convert(conversionParams, new IntPtr(ptr), dataSize);
        }

        IBarcodeReader barcodeReader = new BarcodeReader();
        // The 'grayscalePixels' array now contains the image data.
        var result = barcodeReader.Decode(grayscalePixels, image.width, image.height, RGBLuminanceSource.BitmapFormat.Gray8);
        // If the result is not null, we set the browserOpener's string to match the text from result.
        if (result != null)
        {
            browserOpener.pageToOpen = result.Text;
            tmpText.text = result.Text;
        }
    }
}

[Sterling-Malory-Archer]

I have no idea if this will work, but based on what you've provided, this might work:

if (cameraManager.TryGetLatestImage(out XRCameraImage image))
{
    // 'image' is an acquired resource that must be disposed when we are done.
    using (image)
    {
        // - It's /much/ faster to convert to a single channel format, e.g., TextureFormat.R8, if
        //   your 'BarcodeReader' can accept that (looks like it can).
        var conversionParams = new XRCameraImageConversionParams(image, TextureFormat.R8, CameraImageTransformation.MirrorY);

        // Get the size (number of bytes) of the buffer required to hold the grayscale pixel data.
        var dataSize = image.GetConvertedDataSize(conversionParams);

        // Since the 'BarcodeReader' code you posted accepts a managed
        // array, that's what I've used here. It would be more efficient
        // to use a NativeArray if the BarcodeReader can use that.
        var grayscalePixels = new byte[dataSize];
        fixed (void* ptr = grayscalePixels)
        {
            image.Convert(conversionParams, new IntPtr(ptr), dataSize);
        }

        IBarcodeReader barcodeReader = new BarcodeReader();
        // The 'grayscalePixels' array now contains the image data.
        var result = barcodeReader.Decode(grayscalePixels, image.width, image.height, RGBLuminanceSource.BitmapFormat.Gray8);
        // If the result is not null, we set the browserOpener's string to match the text from result.
        if (result != null)
        {
            browserOpener.pageToOpen = result.Text;
            tmpText.text = result.Text;
        }
    }
}

Hi Tim,

Sorry for the late reply, I built and tested the scanning with the code you provided here, but it doesn't do anything when I try to scan colored QR's, but it works instantly on black and white QR's.

[tdmowrer]

So it sounds like ARFoundation is providing a grayscale image, then. That's all I can really help you with. I suggest you look for examples, documentation, or a forum/support contact for your barcode reader.

[midopooler]

I was having same question, but can anyone conclude this thread of how to get it to work? (Its been a very long thread to read)
@tdmowrer @Sterling-Malory-Archer @Blackclaws @kjyv