Arcore-unity-sdk: ArCore GPU texture always comes back as ImageFormatGrayscale

[dzak83]

Hi,
Thank you for exposing ArCore GPU texture on CPU but from what I see, it doesn't come back with color and it's always gray.

So setting
TextureReaderApi.ImageFormatType.ImageFormatColor to ImageFormatGrayscale OR ImageFormatColor doesn't change the returned texture.

Thanks,
Jacek

[harrywangwork]

Hi dzak83,
The "ImageFormat" member in TextureReader.cs file is a property that links to the "Image Format" field in the TextureReader component. The default value of the "Image Format" field of the TextureReader component is set to "Image Format Grayscale". If you would like to sample color image, you can change the "Image Format" field from the TextureReader component, or you can set TextureReader.ImageFormat field after the TextureReader component is constructed. Changing the default value of the "ImageFormat" variable in TextureReader.cs has no effect.

[dzak83]

Thank you Haryy, I know that ImageFormat is a member of TextureReader.cs and that default value is ImageFormatGrayscale and you can set it in Unity etc... I am reporting here the problem that when you set it to ImageFormatColor (regardless, if you change the default value, set it in unity or just override it inline in the Update method in the m_TextureReaderApi.Create(..)) it always return gray image for me. I am only getting ARGB32 texture instead of R8, which is expected.

I apologize if it wasn't clear in my first post.

[harrywangwork]

Hi dzak83, I am sorry that I misunderstood you. I realize that you said "gray" image(which is a blank image with gray color), in stead of "grayscale" image.
This is expected since the edge detection algorithm in this sample only handles grayscale image. The sample simply returns if a color image is detected. You can see that in line 88 in ComputerVisionController.cs:

public void OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
{
    if (format != TextureReaderApi.ImageFormatType.ImageFormatGrayscale)
    {
        Debug.Log("No edge detected due to incorrect image format.");
        return;
    }
....
}

If you set the image format to color image, the TextureReader component will sample and return color image for you. You will need to handle the color image in the OnImageAvailable() in your sample.

[dzak83]

Thank you, I am aware about the check in the ComputerVisionController sample code "if (format != TextureReaderApi.ImageFormatType.ImageFormatGrayscale)". If you change that check, you will need later fix the texture format too.

Regardless of what sample code is doing, I believe there is an issue with getting color image. From my observation, image I am getting in the OnImageAvailable() is always in "grayscale", just the format is changing. Can you confirm that you are getting color image?

[harrywangwork]

OnImageAvailable() will receive color image if you change the "Image Format" field in the TextureReader component to "Image Format Color" from Unity UI. You can verify that by replacing your OnImageAvailable() with the following code:

        public void OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
        {
            if (m_TextureToRender == null || m_EdgeImage == null || m_ImageWidth != width || m_ImageHeight != height)
            {
                m_TextureToRender = new Texture2D(width, height, TextureFormat.R8, false, false);
                m_EdgeImage = new byte[width * height];
                m_ImageWidth = width;
                m_ImageHeight = height;
            }

            if (format == TextureReaderApi.ImageFormatType.ImageFormatGrayscale)
            {
                System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeImage, 0, width * height);

            } else {
                byte[] rgbaBuffer = new byte[width * height * 4];
                System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, rgbaBuffer, 0, width * height * 4);

                // Copy the red channel to the grayscale image buffer, since the shader only reads the red channel.
                for (int i = 0; i < width * height; i ++ )
                {
                    m_EdgeImage[i] = rgbaBuffer[i * 4 + 0];
                }
            }
            m_TextureToRender.LoadRawTextureData(m_EdgeImage);
            m_TextureToRender.Apply();
            BackgroundRenderer.BackgroundMaterial.SetTexture("_ImageTex", m_TextureToRender);
        }

This test code shows the grayscale image(instead of edge image) in the lower bottom screen, for both grayscale format and color format. If you set the "Image Format" to color image, you will see the RED channel of the color image, since the shader only reads from red channel.

[harrywangwork]

BTW, if you really want to "see" the color image, you can make the following changes and actually see the color image on the screen:

1. Change line 62 in AREdgeDetectionBackground.shader to:

        gl_FragColor.xyz = color.xyz;

1. Replace OnImageAvailable() function in ComputerVisionController.cs with the following code:

        public void OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
        {
            if (format != TextureReaderApi.ImageFormatType.ImageFormatColor)
            {
                Debug.Log("No edge detected due to incorrect image format.");
                return;
            }

            if (m_TextureToRender == null || m_EdgeImage == null || m_ImageWidth != width || m_ImageHeight != height)
            {
                m_TextureToRender = new Texture2D(width, height, TextureFormat.RGBA32, false, false);
                m_EdgeImage = new byte[width * height * 4];
                m_ImageWidth = width;
                m_ImageHeight = height;
            }

            System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeImage, 0, bufferSize);

            // Update the rendering texture with the sampled image.
            m_TextureToRender.LoadRawTextureData(m_EdgeImage);
            m_TextureToRender.Apply();
            BackgroundRenderer.BackgroundMaterial.SetTexture("_ImageTex", m_TextureToRender);
        }

With the above two changes, if you set the "Image Format" in the TextureReader component UI to "Image Format Color", you will see the color image in the bottom half of the screen. You may need to move the device to see the splitter line between the video and the captured image, since they both are color images.

[dzak83]

Thank you Harry for sample code and investigation. I think what I saw is what you mentioned above, so "RED channel of the color image". It came in grayscale when I used AREdgeDetectionBackground.shader and red when I put it in my own raw image, thus my confusion.

With your provided changes it seems it will work as expect. I will give it a try! Thanks again for the investigation!

[dzak83]

Reopening as I have one follow up question. Is there a way to get a color texture without modifying AREdgeDetectionBackground.shader (gl_FragColor.xyz = color.xyz;) ? Or more precisely, how to get a color screen texture with right rotation without putting it through BackgroundRenderer.BackgroundMaterial.SetTexture("_ImageTex", m_TextureToRender); ?

Maybe there is a way to convert m_TextureToRender?

[harrywangwork]

m_TextureToRender itself is a color texture as it is created as a Texture2D:

m_TextureToRender = new Texture2D(width, height, TextureFormat.RGBA32, false, false);

The line in the shader (gl_FragColor.xyz=color.xyz) is about rendering the texture to the screen.
This page explains the relationship between a texture, a material and a shader. Hope it helps.

[harrywangwork]

Since the initial issue is not a problem, we close this issue. If you have other questions related to ARCore, please create a new issue.

[soniamaryc]

Would you able to access color image by setting TextureReaderApi.ImageFormatType.ImageFormatColor ?
I want to access color image frames for some image processing, but the app is crashing on phone ( only works in unity editor ). I already raised this issue #160 . No responses yet. I couldn't find any references as well.