Arcore-unity-sdk: Camera Capture Image

[markusfehr]

Is it possible to capture a image directly from arcore? My aim is to track the POSE with ARCore and then take a photo directly out of my ARCore App to have the image saved to my gallery and save my POSE to a log file. Like the native camera I want an image with for example 4032x3024 resolution. But I don't want to take a screenshot with the augmented objects. With the GoogleARCore.CameraImageBytes I was only able to get a greyscale picture with 640x480 resolution...
I found some similar SO questions, but I wasn't able to get it working:

• https://stackoverflow.com/questions/49924802/arcore-save-camera-image-unity-c-on-button-click
• https://stackoverflow.com/questions/49579334/save-acquirecameraimagebytes-from-unity-arcore-to-storage-as-an-image

[FelixGirard]

I am also trying to do that. The grayscale is just because it is not converted well. You could get a proper 640x480 color image if you convert the YUV format correctly. Else, if you want full resolution, normally in the ComputerVision example they have a 'TextureReader' with an api that let you take custom resolution image. Although, I've not been able to use that api, see #219.

[pablisho]

Hi, unfortunately using CameraImageBytes you'll get a 640x480 image. But did you take a look at the ComputerVision Example? There we show two alternatives to capture the image:
1) 640x480 using CameraImageBytes
2) High Res using [TextureReaderApi] (https://github.com/google-ar/arcore-unity-sdk/blob/master/Assets/GoogleARCore/Examples/ComputerVision/Scripts/ComputerVisionController.cs#L150)

Let me know if it helps.

[markusfehr]

Hi, thanks for the hint with the TextureReaderApi. It almost works :-) But there are two problems I have:
My resulting picture looks like this:

I wrote "ARCore" on my computer screen, but my captured image is mirrored.
The second thing is that I captured in portrait mode, but the image is rotateted to landscape mode (for my project that's not a big problem, only wanted to mentioned it)

My changes in the "ComputerVisionController.cs" are the following:

public void TakeImage()
        {
            // Function to call from a button to take a picture with the TextureReaderApi
            m_CachedTextureReader.enabled = true;
            m_CachedTextureReader.OnImageAvailableCallback += _OnImageAvailable;
        }

Then I changed the _OnImageAvailable function:

private void _OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
        {
            m_CachedTextureReader.enabled = false;
            m_CachedTextureReader.OnImageAvailableCallback -= _OnImageAvailable;

            m_EdgeDetectionBackgroundTexture = new Texture2D(width, height, TextureFormat.RGBA32, false, false); 
            m_EdgeDetectionResultImage = new byte[width * height * 4]; 

            System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeDetectionResultImage, 0, bufferSize);

            // Save Image
            string date = System.DateTime.Now.ToString("yyyyMMdd_HHmmss");

            m_EdgeDetectionBackgroundTexture.LoadRawTextureData(m_EdgeDetectionResultImage);
            m_EdgeDetectionBackgroundTexture.Apply();

            var encodedPng = m_EdgeDetectionBackgroundTexture.EncodeToPNG();
            var path = Application.persistentDataPath;
            File.WriteAllBytes(path + "/images/" + date + ".png", encodedPng);
        }

Did I missed something or deleted to much from the ComputerVision Example that my picture is mirrored and rotated?

Edit: Okay I think I have another bug in my function. When I take more than one picture, the first picture is saved two times and the last picture isn't saved. I'm not really familar with that callback function, maybe there is something wrong? Thanks for your help :-)

[FelixGirard]

For the rotation and mirroring of the picture it's normal. For the mirroring if you want you can flip it (I can give you the code I use for this purpose). For the rotation you can look at this function in the computerVision example.

[FelixGirard]

And for your other bug I'm not a pro either, but you could try making a boolean like 'requestSavingImage' that your function TakeImage() assign to true. This way the callback would be activated on start(). I'm using this method and it seems to works well.

[markusfehr]

@FelixGirard Thanks. I'll try tomorrow your suggestion and update you if I was successful or not. That would be great if I can get your mirroring code :-)

[FelixGirard]

Hi! Here it is!
The credit : https://forum.unity.com/threads/flipping-texture2d-image-within-unity.35974/

    Texture2D FlipTexture(Texture2D original)
    {
        Texture2D flipped = new Texture2D(original.width, original.height);

        int xN = original.width;
        int yN = original.height;


        for (int i = 0; i < xN; i++)
        {
            for (int j = 0; j < yN; j++)
            {
                flipped.SetPixel(xN - i - 1, j, original.GetPixel(i, j));
            }
        }
        flipped.Apply();

        return flipped;
    }

[markusfehr]

I changed your "FlipTexture" function to also rotate the image correctly landscape/portrait:

Texture2D RotateTexture(Texture2D original, Vector2 topleft, Vector2 topright)
        {
            Texture2D rotated;
            int xN = original.width;
            int yN = original.height;

            if (topleft == new Vector2 (1.0f, 1.0f) && topright == new Vector2(1.0f, 0.0f))
            {
                rotated = new Texture2D(original.height, original.width);

                for (int i = 0; i < xN; i++)
                {
                    for (int j = 0; j < yN; j++)
                    {
                        rotated.SetPixel(yN - j - 1, xN - i -1, original.GetPixel(i, j));
                    }
                }
                rotated.Apply();

            }
            else if (topleft == new Vector2(1.0f, 0.0f) && topright == new Vector2(0.0f, 0.0f))
            {
                rotated = new Texture2D(original.width, original.height);

                for (int i = 0; i < xN; i++)
                {
                    for (int j = 0; j < yN; j++)
                    {
                        rotated.SetPixel(i, yN - j - 1, original.GetPixel(i, j));
                    }
                }
                rotated.Apply();

            }
            else if (topleft == new Vector2(0.0f, 0.0f) && topright == new Vector2(0.0f, 1.0f))
            {
                rotated = new Texture2D(original.height, original.width);

                for (int i = 0; i < xN; i++)
                {
                    for (int j = 0; j < yN; j++)
                    {
                        rotated.SetPixel(j, i, original.GetPixel(i, j));
                    }
                }
                rotated.Apply();
            }
            else
            {
                rotated = new Texture2D(original.width, original.height);

                for (int i = 0; i < xN; i++)
                {
                    for (int j = 0; j < yN; j++)
                    {
                        rotated.SetPixel(xN - i - 1, j, original.GetPixel(i, j));
                    }
                }
                rotated.Apply();
            }

            return rotated;
        }

It works, but the double for-loop is ineficent, does someone know a better solution?

And another problem/question has appeared: I'm using a Samsung S8, the image sensor has an aspect ratio from 4/3 but the display is something like 2:1 (default display resolution 2220x1080 pixels). With the TextureReaderAPI is it possible to get the full image from the camera (4/3) and not a cropped one? Because when I tried to save a 4/3 image with the TextureReaderAPI and compare it with what I see on my display the image is cropped on top and buttom... or can I only get images with the same aspect ratio the display has?

[zarubin-alexey]

@markusfehr Which way you are using the image? in RawImage component you can apply some transform.scale to flip the image. This way it would be more efficient rather than use "SetPixel" function.

[markusfehr]

@zarubin-alexey I just want to save the image as png file. I don't need the captured image in my app. Like I explained in my first post, I want to save an image out of ARCore and write the current position and orientation to a logfile. At the moment it works, but the double for-loop isn't efficient.

[zarubin-alexey]

@markusfehr I suggest you take image bytes array, apply changes to it and "LoadTextureBytes" to texture. AFAIK GetPixel and SetPixel functions are slow, work with bytes array would be the better choice.

[pablisho]

@markusfehr The TextureReaderAPI is actually reading the values from the texture that is used to render in the display, that's why you get the exact same format and resolution that you have in it, and is specific to the device.

[markusfehr]

@zarubin-alexey at the moment I use the GetPixel/SetPixel in a Coroutine, so the image is actually saved about 5 seconds after it was taken. How can I mirroring and rotateing a byte array, is it something like this: https://stackoverflow.com/questions/9199866/algorithm-to-vertically-flip-a-bitmap-in-a-byte-array
Or do I have to assign the byte array to a raw image unity object?
@pablisho thanks for the answer :-)

[pablisho]

I'm closing this as I believe the questions were solved, feel free to reopen if you have more questions.

[ZeroLu]

Hi there,
I encountered the exact problem and need to mirror and flip the texture. I have read the entire thread and the question of

How to efficiently(in real time) mirror and flip the texture for saving to disk?

has not been answered.
Do we need to use a raw image unity component or use a good algorithm on the byte array?
@pablisho , could you give us some insights? Thank you in advance!

[ZeroLu]

[UPDATE] According to @abrageor , it's true that a RenderTexture must be provided to the Graphics.blit API. And to save this RenderTexture, it seems that it have to be converted to Texture2D with ReadPixels. I updated my answer below.
I actually used NatCorder to achieve 30fps video recording.

After many attempts, I finally succeed in correcting the rotation and mirroring of the texture.
I tried changing the byte directly but that's very slow.
My solution is to use Graphics.Blit to copy the old texture to a new one along with a material. The material uses a shader to do the transformation on the texture.
This is quite similar to ARCore's Computer Vision's own approach, but I tried their shader and in vain. So I wrote my own shader:

Shader "ZeroLu/TextureTransform" {
    Properties{
        _MainTex("Texture", 2D) = "white" {}
        _Rotation("Rotation", float) = 0
        _Scale("Scale", float) = 1
    }
    SubShader{

        Tags{
            "Queue" = "Transparent"
            "RenderType" = "Transparent"
            "IgnoreProjector" = "True"
            "PreviewType" = "Plane"
            "CanUseSpriteAtlas" = "True"
        }

        Cull Off
        ZWrite Off
        ZTest Always
        Lighting Off
        Fog{ Mode off }
        ZTest[unity_GUIZTestMode]
        Blend SrcAlpha OneMinusSrcAlpha

        Pass{
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };

            uniform fixed _Rotation, _Scale, _Greyness;

            v2f vert(appdata v) {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;// - float2(0.5, 0.5);
                float s, c;
                sincos(_Rotation * 3.141592653 / 180, s, c);
                float2x2 transform = mul(float2x2(
                    float2(c, -s),
                    float2(s, c)
                    ), float2x2(
                        float2(_Scale, 0.0),
                        float2(0.0, 1.0)
                        ));
                o.uv = mul(transform, o.uv);// +float2(0.5, 0.5);
                return o;
            }

            sampler2D _MainTex;

            fixed4 frag(v2f i) : SV_Target{
                fixed4 col = tex2D(_MainTex, i.uv);
                return col;
            }
            ENDCG
        }
    }
}

Then create a material and set the value like the following:

I modified on top of markusfehr's code(* The code below is not tested because I actually used NatCorder in my own project, which is implemented differently, so feel free to modify it. *):

private void _OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
        {
            m_CachedTextureReader.enabled = false;
            m_CachedTextureReader.OnImageAvailableCallback -= _OnImageAvailable;

            m_EdgeDetectionBackgroundTexture = new Texture2D(width, height, TextureFormat.RGBA32, false, false); 
            m_EdgeDetectionResultImage = new byte[width * height * 4]; 
            System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeDetectionResultImage, 0, bufferSize);

            // Save Image
            string date = System.DateTime.Now.ToString("yyyyMMdd_HHmmss");
           m_EdgeDetectionBackgroundTexture.LoadRawTextureData(m_EdgeDetectionResultImage);
            m_EdgeDetectionBackgroundTexture.Apply();
           RenderTexture finalRenderTexture = new RenderTexture(height, width, 24);
          Graphics.Blit(m_EdgeDetectionBackgroundTexture, finalRenderTexture, TextureTransformMaterial);
           RenderTexture.active = finalRenderTexture;
           Texture2D finalTexture = new Texture2D(height, width, TextureFormat.RGB24, false);
            finalTexture.ReadPixels(new Rect(0, 0, height, width), 0, 0, false);
            var encodedPng = finalTexture.EncodeToPNG();
            var path = Application.persistentDataPath;
            File.WriteAllBytes(path + "/images/" + date + ".png", encodedPng);
            // clean up
            RenderTexture.active = null;
            Destroy(finalRenderTexture);
            Destroy(finalTexture);
        }

You can only use this to save screenshot, not video.
Only with the help of NatCorder can I use this to record video at 30fps.
Hope that this can help someone. Feel free to provide more feedback.

[AvraamG]

@ZeroLu thank you for the suggestion of Shader flip and Graphics.Blit One question though. Are you sure your code is working with Graphics.Blit(m_EdgeDetectionBackgroundTexture, finalTexture, TextureTransformMaterial); I dont see any overload method for Graphics.Blit that has Texture2d,Texture2d https://docs.unity3d.com/ScriptReference/Graphics.Blit.html

[ZeroLu]

@abrageor Texture2d will be treated as Texture in Unity.

[AvraamG]

In this line if I understand correctly you are using Blit with Texture2D,Texture2D,Material Graphics.Blit(m_EdgeDetectionBackgroundTexture, finalTexture, TextureTransformMaterial); I am not 100% sure what TextureTransformMaterial is but I assume it is declared somewhere as a material right? So based on the documentation Graphics.Blit requires a RenderTexture as the second input of its signature. Thats why I am confused. Finally, I think by using new Texture you are basically introducing a memory leak here as the memory of this new Texture is never released.

[ZeroLu]

@abrageor , you are indeed right.
I used NatCorder to achieve video recording and in that I Blit the texture into a RenderTexture. The code I wrote here is not properly tested. I am sorry for this.

TextureTransformMaterial refers to the material I created with my TextureTransform shader.

And it's true that there will be a memory leak. I updated my answer above.

[AvraamG]

No worries, I was just wondering what I was doing wrong. If you could elaborate on the solution it would be very much appreciated.

[ZeroLu]

@abrageor , I think that I will need to write a detailed blogpost on this topic. And it doesn't suit this issue(Camera Captured Image) either. I will send you an email when it's done.

[markusfehr]

@ZeroLu thanks for your solution and your custom shader. The mirroring works perfect, but I'm not sure if the rotation to landscape works? I'm also interesting in your blogpost regarding this topic :-)

[HooliganLabs]

Could someone please share the complete code for capturing a color image with ARcore?

[pranabeshdash]

Hi, thanks for the hint with the TextureReaderApi. It almost works :-) But there are two problems I have:
My resulting picture looks like this:

I wrote "ARCore" on my computer screen, but my captured image is mirrored.
The second thing is that I captured in portrait mode, but the image is rotateted to landscape mode (for my project that's not a big problem, only wanted to mentioned it)

My changes in the "ComputerVisionController.cs" are the following:

public void TakeImage()
        {
            // Function to call from a button to take a picture with the TextureReaderApi
            m_CachedTextureReader.enabled = true;
            m_CachedTextureReader.OnImageAvailableCallback += _OnImageAvailable;
        }

Then I changed the _OnImageAvailable function:

private void _OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
        {
            m_CachedTextureReader.enabled = false;
            m_CachedTextureReader.OnImageAvailableCallback -= _OnImageAvailable;

            m_EdgeDetectionBackgroundTexture = new Texture2D(width, height, TextureFormat.RGBA32, false, false); 
            m_EdgeDetectionResultImage = new byte[width * height * 4]; 

            System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeDetectionResultImage, 0, bufferSize);

            // Save Image
            string date = System.DateTime.Now.ToString("yyyyMMdd_HHmmss");

            m_EdgeDetectionBackgroundTexture.LoadRawTextureData(m_EdgeDetectionResultImage);
            m_EdgeDetectionBackgroundTexture.Apply();

            var encodedPng = m_EdgeDetectionBackgroundTexture.EncodeToPNG();
            var path = Application.persistentDataPath;
            File.WriteAllBytes(path + "/images/" + date + ".png", encodedPng);
        }

Did I missed something or deleted to much from the ComputerVision Example that my picture is mirrored and rotated?

Edit: Okay I think I have another bug in my function. When I take more than one picture, the first picture is saved two times and the last picture isn't saved. I'm not really familar with that callback function, maybe there is something wrong? Thanks for your help :-)

>
Hi Markusfehr - were you able to capture a full resolution image (4032x3024) as stated in your original query or the resolution is limited by the screen resolution?

[markusfehr]

@pranabeshdash No with the Texture Reader I only got max resolution from display. In my case with samsung s8 2960x1440 -> see: https://github.com/google-ar/arcore-unity-sdk/issues/221#issuecomment-394481171

[Yaffa16]

ks for the hint with the TextureReaderApi. It almost works :-) But there are two problems I have:
My resulting picture looks like this:

HI,
I made these changes to the ComputerVision script, my image is always black. Any idea why?