Azure-kinect-sensor-sdk: CSharp Depthmode/IR Fails in Unity

[jmackenzie68]

Is your feature request related to a problem? Please describe.
Accessing the Depth camera or IR camera of the Azure Kinect device does not seem to work in Unity. We can get the color Camera to display however. See Script below.

When calling Device.StartCameras(DeviceConfiguration), it fails with "result = K4A_RESULT_FAILED" with almost any camera configuration. Currently we can only get a success call if we are using ColorFormat MJPG or BGRA32 with Depthmode = off and all other settings at defaults, expect resolution which work between 720p-2160p(haven't tested 3072P).

As it fails anytime depthmode is turned on perhaps its related to [THIS] (https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/429)

Describe the solution you'd like
In Unity, call StartCameras(DeviceConfiguration) with a depthmode valid setup and have it succeed. We are also seeing FAILED on IR camera setup and Depth Only.

Additional context
To get "Microsoft.Azure.Kinect.Sensor" in Unity we built the cSharp and added the compliled dll and pdb files to Assets/Plugins.

We get a successful camera feed with MJPG and BGRA32 setup as long as Depthmode is set to off.

Also, the Viewer and Bodytracker SDKs are both working on this PC.

```c#
using Microsoft.Azure.Kinect.Sensor;
using System;
using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.UI;

using Image = Microsoft.Azure.Kinect.Sensor.Image;

public class WebCamManager : MonoBehaviour
{
public GameObject textureHolder;
Texture2D texture;
public RawImage rawimage;

Image image;
Image depthImage;
Capture capture = null;
Device device = null;
DeviceConfiguration config;

void Start()
{
    config = new DeviceConfiguration();

    config.ColorFormat = ImageFormat.ColorBGRA32; 
    config.ColorResolution = ColorResolution.r1080p;
    config.DepthMode = DepthMode.NFOV_2x2Binned; 
    config.CameraFPS = FPS.fps30;
    //config.SynchronizedImagesOnly = true;
    //config.DepthDelayOffColor = TimeSpan.Zero;
    //config.DisableStreamingIndicator = false;
    //config.SuboridinateDelayOffMaster = TimeSpan.Zero;
    //config.SynchronizedImagesOnly = false;
    //config.WiredSyncMode = WiredSyncMode.Standalone;

    device = Device.Open(0);

    try
    {
        //device.GetCalibration(config.DepthMode, config.ColorResolution);

        device.StartCameras(config);
    }
    catch (Exception e)
    {
        throw e;
    }

    capture = Capture.Create();
    image = new Image(config.ColorFormat, 1920, 1080, 1920);
    depthImage = new Image(ImageFormat.Depth16, 1920, 1080, 1920);

    texture = new Texture2D(1920, 1080, TextureFormat.BGRA32, false);

    rawimage.texture = texture;
}



private void Update()
{

    //TODO:add each possible config setup

    //MJPEG 1080p
    if (config.ColorFormat == ImageFormat.ColorMJPG)
    {
        capture = device.GetCapture(-1);
        image = capture.Color;

        byte[] byteArray = image.GetBufferCopy();
        texture.LoadImage(byteArray, false);
        texture.Apply();
    }

    //BGRA 1080p
    if (config.ColorFormat == ImageFormat.ColorBGRA32)
    {

        capture = device.GetCapture(-1);
        image = capture.Color;
        depthImage = capture.Depth;

        byte[] byteArray = image.GetBufferCopy();
        texture.LoadRawTextureData(byteArray);
        texture.Apply();

    }
}

public void StopCameras()
{
    device.StopCameras();
}

private void OnDisable()
{
    StopCameras();
}

```

[Brent-A]

Hi @jmackenzie68. I think some logs would help narrow down the issue. By default the SDK logs to the console, which I'm not sure if you can see. You can switch it to log to a file using some environment variables.

[Brent-A]

At a guess, did you include depthengine_1_0.dll? That dependency doesn't get included in the SDK when you build it yourself so it's easy to miss.

[jmackenzie68]

@Brent-A Thank you for the response! Adding the folder containing depthengine_1_0.dll was fixed this error. Originally I had added a copy of the depthengine_1_0.dll to the Unity Plugins folder sitting next to the k4A.dll and Microsoft.Azure.Kinect.Sensor.dll (cSharp Build) but apparently that was not enough. I still am not getting a depth image at this point, but I am able to get a success from StartCameras.

[djambo]

I had some time to troubleshoot this and got it to work. Seems like the main problem with your unity script is here:

depthImage = new Image(ImageFormat.Depth16, 1920, 1080, 1920);

Your are trying to get a depth map of 1920x1080 resolution, while the supported resolution for the mode you selected, (NFOV_2x2Binned) is 320x288 px.

You should be able to get it to work changing that line to something like this:
depthImage = new Image(ImageFormat.Depth16, 320, 288, 320);

Since you are trying to grab the NFOV image, if you want a higher resolution depth map, I would suggest you to use:

DepthMode.WFOV_2x2Binned and
depthImage = new Image(ImageFormat.Depth16, 640, 576, 640);

[venscn]

我有时间对此进行故障排除并使其正常工作。好像你的统一脚本的主要问题在这里：

depthImage = new Image(ImageFormat.Depth16, 1920, 1080, 1920);

您正在尝试获得1920x1080分辨率的深度贴图，而您选择的模式支持的分辨率（NFOV_2x2Binned）为320x288像素。

您应该能够将其更改为以下内容：
depthImage = new Image(ImageFormat.Depth16, 320, 288, 320);

由于您试图获取NFOV图像，如果您想要更高分辨率的深度图，我建议您使用：

DepthMode.WFOV_2x2Binned 和
depthImage = new Image(ImageFormat.Depth16, 640, 576, 640);

DepthMode.WFOV_2x2Binned
depthImage = new Image(ImageFormat.Depth16, 512, 512, 512 * 2);

DepthMode.WFOV_2x2 UnBinned
depthImage = new Image(ImageFormat.Depth16, 1024, 1024, 1024 * 2);

[venscn]

stride_bytes is error;
bgra32: width * 4;
depth: width * 2;

[venscn]

u can control the config and use the this.device.GetCalibration().depth_camera_calibration.resolution_width and resolution_height to see the depth image width and height

[WheezardX]

@jmackenzie68 Can you clarify your solution to getting depthengine_1_0.dll to load in Unity? I have the same issue that cameras won't start and it is because depthengine_1_0.dll is not loaded. I put it in the plugin folder but it fails to load. You mention a folder but could you clarify? Thanks!

[djambo]

@WheezardX Putting the depthengine_1_0.dll in the root folder of your unity project should work. For root folder I mean the one containing: Assets, Library ecc.

[WheezardX]

@djambo You are a savior! I've been over here trying to hack shared memory DLL to run the Kinect in a native app to communicate to Unity and that has been a disaster.

[MsDerekMa]

This should be the same issue as #538, which was just fixed.