Azure-kinect-sensor-sdk: "IR to Color" transformation missing

[fractalfantasy]

Tried to get IR to color space conversion using k4a_transformation_depth_image_to_color_camera_custom, but it doesn't seem to work.

an "IR to Depth" transformation should be very easy to add, as it's pretty much exactly the same as the Depth to Color transformation.

[rabbitdaxi]

@fractalfantasy the k4a_transformation_depth_image_to_color_camera_custom should be able to fulfill the purpose here. please provide details of how you are using this function and what issue/error you were hitting. We haven't receive any report of people having issue using this function yet.

[mikkamikka]

@rabbitdaxi We are trying to add k4a_transformation_depth_image_to_color_camera_custom IR to Color transformation to our DLL to be used in Electron app.
Here is the code snippet. I omit all the variables initialization and ir, depth, color frames 'get' functions, as there's no problem with them.

`int ir_to_color_image_created = k4a_image_create(K4A_IMAGE_FORMAT_CUSTOM16, jsFrame.colorImageFrame.width, jsFrame.colorImageFrame.height, jsFrame.colorImageFrame.width * 2, &ir_to_color_image);

int depth_to_color_image_created = k4a_image_create(K4A_IMAGE_FORMAT_DEPTH16, jsFrame.colorImageFrame.width, jsFrame.colorImageFrame.height, jsFrame.colorImageFrame.width * 2, &depth_to_color_image);

if ( (ir_to_color_image_created == K4A_RESULT_SUCCEEDED) && (depth_to_color_image_created == K4A_RESULT_SUCCEEDED) )
{

    jsFrame.irToColorImageFrame.height = 101; //debug

    if (k4a_transformation_depth_image_to_color_camera_custom(transformer, depth_image, ir_image, depth_to_color_image, ir_to_color_image,
        K4A_TRANSFORMATION_INTERPOLATION_TYPE_NEAREST , 0) == K4A_RESULT_SUCCEEDED)
    {
        jsFrame.irToColorImageFrame.width = 102; //debug

        jsFrame.irToColorImageFrame.width = k4a_image_get_width_pixels(ir_to_color_image);
        jsFrame.irToColorImageFrame.height = k4a_image_get_height_pixels(ir_to_color_image);

        jsFrame.irToColorImageFrame.image_length = k4a_image_get_size(ir_to_color_image);
        jsFrame.irToColorImageFrame.stride_bytes = k4a_image_get_stride_bytes(ir_to_color_image);
        jsFrame.irToColorImageFrame.image_data = new uint8_t[jsFrame.irToColorImageFrame.image_length];

    }
    else {
        jsFrame.irToColorImageFrame.width = 1;
    }
}`

k4a_transformation_depth_image_to_color_camera_custom function never succeeded. Output ir_to_color_image data array is empty.
Custom image format is set to K4A_IMAGE_FORMAT_CUSTOM16

Similar k4a_transformation_depth_image_to_color_camera(transformer, depth_image, depth_to_color_image) function works just fine in the same context.

What are we missing?

Sensor config:
depth_mode: KinectAzure.K4A_DEPTH_MODE_WFOV_2X2BINNED,
color_format: KinectAzure.K4A_IMAGE_FORMAT_COLOR_BGRA32,
color_resolution: KinectAzure.K4A_COLOR_RESOLUTION_720P,
camera_fps: KinectAzure.K4A_FRAMES_PER_SECOND_30

[fractalfantasy]

@rabbitdaxi do you have an idea of what could be wrong here? ^

[rabbitdaxi]

@fractalfantasy sorry for the delay response (I am working on other projects now, so not checking this GitHub that frequent).

I see the issue. The k4a_transformation_depth_image_to_color_camera API requires both input and output custom images have the same custom format, e.g. in your case, K4A_IMAGE_FORMAT_CUSTOM16.

In your sample code, you passed the IR image directly into this API, the code internally has a check the format to match between input and output custom images failed. I know it looks like a extra step, but based on how the SDK expectation, you can create a custom_ir_image as the input custom image with the matching format of the output custom image, see below code snippet:

k4a_image_t custom_ir_image = NULL;
int ir_image_width_pixels = k4a_image_get_width_pixels(ir_image);
int ir_image_height_pixels = k4a_image_get_height_pixels(ir_image);
int ir_image_stride_bytes = k4a_image_get_stride_bytes(ir_image);
uint8_t *ir_image_buffer = k4a_image_get_buffer(ir_image);
if (K4A_RESULT_SUCCEEDED != 
            k4a_image_create_from_buffer(K4A_IMAGE_FORMAT_CUSTOM16,
                                         ir_image_width_pixels,
                                         ir_image_height_pixels,
                                         ir_image_width_pixels * (int)sizeof(uint16_t),
                                         ir_image_buffer,
                                         ir_image_height_pixels * ir_image_stride_bytes,
                                         [](void *_buffer, void *context) {
                                            delete[](uint8_t *) _buffer;
                                            (void)context;
                                         },
                                         NULL,
                                         &custom_ir_image))
{
    printf("Failed to create custom ir image\n");
    return false;
}

Then instead of passing in the ir_image, you pass in the custom_ir_image into this API

if (K4A_RESULT_SUCCEEDED !=
            k4a_transformation_depth_image_to_color_camera_custom(transformation,
                                                                  depth_image,
                                                                  custom_ir_image,
                                                                  transformed_depth_image,
                                                                  transformed_ir_image,
                                                                  K4A_TRANSFORMATION_INTERPOLATION_TYPE_NEAREST,
                                                                  0))
{
    printf("Failed to compute transformed depth and custom image\n");
    return false;
}

[mikkamikka]

@rabbitdaxi Thank you for pointing out the right solution. However, creating a custom image from buffer with k4a_image_create_from_buffer method with the parameters specified in your code sample has failed.

I've changed 'stride_bytes' parameter in the above method from ir_image_width_pixels * (int)sizeof(uint16_t) to ir_image_stride_bytes. And that solved the issue.
Now everything works as expected. Thanks again.

Here's the working method:

k4a_image_create_from_buffer(K4A_IMAGE_FORMAT_CUSTOM16,
                             ir_image_width_pixels,
                             ir_image_height_pixels,
                             ir_image_stride_bytes,
                             ir_image_buffer,
                             ir_image_height_pixels * ir_image_stride_bytes,
                             [](void *_buffer, void *context) {
                                  delete[](uint8_t *) _buffer;
                                  (void)context;
                             },
                             NULL,
                             &custom_ir_image)

cc @fractalfantasy

[rabbitdaxi]

Sounds good. Feel free to close if this resolved your issue :)