Mediapipe: For Iris Desktop GPU example, got the error "Side packet "focal_length_pixel" is required but was not provided."

[prantoran]

I tried running the Iris GPU example in Ubuntu 20.04 LTS.

I ran the following for building:

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/iris_tracking:iris_tracking_gpu

And the following for running the binary:

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt

The build was successful but I received the following error when I tried to run:

$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt
I20201106 18:49:44.087379 19292 demo_run_graph_main_gpu.cc:51] Get calculator graph config contents: # MediaPipe graph that performs iris tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipie/examples/android/src/java/com/mediapipe/apps/iristrackinggpu and

# GPU buffer. (GpuBuffer)
input_stream: "input_video"
....
....
I20201106 18:49:44.088325 19292 demo_run_graph_main_gpu.cc:57] Initialize the calculator graph.
I20201106 18:49:44.093295 19292 demo_run_graph_main_gpu.cc:61] Initialize the GPU.
I20201106 18:49:44.104634 19292 gl_context_egl.cc:158] Successfully initialized EGL. Major : 1 Minor: 4
I20201106 18:49:44.108723 19305 gl_context.cc:324] GL version: 3.1 (OpenGL ES 3.1 Mesa 20.0.8)
I20201106 18:49:44.108911 19292 demo_run_graph_main_gpu.cc:67] Initialize the camera or load the video.
Gtk-Message: 18:49:44.297: Failed to load module "canberra-gtk-module"
I20201106 18:49:44.308125 19292 demo_run_graph_main_gpu.cc:88] Start running the calculator graph.
E20201106 18:49:44.313082 19292 demo_run_graph_main_gpu.cc:186] Failed to run the graph: ValidateRequiredSidePackets failed to validate: 
; Side packet "focal_length_pixel" is required but was not provided.

[prantoran]

If I comment out the following line from mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt, then I get the error Only BGRA/RGBA textures are supported, passed format: 24.

Commented out statement:

node {
...
#  input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
...
}

Error:

I20201106 20:12:37.840118 21807 demo_run_graph_main_gpu.cc:57] Initialize the calculator graph.
I20201106 20:12:37.844599 21807 demo_run_graph_main_gpu.cc:61] Initialize the GPU.
I20201106 20:12:37.855563 21807 gl_context_egl.cc:158] Successfully initialized EGL. Major : 1 Minor: 4
I20201106 20:12:37.860030 21820 gl_context.cc:324] GL version: 3.1 (OpenGL ES 3.1 Mesa 20.0.8)
I20201106 20:12:37.860173 21807 demo_run_graph_main_gpu.cc:67] Initialize the camera or load the video.
Gtk-Message: 20:12:38.025: Failed to load module "canberra-gtk-module"
I20201106 20:12:38.031477 21807 demo_run_graph_main_gpu.cc:88] Start running the calculator graph.
I20201106 20:12:38.035310 21807 demo_run_graph_main_gpu.cc:93] Start grabbing and processing frames.
INFO: Created TensorFlow Lite delegate for GPU.
ERROR: Following operations are not supported by GPU delegate:
DEQUANTIZE: 
164 operations will run on the GPU, and the remaining 0 operations will run on the CPU.
I20201106 20:12:47.249657 21807 demo_run_graph_main_gpu.cc:175] Shutting down.
E20201106 20:12:47.410118 21807 demo_run_graph_main_gpu.cc:186] Failed to run the graph: CalculatorGraph::Run() failed in Run: 
Calculator::Process() for node "facelandmarkfrontgpu__facedetectionfrontgpu__ImageToTensorCalculator" failed: Only BGRA/RGBA textures are supported, passed format: 24

[prantoran]

Was able to run by converting RGB to RGBA in mediapipe/examples/desktop/demo_run_graph_main_gpu.cc:

...
if (!load_video) {
      cv::flip(camera_frame, camera_frame, /*flipcode=HORIZONTAL*/ 1);
    }

    uchar* camData = new uchar[camera_frame.total()*4];
    cv::Mat continuousRGBA(camera_frame.size(), CV_8UC4, camData);
    cv::cvtColor(camera_frame, continuousRGBA, CV_RGB2RGBA, 4);

    // Wrap Mat into an ImageFrame.
    auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
        mediapipe::ImageFormat::SRGBA, camera_frame.cols, camera_frame.rows,
        mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);
    cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
    camera_frame.copyTo(input_frame_mat);
...

[Rassibassi]

hej, same problem, for me your code didnt work out, but this instead:

    cv::cvtColor(camera_frame_raw, camera_frame, cv::COLOR_BGR2RGBA);
    if (!load_video) {
      cv::flip(camera_frame, camera_frame, /*flipcode=HORIZONTAL*/ 1);
    }

    // Wrap Mat into an ImageFrame.
    auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
        mediapipe::ImageFormat::SRGBA, camera_frame.cols, camera_frame.rows,
        mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);
    cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
    camera_frame.copyTo(input_frame_mat);

basically adding a A to both COLOR_BGR2RGB and to SRGB

[prantoran]

Nice! That is concise and would save running time. I guess A just adds another channel for alpha transparency value

[HelloWord2020]

hej, same problem, for me your code didnt work out, but this instead:

    cv::cvtColor(camera_frame_raw, camera_frame, cv::COLOR_BGR2RGBA);
    if (!load_video) {
      cv::flip(camera_frame, camera_frame, /*flipcode=HORIZONTAL*/ 1);
    }

    // Wrap Mat into an ImageFrame.
    auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
        mediapipe::ImageFormat::SRGBA, camera_frame.cols, camera_frame.rows,
        mediapipe::ImageFrame::kGlDefaultAlignmentBoundary);
    cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
    camera_frame.copyTo(input_frame_mat);

basically adding a A to both COLOR_BGR2RGB and to SRGB

Yes, your code worked out from my side too! Thank you!

[ianni67]

Thank you both, for pinpointing the problem and fixing it in the best way.
The more we share, the farther we go.