Azure-kinect-sensor-sdk: crash/hang in g_deloader.plugin.depth_engine_create_and_initialize

[krasin]

The issue affects Ubuntu 18.04 and Kinect SDK 1.1.1 (the latest at the time of filing this bug).

Firmware version:

RGB camera firmware: 1.6.98
Depth camera firmware: 1.6.70
Depth config file: 6109.7
Audio firmware: 1.6.14
Build Config: Production
Certificate Type: Microsoft

Steps to reproduce:

1. Download https://github.com/microsoft/Azure-Kinect-Sensor-SDK sources and install dependencies according to instructions.
2. Build it:

mkdir build
cd build
cmake -GNinja ..
ninja

alternatively with Make:

mkdir build
cd build
cmake ..
make -j4

1. Run depth_ft test with a Kinect for Azure camera connected to the computer:

./bin/depth_ft
...
PASS (all tests)
...

1. Switch to a different TTY by pressing Ctrl+Alt+F4. Log into the computer, go to the build directory and run the test again:

export DISPLAY=:0
./bin/depth_ft

If no NVIDIA driver is installed, it will hang indefinitely until the user switches to the original TTY with Ctrl+Alt+F2 and then back with Ctrl+Alt+F4. Then the test passes.

The hang happens in this code:

return g_deloader.plugin.depth_engine_create_and_initialize(context,
                                                                cal_block_size_in_bytes,
                                                                cal_block,
                                                                mode,
                                                                input_format,
                                                                camera_calibration,
                                                                callback,
                                                                callback_context);
}

With NVIDIA driver installed, this timeout is finite (~2 seconds) and then it just says that frames are not captured.

The underlying issue seems to be the same: something inside the binary blob need UI to process something.

It would be very nice to be able to run Kinect Azure camera in a headless mode (and switching to a different TTY with Ctrl+Alt+F4 is just an approximation of that). In particular, because it would allow to access the camera from systemd services, which is handy for non-interactive applications, such as robotics.

I will be happy to provide any additional context and run any proposed workaround. I am also open to be a guinea pig for any unreleased firmware or depth engine versions, if there's a feel that the issue might already be fixed.

Thank you in advance. Kinect Azure camera is pretty awesome! :)

[rabbitdaxi]

related to #495

[cdedmonds]

@krasin the Azure Kinect depth engine has a dependency on OpenGL. Unfortunatley on Linux it is necessary to have an X server running and a Windows present for OpenGL to function correctly. On headless machines it is possible to get an X server running without a display.

[krasin]

@cdedmonds yes, I understand that. The machine does have X server running (and DISPLAY variable is properly set up). OpenGL is also working and of the correct version.

There's some one-time access to the "real" UI screen that freezes the program. If I just switch to real UI once and then back to another tty, the program runs successfully.

I would conclude that this is a separate issue from regular "headless" requests. It's okay for us to run X server, but we need to be able to run our program under systemd. And that currently causes this hang.

[krasin]

I will probably spend more time today to record what's happening and hopefully making it easier to reproduce on your side.

[krasin]

Apparently, Ctrl+Alt+F4 trick is a bad proxy for systemd. :)

I've got everything working under systemd with just DISPLAY and XDG_RUNTIME_DIR env vars set. And while it would still hang in the case described in my original report (so, the bug is real), it's no longer a blocker for our team.

Thank you for your replies and for working on this awesome tech!

[ruis1996]

@krasin
Hi krasin.
I'm sorry to bother you.
I have encountered the same problem as you. Can you tell me how you solved it?
Thank you.

[krasin]

@ruis1996 as mentioned in my previous update, I was unable to solve the issue when running under a different terminal session.

But my problem was to run it under systemd, so providing proper values for DISPLAY and XDG_RUNTIME_DIR was enough.

This is an example of systemd config that should work (adopted from the one I wrote):

[Unit]
Description=Kinect Service

[Service]
Type=simple
Restart=always
RestartSec=3
User=myuser
Environment=XDG_RUNTIME_DIR=/run/user/1000
Environment=DISPLAY=:0
WorkingDirectory=/opt/my-kinect/
ExecStart=/opt/my-kinect/my-kinect
ExecStop=

[Install]
WantedBy=multi-user.target

It's important to make sure that /run/user/1000 matches the user id for the user (myuser) in the example. To check that, run:

id -u myuser

Overall, I believe that Kinect depth engine library has a bug, and it should ideally be fixed. Unfortunately, given that it's a precompiled blob, the community is not empowered to understand what exactly is wrong there.

[StevenButner]

Many thanks to @krasin for contributing a systemd unit that can be used to configure and run a program that grabs depth frames from the K4A. I need to do this for my robotics application. Unfortunately, I have not been able to find a way to extend @krasin's approach to operate on a fully-headless system (which my system is). The key issue is that the userid that is referenced in the systemd unit must be logged in and the login needs to be via a graphics desktop (i.e. something with an actual screen). I have tried using Xdummy, xserver-xorg-video-dummy, and xpra and many, many variations on similar themes. I've managed to get "close" to a solution but in every case the requirement remains for a real physically-attached graphics screen.

I'm troubled that issue #562 got closed. There remains #796, but this is marked as "Documentation" rather than a bug or an enhancement. Does that mean this is the end of the line for headless applications of the K4A? I cannot tell, but it seems this problem has been worked on since June 2019 (perhaps earlier) and still we do not have a truly headless solution.

@Krasin's systemd method is close but still requires that, somehow a user must log in via a graphics console so that there is an X session in place (with all of the requisite session setup and environment setup). The latter part (i.e. the session and Xorg setup) is the remaining unsolved issue for headless systems.

Am I just missing something obvious or should we accept the (apparent) fact that the K4A camera cannot be used on a headless system?

[cognitivesystems]

Many thanks to @krasin for contributing a systemd unit that can be used to configure and run a program that grabs depth frames from the K4A. I need to do this for my robotics application. Unfortunately, I have not been able to find a way to extend @krasin's approach to operate on a fully-headless system (which my system is). The key issue is that the userid that is referenced in the systemd unit must be logged in and the login needs to be via a graphics desktop (i.e. something with an actual screen). I have tried using Xdummy, xserver-xorg-video-dummy, and xpra and many, many variations on similar themes. I've managed to get "close" to a solution but in every case the requirement remains for a real physically-attached graphics screen.

I'm troubled that issue #562 got closed. There remains #796, but this is marked as "Documentation" rather than a bug or an enhancement. Does that mean this is the end of the line for headless applications of the K4A? I cannot tell, but it seems this problem has been worked on since June 2019 (perhaps earlier) and still we do not have a truly headless solution.

@krasin's systemd method is close but still requires that, somehow a user must log in via a graphics console so that there is an X session in place (with all of the requisite session setup and environment setup). The latter part (i.e. the session and Xorg setup) is the remaining unsolved issue for headless systems.

Am I just missing something obvious or should we accept the (apparent) fact that the K4A camera cannot be used on a headless system?

@StevenButner @krasin any luck in getting a full headless environment running. I have had some luck but not completely headless. Here is what I did

Initially I wanted my K4A nodes (mainly ros nodes but inside a docker container) to run completely on a Ubuntu 18.04 server. I have my Kinect V2 nodes running there where I setup the entire Ubuntu System with a one press Ansible playbook. Works like a charm. Its deployed on a intel nuc
Now the issue I faces using Ubuntu Server was the same as us. The Depth engine failed
I tried to do what Krasin suggested. I did as below
firstly I switched to a ubuntu 18.04 desktop version on a intel nuc
-- setup a systemd service script to start the docker container which starts the cameras
-- in the above script I have a wait using "After" for the docker service to start and also the X Sever to starts
-- I have enable aut user logon
-- On boot after the gui shows up I see that the kinect truns on Iand I see all data including point cloud being published
-- now I remove the display cable and reboot the computer. Everything still works (without diaply physically connected)

Any comments?