Depthai: [BUG] Encoder Problem With Latest Raspberry Pi OS

[Luxonis-Brandon]

So we discovered in internal testing, thanks to Chris, that after upgrading our Raspberry Pi's to the latest Raspberry Pi OS (with dist-upgrade, exact version below) that the DepthAI video encoder is not stable, so this example (04 - RGB Encoding) fails quickly. Prior to this upgrade to Raspberry Pi OS (the latest as of this writing), this instability did not happen. And we do not see the instability on other platforms (to our knowledge).

But on the latest Raspberry Pi OS, every Gen2 DepthAI version we tested has this instability.

The versions tested are below:

1.
depthai==2.0.0.1+4450dc3a7246f6d80f143c57db3f7f797f677882
segmentation fault after ~2 minutes
2.
depthai==2.0.0.1+31e14bdf1d2de39e85ab2d4806ad63f28970baf2
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'h265' (X_LINK_ERROR)'
3.
depthai==0.0.2.1+c9a19df719cb668e438d6eafd193cdf60a0d9354
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message '[json.exception.parse_error.110] parse error at byte 1: syntax error while parsing MessagePack value: unexpected end of input'
4.
depthai==0.0.2.1+1adf28f17fc3c5409f8226f309eb7122a7eb24bb
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'
5.
depthai==0.0.2.1+e9930fcf2bf2f6d81b7c5ba8455e9d2f6a8d8685
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'
6.
depthai==0.0.2.1+9ad44703a6d25d42d1418cc8660f19a8ed8d5346
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'

And we have found that Gen1 does not suffer from this issue on Raspberry Pi OS. We are continuing to debug.

Raspberry Pi OS version where we first noticed the problem:

OS version:
VERSION="10 (buster)"
kernel version:
Linux raspberrypi 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l GNU/Linux

[jasonopto]

What's the latest os verion that worked with Depthai? It'll be great to have a link to it for download/install.

[Luxonis-Brandon]

Great question @jasonopto . So I didn't write down what was on my Pi that _was_ working, unfortunately. So we'll need to test. But I do know that we tested both 5.4.83 and 5.10.17 and they both have the problem (see here for a great version chart). I think I was most likely running 5.4.51 (2020-08-20), but I'm checking. We have logs as to what was flashed. Will circle back.

[aviogit]

I don't know if it can be a useful information @jasonopto, but 04_rgb_encoding.py works perfectly on my RPi4. I'm using the official Raspbian 5.10.9-v7l+ kernel (but after that everything else is Ubuntu 20.04 x86).

[jasonopto]

@aviogit Thanks for the info. @Luxonis-Brandon's crew shared a prebuilt wheel. It works well now. You can see details on depthai discuss, search for pi zero. I tested multiple examples in depthai, all working. I also realized how slow Pi Zero is in handling videos.

[Luxonis-Brandon]

So we are learning more as we dig in. We have 2 engineers trying to figure out the crash behavior here. One is full-time on this and the other wrapping up https://github.com/luxonis/depthai-core/issues/94 while testing.

What we've found so far is that it doesn't crash with non-blocking queue IIRC, which was the default behaviour in gen1 (if we remember correctly).
We were excited that we thought we found an is that was culprit, but it was from the previous PR. (So that was found/fixed at least.)

As @aviogit mentioned, it definitely something related to the fact that the packets are not consumed in time by the host, on RPI4 is working fine, while RPI3 queues get full.

And yes, @jasonopto 's case it was because we do not yet have Pi Zero wheels prebuilt (which is largely because the thing is so slow/hard to work with, ha, like @jasonopto mentioned - we'd need to figure out some reliable cross or cloud-compile for armv6/PiZero).

[uchihatashi]

I am also facing the same error on Raspberry Pi OS on gen2 API.
Please let me know how to solve this error:
'Error while reading data from xlink channel: (my_nn) (X_LINK_ERROR)'
and I am using Linux raspberry pi 5.10.17

I need some suggestions from you guys:

1. So should I downgrade from pi 5.10.17, and will it solve the error?
2. Or should I wait for the new update to come, and how long will it take?
3. Also, sometimes it crashes (throwing errors), and sometimes it's not crashing at all.
4. One of the errors I m getting is the device not found even when the device is connected properly.
5. with dai.Device(pipeline) as device:
6. RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
   ---- [After running the code, it freeze my Raspberry Pi and gives this error, and also I m getting this error when I run my program first time, when I start my Raspberry Pi, or when I removed connect my OAC-D to Pi]

[Luxonis-Brandon]

Thanks for the report and sorry about the issue here @uchihatashi .

So actually a 3rd option could be if you can accomplish what you need from Gen1 API in the meantime. If you are mainly trying to do say encoding in parallel with single-stage object detection (in 3D space or not) and depth, you can actually probably use our Gen1 API, which is currently seeming rock-stable on Pi.

Could you share a bit more on what you're trying to accomplish?

Thanks, and sorry about the bug/trouble here. We're still hunting this one.

[uchihatashi]

Hello sir, @Luxonis-Brandon
Thank you for the reply. We are trying to run Human Pose Estimation on multiple threads, and due to this, we can't use Gen1. Is it possible to run it if we downgrade the Pi version from 5.10.17-v7+?

[Luxonis-Brandon]

Hi @uchihatashi ,

Sorry about the trouble here. So I thought we had had a breakthrough this morning, but alas it was something different. Downgrading might work. For example I didn't see problems on my older Pi install, but it's a bit random it seems. As some aren't seeing the issue with the latest Pi OS install. So it may be in vane. But it could bear fruit, as I think in my case 5.4.51 was running without a problem the multiple times I tested it (but the possibility exists that I was just getting lucky).

We're still heads down trying to figure out this crash.

Sorry again about the trouble here.

Best,
Brandon

[dhruvsheth-ai]

To add here, 5.10.17-v7l+ version of Raspbian OS works well for me on Raspberry Pi 4b - 8gb
This seems a bit weird contradicting what others are facing. Sometimes I've faced Xlink erros but they are mainly due to abrupt change from Gen1 to Gen2 or using USB2 instead of 3.

But, I've seen segmentation faults often on 2.0.0.1. The 2.2.1.0 depthai lib does not seems to cause trouble.

Also, if there's any testing I can do to check errors, do tell me :)

[uchihatashi]

Ok sir, @Luxonis-Brandon
I will try to downgrade it and check whether it's running properly or not. I am also thinking of using ubuntu-OS on Pi because I m not getting an error when I run the same code on my laptop (ubuntu v18).
@dhruvsheth-ai sure!!!

[diegomarvid]

Im using Linux raspberrypi 5.10.17-v7l+ on Raspberry Pi 4 8GB and having this problem when I run Gen2 Object Tracking example:

File "test.py", line 92, in <module> with dai.Device(pipeline) as device: RuntimeError: Failed to find device (ma2480), error message: X_LINK_DEVICE_NOT_FOUND
And when I run depthai_demo.py this error:

`
WARNING: Usb rules not found

Set rules:
echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules
sudo udevadm control --reload-rules && udevadm trigger
Disconnect/connect usb cable on host! `

I have Python 3.7.3.

It works well on my Windows 10 PC.

[Luxonis-Brandon]

Hi @diegomarvid ,

Sorry about the delay. I think this is a separate issue actually. After you set the USB rules on the pi, did you unplug the USB cable and plug it back in?

This step (the udev rule and unplug/replug) is not needed on Windows. That's why I ask.

Thoughts?

Thanks,
Brandon

[diegomarvid]

Ok! It worked thanks! Didnt know what usb rules was. Thanks

[Luxonis-Brandon]

We think we figured this one out. We're testing now and it's looking good. If anyone else is interested in testing it, see below:

https://github.com/luxonis/depthai-python/actions/runs/850855986
If someone has free rpi3/4 etc. when CI finishes (or build from sources), can you set up for test?

git fetch --all
git checkout origin/rpi_crash_mitigation
cd examples
./install_requirements.py
./04_rgb_encoding.py

So far running for more than 1 hour on rpi3 (although cpp, but shouldn't matter).
It will write a lot to the SD card, so optionally redirect output to /dev/null.

[chris-piekarski]

USB transfer is definitely more stable @Luxonis-Brandon; no XLINK crashes yet
FYI the video is unwatchable on the rpi3 using VLC; had to transfer to Windows PC for playback to work
There is a ffmpeg mux "fix your code to set timestamps properly" message but unclear if related to rpi3 playback failure
Also, ./install_requirements.py might be missing 'sudo apt-get install libatlas-base-dev' package

[Luxonis-Brandon]

Sweet, thanks for testing @chris-piekarski ! And thanks for originally finding this bug and reporting as quickly/early as you did.

I'm not sure on the timestamps thing... that's been there for a while. And yes, I think the Pi often just isn't performant enough to decode/play on the Pi itself. Not sure if this is a result of something we can improve on our encoding settings, or if it's just that the Pi can't decode this bitrate of video real-time and play it. Unsure.

Also, ./install_requirements.py might be missing 'sudo apt-get install libatlas-base-dev' package

On this - thanks for reporting. Will ask internally.

[aviogit]

VLC fails to play .h264/.h265 videos but mplayer (or MX Player on Android) can reproduce them. However, for every frame, it spams:

No pts value from demuxer to use for frame!
pts after filters MISSING

Also, it would be a real plus on the Pi to have video playable by Chrome/Firefox because this would allow serving videos via any web server, the ideal configuration for watching captured videos on the field.

[VanDavv]

@chris-piekarski I think we already have libatlas-base-dev in the arm dependencies here. I'm wondering what might go wrong here, that it didn't end up installed on your end. Which RPi and OS are you using?

[chris-piekarski]

Hi @VanDavv - so I used a fresh copy of rpi OS, cloned luxonis/depthai-python, executed install_req script, cloned luxonis/depthai, ran install_req script, executed demo script and got failure from import cv2. Installed missing package and demo script executed successfully. More than possible I incorrectly assumed the two requirement scripts would cover this package. No big deal, was just an FYI comment.

[Luxonis-Brandon]

Thanks for the report.

[szabi-luxonis]

Fixed in https://github.com/luxonis/depthai-python/releases/tag/v2.4.0.0