Depthai: test.py hangs after a few frames

[itsderek23]

I'm now seeing similar issues as #32 with example_run_emotion_recognition.py. The script runs for a few frames, then hangs.

I'm fairly certain both of these scripts worked at the time of the MVP release.

[itsderek23]

Fixed by 98fb7753650fbc0326fdff31a10df758f98dab9e.

[itsderek23]

Re-opened due to c152ab74ab6e65344b0fb0adba87a528cabc76ad.

[asmorkalov]

@itsderek23 I experience the same issue with my device.
Commit: db28c93deeb1a607f83056c8e78e9a4a0f451d91
System information from log_system_information.sh:

Linux paradox 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
depthai.__version__ == 0.0.10a
depthai.__dev_version__ == 60a814b225bc4aa6709045f11c9306bf9c647222
numpy.__version__ == 1.16.4
Python 3.6.9

Log for test.py app is attached.
test.log

[Luxonis-Brandon]

Thanks @asmorkalov .

So this model is a 'multi-tensor output' model which we're going to put on hold for now as we stabilize more basic object detection and get training flow working.

Once that's in place we'll circle back and fix this. CC: @GergelySzabolcs

[asmorkalov]

@Luxonis-Brandon The issue is actually reproducible with all samples and tutorials I tried: emotion recognition, test app and face detection tutorial. I posted log for test as it it's a test.

[Luxonis-Brandon]

Thanks @asmorkalov - we'll dig into this.

So we've seen such crash before on specific hosts over USB3. Can you share which host computer you are using and which USB3 cable/length?

And do you have a shorter USB3 cable? On the host that had this problem, it did not happen on a 3 foot USB3 cable. And did not happen at all on USB2. It seems it's a signal integrity problem at the full USB3 speeds for 6 foot cables.

Either way we'll dig in in parallel to see if we can replicate and if get this stabilized. We'll focus on the test app (test.py).

Thoughts?

Thanks,
Brandon

[Luxonis-Brandon]

We had setup a device auto-reset that might be the problem here. This was for detecting if the user had force-killed the host code w/out giving the host code the opportunity to tell the Myriad X that it's no longer running.

This auto-reset code may be not running properly, and triggering false-alarms (false-resets).

We're investigating.

[asmorkalov]

@Luxonis-Brandon I tried the same procedure with another host with and all pipelines work for me. So, it's something related to USB subsystem on host side. I'll debug it a little bit and will come with details.

[Luxonis-Brandon]

Thanks @asmorkalov . So @alex-luxonis did an intense debugging session yesterday. We were able to find a computer in our lab machines that reproduces this reliably. It's actually in Colorado (local to @Luxonis-Brian) and Alex is in Romania so he's been debugging remotely.

According to our building/ordering logs this is the machine.
https://pcpartpicker.com/list/KTDFQZ

So @Luxonis-Brian - could you confirm that this is the machine (most importantly, that the motherboard is correct)?

Here are some observations he shared from our slack with this machine:

This does not look like a hardware issue, but more like a software bug.
A libusb bulk read transfer on host is hanging, and it has an unexpected requested size (none of our fixed frame/meta sizes, but it is a multiple of 1KB -- the low level USB block size).

There are no USB link state events when this happens, the connection is still fine both on host and device. And no errors in the logs.

The firmware on device continues to run fine, but can no longer send on USB because the host is stalled. Adding a timeout on the host and retrying the transfer doesn't help, can't recover.

It could be related to some timings, not sure. Also, there could be some underlying USB errors that contribute to a timing change, but the USB bulk transfer type ensures data integrity (there would be transparent retransmissions in case of errors).

It may be possible that the libusb version on Ubuntu 16, or the platform used (AMD with some Realtek chipsets) may have a role here, but not sure.

Will continue to look at this tomorrow.

So @alex-luxonis is attacking this bug directly, trying to debug it and implement the USB communication such that it doesn't happen.

And @GergelySzabolcs is working to implement a host watchdog which could automatically recover in the event that the host stalls like this.

Thanks,
Brandon

[Luxonis-Brandon]

Just an update that we're continuing to debug this. Will circle back as we learn more.

[Luxonis-Brandon]

@asmorkalov so some very interesting results. The only computer that exhibited this issue is this exact computer:
https://pcpartpicker.com/list/KTDFQZ

Can you share the specs of computer you are using?

Long story short:

• This -seems- to be something to do with hardware IOMMU and AMD CPUs, from what we understand. And it seems like Ubuntu 18.04 and Ubuntu 16.04 handle this differently, with perhaps 18.04 disabling the hardware IOMMU, which then prevents this issue.

We can't get it to happen on any other computer. And interestingly, this issue shows up in Ubuntu 16.04 on this computer, but does not show up at all in Ubuntu 18.04 on this same computer.

We had spent a week tweaking with settings in Ubuntu 16.04 prior to discovering this, and we were able to increase the mean time between this issue from once a minute or so to about once every couple hours or so.

And then upgrading this same computer to Ubuntu 18.04 from Ubuntu 16.04 resulted in no issues whatsoever under the same settings/tweaks which previously caused the computer to have issues within a minute (running multiple boards for over an hour together... and on Ubuntu 16.04 running multiple boards in parallel would make the issue occur within a handful of seconds).

So this -seems- to be something to do with hardware IOMMU and AMD CPUs, from what we understand. And it seems like Ubuntu 18.04 and Ubuntu 16.04 handle this differently, with perhaps 18.04 disabling the hardware IOMMU, which then prevents this issue.

Thoughts?

Thanks,
Brandon

[Luxonis-Brandon]

OK, to summarize and close this issue:

• We've found that some hosts have trouble with USB3 + long cables (2 meter). It seems to have something do do with the USB controller on the host side.
• Other hosts have no problem at all and run for days (tested well over 3 days on some), even with long cables (tested w/ a total length of a bit over 8 feet). For example, all Apple computers we've tested with have never exhibited the problem.
• Ubuntu 16.04 has an independent USB3 issue, seemingly only on new machines though. We think this has to do w/ Ubuntu 16.04 being EOLed prior or around when these new machines having hit the market. For example, on this computer (here) has rampant USB3 disconnect issues under Ubuntu 16.04 (with a 1 meter cable), but has none under Ubuntu 18.04 (with a 1 meter cable).

So unfortunately we discovered this after we shipped with long USB3 cables (2 meter cables) with DepthAI units.

So if you have see this problem with your host, switching to a shorter USB3 cable (say 1 meter) will very likely make the problem disappear.

And if you're the curious type, and want to help us crowd-source some more data, the following branch (which is a result of digging deep into the trenches of the USB PHY and MAC) does give us way better results on the host machine (named 'Ruth') we have that exhibits these long-cable issues.

If you don't mind doing trying this out:
git pull
git checkout https://github.com/luxonis/depthai-python-extras/tree/usb3_stability_phy

https://github.com/luxonis/depthai-python-extras/tree/usb3_stability_phy

On Ruth, with a combination of USB3 cables totaling ~8 feet, without these changes we see the issue about every 18 minutes. With these changes we did not see an issue in the 1 hour and 37 minutes we tested it.

What we don't know is if this just changed the MAC/PHY settings to accomodate this one machine better, or if it actually helps others. So if you have the problem and want to contribute - please do test the branch above!