Px4-autopilot: pixhawk i2c deadlock in v1.8.2 with Pixhack v5

[dlwalter]

This is very similar to a previous issue: https://github.com/PX4/Firmware/issues/7951

I am running a Pixhack v5 with an external i2c compass(HMC5883)/gps combo. By repeatedly shorting out SDA to ground I can interrupt an i2c message and cause the entire system to freeze up and deadlock.

To Reproduce
Steps to reproduce the behavior:

1. Drone on with hmc5883 external compass running
2. quickly and repeatedly short SDA to ground
3. System will lock (no serial, rgb LED will freeze

Expected behavior
Not an i2c expert, but I imagine that i2c should be robust enough to handle an interruption on the SDA line mid-message. It seems this issue was fixed back in 1.6.*

Log Files and Screenshots
logic analyzer screenshot
logic analyzer detail

Additional context
Add any other context about the problem here.

[dagar]

https://github.com/PX4/Firmware/pull/7957

[LorenzMeier]

Thanks for reporting - we're looking into it. I expect it will be an OS-level issue we will make sure it's being taken care of.

[davids5]

@dlwalter - Thank you!

I have identified one issue with the i2c driver and will continue to debug this in the AM.

[bys1123]

Looks like I got this error on Pixhawk4 too. Not need to connect SDA to ground, system just random lock sometime on 1.8.2(rgb LED will freeze, speaker keep ringing). But 1.8.0 and master branch is ok.

[AlexKlimaj]

I think a driver is performing a read or write with the wrong amount data in the buffer. Either the master or slave is expecting more data and not getting it. Then PX4 is not handling the timeout and locking up.

[davids5]

@dlwalter Please test with https://github.com/PX4/Firmware/pull/11508

1.8.2 has a really old version of the F7 I2C driver and should NOT be used going forward. (The list of changes is https://github.com/PX4/Firmware/pull/11508)

[dlwalter]

I still managed to lock up the system by shorting the SDA line. Here is a screenshot:
(top - SCL, bottom - SDA, I am tapping the SDA line to ground, not sure why SCL went low))
https://imgur.com/cdtvWq0

detail:
https://imgur.com/DWp0GFM

I can send the Logic file, but it's too big to share here.

I did have a couple tests where I interrupted the SDA line mid-message and it worked fine. (SDA is the top line here)
https://i.imgur.com/SNrDELU

detail:
https://imgur.com/S2qjpm2

[davids5]

@dlwalter - please share the output of the 'ver all' command.

In looking at the pictures I do not understand the display: You have data without clock. But you state you are grounding the data.

Could we arrange as skype call to discuss this? My Skype name is:david_s5

[dlwalter]

nsh> ver all
HW arch: PX4_FMU_V5
HW type: V515
HW version: 0x00000001
HW revision: 0x00000005
FW git-hash: d74d6b0cdbbf8e2029e700472d4692217209211d
FW version: 1.9.0 80 (17367168)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 1f2f6d8cbe7c85f30fd75ec53c20566e91b14f21
Build datetime: Feb 20 2019 17:04:54
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID: 0002000000003635393731375102001c002e
MCU: STM32F76xxx, rev. Z

I updated the description above. Top two links (of the failure) SCL is the top line and SDA is the bottom.

The bottom two links SDA is the top line.

I also just had a scenario where the uart console locked up, as well as the RGB LED, but I kept getting i2c messages going.

Skype call or Slack would work for me.

[AuterionWrikeBot]

➤ David Sidrane commented:

Wating on OP to retest with master. There may have been some issues with wiering shorting the clock

[dlwalter]

I'll try again with the current master branch but I'll have to wire up a test harness. We moved back to the Pixhawk V2 board since the V5 was giving us these issues.

[davids5]

@dlwalter We found and fixed an issue on the F7 I2c. See https://github.com/PX4/Firmware/pull/11871 Please retest and reopen if the issue was not resolved .

[dlwalter]

That's great, David. Let me run this on the v5 and test it.

[dlwalter]

Ran it on my Pixhack v5 with an external QMC5883 and had no issues. Doesn't lock up anymore. Nice work!

[davids5]

Thank you @dlwalter!