Px4-autopilot: UAVCAN hardfault on "uavcan param list"

[JacobCrabill]

Describe the bug
After a few calls uavcan param <list/get/set>, I encounter a hardfault that leads to a reboot.

To Reproduce
Steps to reproduce the behavior:

1. Black Cube (px4_fmu-v3_default) running latest PX4 master branch
2. Power via USB
3. Attach a Here2 GPS to CAN1 (the "real" CAN1)
4. Connect to serial console via an FTDI adapter
5. Call uavcan status to ensure UAVCAN is running (this will produce a crap ton of node-spin and vehicle_air_data errors, these don't appear in v1.10...). It doesn't matter if you set UAVCAN_ENABLE to 0 and manually start it after boot, or set UAVCAN_ENABLE to 1.
6. Do a command like uavcan param list <nodeid> a few times
7. Cube will hardfault after anywhere from 2-5 calls from a fresh boot

Log Files and Screenshots

uavcan status error messages:

The hardfault after some get and set commands:

Reproducing the hardfault with 3 calls to uavcan param list 3:

[dagar]

This is also a bug.

[dagar]

This is probably just the command line uavcan param helpers missing locking.

[dagar]

Working on a fix in https://github.com/PX4/Firmware/pull/14741.

[JacobCrabill]

Forgot to update here - the possible fix #14741 does not seem to work. Still unclear what the root cause might be. The suggested solution makes sense (race condition on resource access), but the location of the error has not yet been found.

[dagar]

Thanks for the update, I'll try to reproduce locally with a debugger attached.

[JacobCrabill]

Cool, that would make things much easier for sure. It doesn't always throw a hardfault though; whatever memory corruption is occurring seems to happen in a few possible locations.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[amikhalev]

I experienced this too, I think this may have been a stack overflow, potentially fixed by #15864

[dagar]

Should be fixed by master, please re-open if that's not the case.