Px4-autopilot: Commander mode offboard triggers PX4 freeze

[prothen]

Describe the bug
Using px4_sitl_default together with gazebo and sending set_actuator_control_target via pymavlink successfully establishes a mavlink connection and actuator commands are received on the px4 stack and uorb topic offboard_control_mode is published. When using the system console and sending commander mode offboard the stack freezes. No uorb topics are published and no mavlink messages are received (gazebo plugin correspondingly outputs [Err] [gazebo_mavlink_interface.cpp:1155] poll timeout in the mavlink poll function). The pxh system console remains responsive but commands to any module (e.g. commander, uorb) have no effect and hence indicating the system wide freeze. No error message from PX4 is shown and gdb debugger catches no signals.

To Reproduce

• run px4_sitl_default gazebo
• send mavlink messages via pymavlink c.mav.set_actuator_control_target_send(0, 0, 0, 0, controls)
• switch to offboard mode using commander mode offboard

Expected behavior
Px4 was expected to switch vehicle_control_mode into offboard mode.

Log Files and Screenshots

• https://review.px4.io/plot_app?log=86db944d-af82-456a-bb6c-79e8e5b223dd

Additional context

• PX4 firmware (v1.9.2)
• mavlink (release/kinetic/mavlink/2019.10.10-1)
• mavros (0.33.0)
• Approaches so far:
  
  
  
  • PX4_INFO() print outputs showed that commander reaches publishing in run() loop
  
  
  • gdb info variables and inspecting last variables gave no further insights
  
  

The freeze presumably happens after the system console arguments are parsed and published in the commander run loop. It could also be that the crash happens earlier and all modules get stuck at the file descriptor polling. Maybe a problem from global variables causing a race condition with the main commander thread? Maybe I am missing something but it does not seem that variable access is coordinated when parsing the mode argument.

Any advice on how to approach or debug this problem are very much appreciated!

[julianoes]

Thanks for the issue. I think this is "just" a simulation issue. Publishing actuator_controls directly messes with the lockstep implementation.

I suggest you disable lockstep, rebuild and try again.

[prothen]

Problem resolved with disabling lockstep as recommended. Thanks for your fast help!

[prothen]

Not using lockstep now triggers the following error messages after some time before establishing any additional mavlink communication:

ERROR [sensors] Accel #0 fail:  TIMEOUT!
ERROR [sensors] Sensor Accel #0 failed. Reconfiguring sensor priorities.
WARN  [sensors] Remaining sensors after failover event 0: Accel #0 priority: 1
ERROR [sensors] Gyro #0 fail:  TIMEOUT!
ERROR [sensors] Sensor Gyro #0 failed. Reconfiguring sensor priorities.
WARN  [sensors] Remaining sensors after failover event 0: Gyro #0 priority: 1
ERROR [sensors] Mag #0 fail:  TIMEOUT!
ERROR [sensors] Sensor Mag #0 failed. Reconfiguring sensor priorities.
WARN  [sensors] Remaining sensors after failover event 0: Mag #0 priority: 1

Are these critical or should I adapt something ?

[julianoes]

That sometimes happens in SITL if timeouts are detected. It doesn't look great but it shouldn't make a difference.

[prothen]

I see, thanks again !