Px4-autopilot: Must reboot Pixhawk Between Missions

[ryanjAA]

After we land and finish a mission, if we disarm then arm back up or simply don't disarm but re-run the mission, mission mode engages but not take-off occurs (no throttle spin up). Flaps go down (take-off flaps are enabled) but there is no motor-spin up. The only way to fix it is to reboot the plane from QGC.

This is occurring on master but likely not new since it has done this repeatedly (nothing but this) on master from a week or two ago as well.

FMU_v2 with v3 firmware on fixed-wing.

Logs:
Successful flight:
https://review.px4.io/plot_app?log=ba1a4634-1c12-4461-9938-097c743a3e6c
Log right after taxing back:
https://review.px4.io/plot_app?log=3cca3276-7b5a-4a50-8073-9a4fd1947557
Quick Disarm then arm and then taxing to try and takeoff (which didnt work:
https://review.px4.io/plot_app?log=575072f3-09a1-4c50-bbe9-3edcd923b9b6
After the last log, a reboot of the plane from QGC and the next mission (same exact mission) flew fine.

[ryanjAA]

Same today during 20 flights...

[Antiheavy]

@ryanjAA are you guys re-uploading the mission between flights? I wonder if that would also fix the issue, instead of needing to reboot. I'm suggesting this is a solution, but I'm wondering if it has to do with the active mission item not resetting to the beginning after landing. I notice this sort of thing in SITL a lot.

[Antiheavy]

I wonder if an improved state machine can address the issue reported by @ryanjAA here, and also maybe this related topic: https://github.com/PX4/Firmware/issues/10657

[ryanjAA]

No, not re-uploading the missions. Generally re-running them over and over and sometimes with just parameter changes. Re-uploading might work as a work-around it but still not exactly a solution since the vehicle is either (both have been done to no avail), taxied back to the opposite side of the runway after being disarmed first (to stop the log), then rearmed (which it will and the prop will spin) or kept armed and taxied to the opposite side before a disarm then re-arm. Either scenario it just wont throttle up when switched to mission mode, stabilized or other modes work fine in both cases. Interestingly, even the takeoff flaps engage, just no throttle (just checked from a recent log), the mode is switched to Mission, no error on entering mission mode, the flaps go down but nothing else. No movement.

[julianoes]

I tried this in SITL and could not reproduce the issue yet. I can fly the same mission multiple times. I'll keep digging.

[julianoes]

How is the taxing done? Is that using manual override? And is a (first) mission possible after taxing?

[julianoes]

I have one hypothesis:
If for some reason this runway takeoff reset is not called, throttle will stay at 0 as you observe it.
https://github.com/PX4/Firmware/blob/75f1ad36d1210560f00da8f4fa3bddbb3da9da16/src/modules/fw_pos_control_l1/FixedwingPositionControl.cpp#L1878

Therefore, I wonder if we should add a reset for the runway takeoff here as well:
https://github.com/PX4/Firmware/blob/75f1ad36d1210560f00da8f4fa3bddbb3da9da16/src/modules/fw_pos_control_l1/FixedwingPositionControl.cpp#L1271-L1275

The comment even says that we reset both runway takeoff and launch detector, however, the comment is a lie and we do not actually reset the runway takeoff.

[ryanjAA]

Interesting about it now actually resetting the runway takeoff detector.

Yes, a first mission is possible after manual taxing.

Issue is when the plane lands from mission and switched to stabilized mode. Then it is either disarmed right there, armed again (to stop and then start the log) and then taxied back and then try to start the same mission again. The other way it occurs is when the plane lands, is switched to stabilized and taxied back and then disarmed and then re armed. It is taxied always in stab mode so done manually with rc. It will engage mission mode in either those two scenarios (no error), just not throttle up. A good test was enabling the takeoff flaps as they would go down but the throttle would not start. A quick soft reboot from qgc and then it will go.

For good measure and to throw any wildcard issues, we changed physical autopilots and then planes completely, same problem so it’s not hardware.

[julianoes]

@ryanjAA could you test with the change I suggest? Or should I create a branch/pul request to test?

[ryanjAA]

Test team is already on site. Let me see if we can get it compiled and out to them.

[julianoes]

This would be my suggestion: https://github.com/PX4/Firmware/pull/11738
Obviously test with caution!

[ryanjAA]

Thanks. I couldn’t get it tested since only the composite test planes were out flying due to today’s high winds (20mph sustained). Our safety testing protocol requires us to have our foam plane mockups flown prior to the larger craft for new changes but the foam planes can barely handle that type of wind so they were grounded for the day. I’ll get someone on this ASAP to confirm though.

[julianoes]

@ryanjAA Is there a chance you could test that this week?

[ryanjAA]

I will test asap. Still too much wind for foam (and a little rain) here but I’ll get something out this week for sure.

[LorenzMeier]

This has been fixed. Thanks!

[1Gump]

glad to see this fixed. this happens every time to me same as @ryanjAA. I am using stable 1.8.2 w/ pxhack v5. I can manual takeoff and resume switch to mission in air, but auto t/o requires reboot.
again, glad to see this fixed.