Px4-autopilot: Throttle drops to 0 sometimes for a second during position hold (MC)

[ArminStrobel]

I put the drone in position hold about 10m above ground. Without me doing anything it suddenly dropped quiet fast, lucky it recovered before it hit the ground (in most cases). At first I was thinking some ESC problem but I had the same issue with a different MC. Also does "actuator_controls_0_0.control[3]" and "vehicle_attitude_setpoint_0.thrust_body[2]" show that the command was for throttle was set to 0 for a very short period.
That is with firmware version 1.9.2 and on pixhawk as well as on pixhawk4 hardware the same.

[dusan19]

instead of landed check for ground contact.
although you should probably upload the log

[ArminStrobel]

You are right it is a false ground contact indication in mid air. It seems it is related to #11683 .

[julianoes]

@ArminStrobel is this still present with 1.10-beta or base after this fix #11172?

[ArminStrobel]

I only can confirm that for the latest stable release:
Firmware Version: 1069058 (1.9.2 release)
I damaged one of my vehicles badly due to this. I am a bit hesitant to test beta versions.

I assume not everyone has that problem, could it be related that I might don't have default values for the LNDMC parameters?

Regarding #11172 I am not sure if I understand the intend here proper, but what I want to point out and it might be considered that in position hold you want to have vertical and horizontal speed to be zero. Using that alone for ground contact detection will not work since a good tuned position hold controller will have vertical and horizontal speed to zero (within noise). So there is no way you can reliably detect and difference between hovering and having ground contact.

Also I would consider to give the option to choose if the motor gets switched off on ground contact, since any unintentional contact will switch it off. My suggestion to make that an option and have it by default deactivated.

Again, I am not understanding the discussion in #11172 completely, so please take my comments with a grain of salt.

I think the only way to detect reliable (except with a ground contact sensor) is to monitor the thrust when sinking is commanded and see if it falls below a certain limit and the vertical velocity is close to zero as well as the acceleration needs to have a mean of 1G.

[ArminStrobel]

Actually I was reading my own comment. I never commanded a landing (not by stick, nor by telemetry). Shouldn't be shutting down the motor just be allowed on ground contact when a landing is commanded? #11172

[julianoes]

Sorry about damaging the vehicle, and I understand your hesitation.

Using that alone for ground contact detection will not work since a good tuned position hold controller will have vertical and horizontal speed to zero (within noise). So there is no way you can reliably detect and difference between hovering and having ground contact.

Right, that makes sense. @MaEtUgR or @bresch do you have thoughts on this

[dusan19]

well there is also an "in descend" condition where the vertical velocity setpoint has to be such that the vehicle has the landing intention (aimed donwards with at least 0.9 of landing speed). So hovering doesn't fall under this case. The problematic part could be the switch from hovering to fast descending, but the hysteresis logic deals with this.
As a suggestion, the "in_descend" check might be made more reliable in missions such that instead of z velocity we check for the SETPOINT_TYPE_LAND

[ArminStrobel]

@dusan19
There should be no way that the motors can be set to 0 during any kind of maneuver, without the intend of the operator to land. So any decision to switch off the motor has to include the intend of the operator / mission. Only if mode is set to landing the motors will be allowed to be switched off.

You need to check for following all together:

• Thrust below a certain level (output of the position controller)
• Stick all the way down or SETPOINT_TYPE_LAND == true
• z velocity zero
• z acceleration close to 1G (to cover the case where you in mid air and you about to to decent fast but not yet have built up speed to indicated that)

@julianoes Thank you for understanding. Is there a way I can deactivate this function with a parameter? Because i don't have any use of that function and only poses a risk. I do not want that the vehicle has the authority on its own to shutdown the motor anytime.

[dusan19]

Thats what im trying to say, landing intention logic is already a condition for ground contact, so if you were just hovering this should have never happened since you would normaly not have low thrust or vertical velocity setpoint close to the landing one.

It would be best if you could share the log file.

I agree that in general the switch to descending could be more robust to false positives

[ArminStrobel]

@dusan19 I think you we should clarify how landing intention is defined. With landing intentions I mean it is in landing mode either by a switch from the user or part of a mission to land. I don't think that you understand what I am trying to say. let me try it again.

Landing intention is not to have low thrust. Low thrust is a state not an intention.

Sorry I am getting frustrated with this. I would like to know how I can deactivate this function.

Like many other ones I am relying that this software is not killing my vehicle.

[mhkabir]

It would be great if we could look at all the data available in the log to come up with an accurate diagnosis. Would you be able to share it?

[ArminStrobel]

Sorry, I prefer not to share it because it contains a lot of information I prefer not to share.
But I can put any kind of plot up you want, if you can work with that?

I set LNDMC_LOW_T_THR to 0 now, I assume that should make sure it will not happen again?

[BazookaJoe1900]

I am agree with @ArminStrobel intention is a flight mode, not conditions that might lead to low trust and/or velocity and/or acceleration.
I am not sure why but also in my craft thous condition met even when I just hovered, that might be related to vibrations, or the fact that I set minimal thr to ~25%.
On my custom firmware I did changed the logic so ground contact can only be on landing mode.
I didn't merge it to master because I still waited to responses about #11683, that the trust to 0 should be only be on the LANDED state, (there are FLYING->GROUND_CONTACT->MAYBE_LANDED->LANDED states)

[ArminStrobel]

It happened again. This morning I made a short flight test and again a vehicle badly damaged. I did set LNDMC_LOW_T_THR to 0. It was really nice hovering before the crash.

@BazookaJoe1900 thank you understanding what I try to say. The motors should be only deactivated in the LANDED state and nowhere else(especially not before Maybe landed). If I look at the log data the LANDED state is absolutely accurate. So why not using this state to deactivate the motors as it was probably intended by the people who created that state.

I am quiet disappointed and lost trust into the PX4 software stack, since that is supposed to be a stable release.

@julianoes can you tell me which release does not have this "function / issue"? I assume this issue and #11683 are the same and both are related to #11172. Can somebody look at #11172, for me it looks like very bad idea?

I can share the log file from this morning.
https://review.px4.io/plot_app?log=6e0ede8b-882b-4146-9fd7-fbc162bb5724

[dusan19]

here are some log analysis:

For false ground contact detection at t=~97s:

• the drone is hovering in mission mode, and having a steady state error in z velocity controler (around 0.3m/s, might be the tuning), so the vz setpoint is around 0.4 and slightly increasing.
• Landing speed is set through a parameter to 0.5m/s, even though i see that in the parameter definition file the min value is noted to be 0.6.
  Note: In position and altitude controled modes the intention to land has to be determined from the z velocity setpoint (up-down stick input) and is set to true when the user controls the drone to go down with velocity close to the landing one. Unfortunately the same logic is used for auto modes, which can be improved as suggested in one of my first comments here.
• In this case the threshold for landing intention is 0.45m/s due to the land speed parameter setting. Velocity setpoint increases and when it reached this threshold, landing intention was set to true.
• due to the steady state error, the z velocity was pretty low (hovering), so the "no vertical movement" was fulfilled as well
• there was no horizontal movement

landing intention + no vertical and horizontal movement = ground contact

For the false ground contact detection at t=~102:

• The operator was flying in altitude mode. The throttle stick was all the way down so vz setpoint was 0.5m/s which was interpreted as landing intention.
• The drone was still trying to recover from the first false ground contact and even though the thrust was at max it was still losing altitude. When it started gaining altitude it switched to altitude mode with the throttle stick all the way down. It continued gaining altitude due to a slow vz response and overshot the 0.5m/s setpoint all the way to 0 when it detected no movement and ground contact.

Suggestions (if possible with this drone):

• tune the land speed parameter, higher value would increase the landing intention threshold but also the vertical movement one, so carefully.
• tune the z position-velocity controller to get a decently quick response and reduce the steady state error.
  Note: the drone has hover thrust at ~75% which probably makes it difficult to get fast control of vertical position and velocity.

Hope it helps a bit, although I assume you are hesitant to test more if it keeps destroying your drones :(

[ArminStrobel]

@dusan19 Can you explain exactly what values and parameters are used for the so called "landing intention detection"? Wouldn't be simply checking if the stick is all the way down much more reliable?

Btw at t=102 drone already crashed and was on its side and I pulled throttle all the way down, but had to switch to manual mode to disarm the motor.

[LorenzMeier]

@ArminStrobel The reason you're experiencing this corner case is the combination that you've set the land speed parameter below the absolute allowed minimum (it is set to 0.5 for your vehicle, the lowest hard limit is 0.6, the default 0.7) QGC will have warned you that this is potentially unsafe in the UI - please let us know if you think it didn't and your Z-controller lacks enough integral gain. It needed the combination of both for it to happen.

And if you hadn't have a high hover throttle (overweight airframe) it wouldn't have dropped that fast. So it really needed all three factors to lead to a crash. The high hover throttle also reduces your attitude stability margins, making it harder to recover.

That is why it is working for most other users. However, we want the system to be easy to configure and more flexible, so we're currently discussing with the dev team how we can re-architect this.

The goals for this are:

1) Option to hard-disable any auto-throttle handling behavior (for setups like yours)
2) Forcing safe parameter values also on the PX4 side, not just in QGC (so if you set the landing speed too low it will either increase the landing speed or disable auto-disarm)
3) Auto-estimating hover throttle, which should allow this to work better and without you needing to configure anything.

[steffensol]

FYI; We just saw this same behaviour on a large 2m tip-to-tip octorotor, in a high wind condition. With large props and loads of intertia on the prop/motor combo, this is extremely problematic behaviour. If someone contacts me i can send you the logs on PM.

[jzazbert]

In our experience this is caused by MPC_HOVER_THR being set incorrectly. https://dev.px4.io/v1.9.0/en/advanced/parameter_reference.html#MPC_THR_HOVER

The land detector thresholds are driven by that parameter, and if the aircraft is hovering at a significantly different throttle than the the parameter value, we see those hops caused by the land detector switching in and out of GROUND_CONTACT state. In our case, we often have hover throttle set for a payload weight and then fly it light. The behavior is after slowing down from a climb typically will trip it in the deceleration. Setting hover throttle correctly makes it go away in all our cases.

[ThomasRigi]

Today we also had issues with the land detector as you can see in this log : https://logs.px4.io/plot_app?log=4eea26ba-02eb-4b37-9de2-3e576693227b (5:58-6:09) (don't bother with all the other weird stuff going on :P)

The ground contact detection really isn't robust enough. Having a drone with large inertia (in this case a VTOL where wings can generate extra lift) on Z velocity and a fast change in the Z velocity setpoint when previously hovering is enough to trigger a ground contact. The thrust was even above MPC_THR_HOVER ! (which I set too low on purpose) If it is that easy to get around the low thrust criterion, why even put it there in the first place? I'd prefer to have a low thrust criterion as an individual AND criterion.

In high winds we've consistently had the same problems with our larger VTOL drone (which already in calm conditions is rather slow to react with the large wing surface). I would appreciate if we could set the hysteresis time for ground contact as a parameter to take into account the dynamics of the drone. A larger hysteresis time on ground contact could have spared us a few ugly moments. I don't see any danger, only an inconvenience, in late ground contact detection, but a big danger on triggering it too early, as this issue proves well enough.

(If someone has inputs on how to improve the z velocity controller, I gladly take them :) )

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[troy-at-skyfront]

@julianoes julianoes

@ArminStrobel is this still present with 1.10-beta or base after this fix #11172?

Hello, I am using v1.10.1 stable release and I had a very similar problem where the drone would detect landing while flying (about 8 meters above ground) and then decrease thrust to zero. We were in RTL mode set to land after 60 seconds of loitering when this happened. There were no winds. We had a total loss of the vehicle--our vehicle is large and expensive and it was quite heartbreaking to see this happen. In answer to your question, the problem still exists.

@LorenzMeier

Our landing speed is 0.6 m/s and we still encountered the problem. And our MPC_HOVER_THR parameter was set such that center stick induced a hover. I do not think that the proposed solution of setting the correct landing speed solves the problem.

[ArminStrobel]

@julianoes I updated to v1.10.1 and adjusted my vertical control parameters appropriately (I had I-part originally at 0). Since then I never had an issue anymore. I wold say those are about 50 flights.
If you look at your data does vehicle_local_position_0.z fit well with vehicle_local_position_setpoint_0.z in general?
I am not sure what the current state of the logic is behind the ground contact detector (specifically the landing intention determination), but in the original implementation (if I understood it correctly) it was in my opinion the wrong approach. Maybe @BazookaJoe1900 can help.

[troy-at-skyfront]

Yes, the vehicle local position Z follows the setpoint well, though Z-velocity could use some improvement. Please see the graphs of the last moments of the flight before the landing detector took our bird's life.

Correct me if I'm wrong, but if there are high wind or turbulent conditions that could cause a divergence in Z or Z-velocity, then this landing detector could get triggered again as well. It seems like a time bomb.

We had about 7 hours of flight on v1.10.1 across approximately 20 flights before this crash occurred. We've been on PX4 since 2016 and haven't seen anything like it. There is a marked difference between this behavior and the landing behavior of v1.8.2, the last version of the firmware we were using.

[dagar]

@julianoes julianoes

@ArminStrobel is this still present with 1.10-beta or base after this fix #11172?

Hello, I am using v1.10.1 stable release and I had a very similar problem where the drone would detect landing while flying (about 8 meters above ground) and then decrease thrust to zero. We were in RTL mode set to land after 60 seconds of loitering when this happened. There were no winds. We had a total loss of the vehicle--our vehicle is large and expensive and it was quite heartbreaking to see this happen. In answer to your question, the problem still exists.

@LorenzMeier

Our landing speed is 0.6 m/s and we still encountered the problem. And our MPC_HOVER_THR parameter was set such that center stick induced a hover. I do not think that the proposed solution of setting the correct landing speed solves the problem.

Can you share the log?

[troy-at-skyfront]

@dagar Yessir, please see this link below to download the ULG. The false landing happens at the end, ha ha.

https://skyfront.wetransfer.com/downloads/0750524b35a8388f19c3f0e1ceb8ad3120200610130545/0e31ed

What's interesting is that the landing slowed down at 8 m AGL, and then it fell out of the sky. Take a look at the distance sensor. Coincidentally?, 8 m is the maximum height for using the range finder (EKF2_RNGA_HMAX). Perhaps the range finder has something to do with this.

Please also take a look at the RTL at the end. The drone did not ascend to 30 meters first as programmed. Instead, it headed straight back to the launch point and ascended to 30 m along the way. In the old firmware (v1.8.2), in RTL the drone would ascend first and then headed back to the launch point. This new behavior is dangerous in my opinion and may result in a collision.

Thank you so much for your help.

[dagar]

@dagar Yessir, please see this link below to download the ULG. The false landing happens at the end, ha ha.

https://skyfront.wetransfer.com/downloads/0750524b35a8388f19c3f0e1ceb8ad3120200610130545/0e31ed

What's interesting is that the landing slowed down at 8 m AGL, and then it fell out of the sky. Take a look at the distance sensor. Coincidentally?, 8 m is the maximum height for using the range finder (EKF2_RNGA_HMAX). Perhaps the range finder has something to do with this.

Please also take a look at the RTL at the end. The drone did not ascend to 30 meters first as programmed. Instead, it headed straight back to the launch point and ascended to 30 m along the way. In the old firmware (v1.8.2), in RTL the drone would ascend first and then headed back to the launch point. This new behavior is dangerous in my opinion and may result in a collision.

Thank you so much for your help.

@troy-at-skyfront exploring potential fixes in https://github.com/PX4/Firmware/pull/15083.

[bresch]

And our MPC_HOVER_THR parameter was set such that center stick induced a hover

From your log, it seems that this isn't exact; your hover thrust is ~37% while MPC_THR_HOVER is set to 50%. We're now running an estimator to automatically adjust this value, but you apparently don't have it in your version of the code (edit: it will be in v1.11 https://github.com/PX4/Firmware/issues/14402)

[dagar]

MC land detector ground_contact detection fixes have been merged in https://github.com/PX4/Firmware/pull/15083.