Ardupilot: Copter: ensure hexacopter with failed motor can make safe landing

[t0nimas]

Hello,

I would like to know the APM response in fornt of an engine failure. As far as I knew, while flying an Hexa/Octo, the APM should be able to ensure the safety of the platform. However, last days, during one of my tests, I tried to fly (automatic) an hexa-rotor with one engine failure and the platform fell down (1000 feets) with any sign of APM response.

Then, once I arrived to this point, the theory that I was using an hexacopter to ensure the safety due tu redundancy is not working anymore.

This is the main reason why I am writing this post. Which is the best way to ensure safety infront of an engine failure in flight? APM code is designed to confront this kind of situations?

Which is better:

• Octocopter?
• Coaxial copters (quad,hexa,octo)...

[billbonney]

I've seen a APM hex fly with one motor out. But you do need enough power from the 5 motors to fly. It's the same with any redundancy setup. You need enough spare power from the other motors to be able to compensate the loss

[rmackay9]

A hexacopter also loses yaw control when it loses a motor. Because of the set-up of the motors it's actually not simply a quad with two extra motors.

Coaxial copters like the Y6 can fly with one motor failure. Octacopters (whether in a quad shape or spider shape) can also survive a motor failure.

We have plans to integrate some smart ESCs that will help us detect motor failures and perhaps they will be more resilient to sync failures but at some point physics takes over and there's little the software can do especially on the APM2 where the CPU is already overloaded.

I think there are some things we can do if we know a motor has failed on a hexacopter including giving up on yaw control. I've renamed this issue's subject line to make it more specific.

[rmackay9]

A blog post about a quad which can recover from a single motor failure. http://diydrones.com/profiles/blogs/onboard-quadrocopter-failsafe-flight-after-actuator-failure

[francisvierboom]

Are there two options for hex redundancy behaviour?

• give up on yaw (same as the quad), spin like a madman and maintain most of 83% (five 6ths) of original power - spinning continually compensates for pitch/roll motor imbalance. Possibly some loss would be in actually powering down the two minority motors to get yawing up to the required speed. Randy, as you say, the rotational algorithms to maintain roll and pitch control in this state must be a bit heinous, which is the special sauce in the (incredibly impressive) quad redundancy video from IDSC/Flying Machine Arena
• maintain full attitude including yaw, by detecting a motor failure somehow (possibly detected with existing sensors from a sudden unprompted roll/pitch change that's exactly along the angle of a motor?), which leaves only 66% (four 6ths) of original throttle sum - cut out the opposite motor, and fly as a quad, rather than distributing it among the three on the healthy side for roll/pitch compensation (which would cause unstable yaw). But this 'quad' may still have yaw problems because each of the remaining motors will be spinning the same way as the one next to it across the dead arm (rather than the usual quad setup of matching motor spins diagonally opposite each other). And 66% of original power is not fantastic.

I guess it's a major benefit of quoctocopters (can this catch on for X8s?) that they can lose a motor, cut one opposite motor and maintain R+P attitude with 75% of original throttle available (6 out of 8 motors) plus the efficiency bonus of no longer having two props sharing the same wash - is that right? Does the spin imbalance across the unhealthy arms still cause a yaw problem and force it to power back down to effectively 4 out of 8? Don't know enough about how yaw is controlled to assess this :S

[wjax]

This is a great feature in terms of safety
Just seen the video of NAZA managing a motor loss in an hexa beautifully! I am sure we can do it too! And better!

[ghost]

Some interesting reading about solution this guy uses a Kalman filter for estimation & check if something failed I checked it already but it is out of my mathematical knowledge :)
http://www.feec.vutbr.cz/EEICT/2014/sbornik/03doktorskeprojekty/03kybernetikaaautomatizace/[email protected]

[rmackay9]

I've created a patched version of AC3.2 which allows reducing the speed of one motor in case anyone wants to give it a try, this is how to use it:

1. Load one of the two firmwares below onto your Pixhawk (sorry, only quad or hexa although I could compile others)
   a. Hexacopter: https://dl.dropboxusercontent.com/u/3852647/AC3.2-motfail/ArduCopter-3.2-motfail-hexa.px4
   b. Quadcopter: https://dl.dropboxusercontent.com/u/3852647/AC3.2-motfail/ArduCopter-3.2-motfail-quad.px4
2. Go to the full parameters list (or tree) and set these parameters:
   a. TUNE = 99
   b. TUNE_LOW = 0
   c. TUNE_HIGH = 1000
   d. MOT_FAIL_NUM = 1 (the number of the motor that you want to fail)
3. Test it’s working by moving the ch6 tuning knob up and down and push the Refresh button on the MP’s full parameter list and check the MOT_FAIL_PCT is changing from 0 to 100 (0 = motor off, 100 = motor normal power)
4. Plug in the battery and test again that you can shut down the motor by putting ch6 to its lowest position

So with this it should be possible to test a motor failure in a somewhat safe manner because you’re able to reduce its power in stages to see how it reacts. So you could try reducing the power by 10%, 20%, 50% etc to see how it responds. In the simulator, the overpowered quadcopter can handle one motor’s efficiency being reduced by 45% before it flips.

The patch consists of the last two commits in my repo’s AC32-motorstop1 branch: https://github.com/rmackay9/rmackay9-ardupilot/commits/AC32-motorstop1

I’m not planning on putting this into master because I can just imagine it’ll lead to more harm than good but that’s open for debate. At least for now we have a patch that people can use on an as-needed basis.

[rmackay9]

pushed to AC3.4.

[firdel]

Interesting thread.
My partner and I have to show a working hex with redundancy at a flight test for the Danish Transport Authority in aprox a month. We choose the pixhawk FC over the A2 because of the open platform.
It is a demand when flying with a setup above 1800grams, with people located inside your safty zone (Danish rules), that the UAV can be moved away from a specific point, with 1 engine failed.
Do i understand correctly, that with your patch rmackay9, we can demonstrate this, in flight?
With yaw out of controle, is there any controle as to moving the UAV in a specific direction?
Im very new to the pixhawk, and im not responsible for the patching/programming part. Its not my area. We do have a very competent man helping us. He guided me to this thread.
If there is some directional controle over the UAV, with your patch, and 1 engine shut down, could this be implemented in our pixhawk to work "IRL", and not just in a test enviroment.
As i read the posts here, the only controle left, in case of engine failiure, is vertical controle?

Best Regards
/firdel

[ricardocrl]

Hello everyone,
I'm little familiar with the ardupilot and I am trying to understand its current status and future approach on this topic: redundancy in Hexa and Octa copters in case of motor failures.

I'm working in an engineering company and we are happy to see this discussed and at least know that it is a question of time to see something implemented.

My question now is, what is the planned approach? Or no plans yet?
My understanding is about the ardupilot philosophy is to make everything as hardware-independent as possible (e.g. special ESC). Would this be possible?

Thank you

[firdel]

Hi Randy.
I have been flying quite some with with your motor stop simulation branch on a Hexa (https://github.com/rmackay9/rmackay9-ardupilot/commits/AC32-motorstop1), with mixed results. What I am trying to achieve with one engine out is to fly the hexa away from its motor fail location (something like 20-30 meters), and then if possible land.
When testing I have mostly been in stabilize, and what happens as one engine is turned down is that I have to compensate for yaw to keep it from spinning. This results in there not being enough thrust to keep it in the air, nor to fly it 20-30 meters away.

If you could give me your 2 cent on the following questions I will be very grateful.

Which flightmode do you think would be best for the above situation? -And which would only make matters worse?
Should one ignore the spin, and then do the flyaway in simple/super simple?
I have currently been using 3.2 firmware with the motor fail branch, would there be any benefit in using 3.2.1 or 3.3-rc10 with the motor fail branch instead?

Regards

[lthall]

Hi all,
A hex will need to reduce yaw authority to maintain roll and pitch control. So if you want to make sure your hex will survive a motor out you need to ensure you have a good roll and pitch tune. You must ensure you have plenty of additional power, your copter should be able to lift 3 times it's take off weight (not as hard as it sounds).

Things that will improve the copters ability to maintain roll and pitch performance you should keep the Rate_Yaw_Imax and MOT_YAW_HEADROOM low.

So because you should be losing your Yaw authority if you lose a motor, you should use simple mode of some flavor or simply RTL.

[Sunrik]

Which flavor of simple/super simple do you reckon would be safest to test with loiter+super simple, alt hold+super simple or stabilize+super simple?

What is considered a low value of RATE_YAW_IMAX?

Finally, I cannot find MOT_YAW_HEADROOM in the parameters list?

[rmackay9]

Linking new issue related to this one: https://github.com/diydrones/ardupilot/issues/3294

[roque-canales]

Randy, there's possible for you to share compiled 3.3.1 hexa motostop version for test it ?

[csriddel]

Randy,
I think I have a real example of a motor (or ESC) fail for a hexcopter (DJI S900). I believe motor 2 (left) stopped working for whatever reason -- and it would seem the pixhawk shut down motor 1 to compensate... is that what you'd expect? Log linked below. The result was a rapid descent and crash. I'm not sure I'm seeing a loss of yaw control, just severe pitching around the longitudinal (x?) axis.

https://www.dropbox.com/s/bd9j9zv70spx3wn/2015-12-22%2012-06-24.zip?dl=0

[matheweis]

Just came across this thread while discussing ArduCopter's capabilities in terms of handling rotor failures in hex mode with another user. Just in case this will be of any help, here is the log of a flight that experienced catastrophic failure of motor 5 (front right) while running stock 3.2.1: http://eisbox.net/tmp/15-08-17_19-13-20.bin Sorry about the age, I knew the failure wasn't due to ArduCopter so I wasn't posting all about it, and I'd forgotten all about any use for the flight logs until I downloaded them recently.

[rmackay9]

I think AC-3.4 performs fairly well but perhaps we haven't done enough controlled tests to close this issue. Pushing to AC-3.5.

[RJBeckwith]

Thanks Randy. I had a motor failure once with a Hexa running a DJI Naza32 and it recovered very nicely. Turns out a motor from a previous crash had a connection problem and I didn't know it until the motor stopped in mid air. Very cool to land the thing without incident.

I would love the same capability with Arducopter.

[lthall]

There is some basic and unavoidable maths at play here.
To maintiain roll and pitch athority in a hex conficuration without 1 motor and without chainging the motor mixers you give up a lot more than 1/6th the lift capacity (1/3 I believe but I don't have my calculations on me).
You can get a little of that lift back but if you know what motor is out but you still have only 50% thrust (I have to double check my results).

So if you have a hex the limitations above are unavoidable and represent the best any controller can do (without making use of the dynamics of a fast yaw spin).

The only way to not loose yaw control is if the CG is offset in the right direction so if you loose a motor and you don't loose yaw control then you probably got lucky because if you lost the motor on the other side you probably would have had an unavoidable crash.

Given a properly set up hex you also need a very well tuned hex with yaw Imax and yaw headroom set low to maximise the roll and pitch athority. If you don't have this then again the copter crashes.

So while I understand the desire to have ensure a hex can make a safe landing we can't do rewrite the laws of physics and we can't build and tune every hex as we arn't selling ready to fly copters. The only thing I would say is we may be able to add some notes to the wiki on how to maximise the chances of a motor out situation on a hex. We may also be able to drop the yaw mix to zero in the case of large roll and pitch errors.

I personally recomend we close this issue.

[R-Lefebvre]

I would just like to add to this issue, that while we cannot rewrite the laws of physics, we should at least try to detect the condition and warn the operator. This is the case especially with octocopters. But I've even seen hexacopters flying normally with a motor failure. But if it's far away, the operator doesn't know. They might want to land right where they are if it's safe. If they starting trying to fly fast, it could lead to a crash even though the crippled copter could hover safely.

I know we've talked about this Leonard, just officially entering it in Github. :)

[robustini]

During a single motor failure with S900 and DJI FCU the pilot It can govern the hexa where it wants using the "Course or Home Lock" (Simple/SSimple in APM:Copter):

https://www.youtube.com/watch?v=HQ7wa5cBT_w

It's not difficult to move the drone with "S" and "SS" during yaw turn and I often shown:

https://youtu.be/RF0wnQv3aAc?t=334

So it's just a matter of code and payload, ArduPilot can and should do.
I asked for a long time to have the possibility to turn on/off a motor through a switch for testing, upon request ignored... :-)

[panky0]

hi, noticed your discussion because today my hexa crashed because a self locking prop (Motor #2 "left-middle") did unlock itsself and my hexa went down (not really crashed, just went down in a more or gentle way and to the side (where the prop was missing) - only my 3d printed legs were harmed);
i was flying position hold;
I have a Navio2 (by emlid) and using 3.4-rc4; but i didn't do any of the recommended changes to MP settings; i use 6x turnigy 3549/6 790kv motors with 4s and ( i know at least 2 inches too big) 13inch props; - but usually my hexa flies overpowered -maybe not overpowered enough; my very raw estimation is about 400w per motor (with these oversized props) and almost 5kg AUW;
here is the log - hope it helps:
https://drive.google.com/file/d/0B9PqIZ8f2EDsZUhfVnEzU09VY28/view?usp=sharing
(current is wrong since i have 2x16AH 4s lihv and cannot fly more than 20min at the moment)
i didn't have much time to save the copter, tried to steer against the drift in direction of the lost prop - but not enough; and i didn't change flight mode; yaw didn't change drastically...

[lvale]

Adding to @robustini comments this video https://youtu.be/hFZ_d9mbVGU?t=1m20s from 1:20 gives some good leads. Stopping motors, enabling SS. (and considering that there is enough power (like @lthall mentions too)

[geofrancis]

i have had motor failures on a couple of hexacopters now and from what I have seen it wont fall out the sky but pitch in the direction of the failed motor and slowly decend, the problem is that when it pitches it accelerates and keeps picking up speed until it hits the ground.

generally it will take less damage than a quad falling vertically downwards but its not ideal if your working in an enclosed area where the things around the aircraft are far more expensive than the aircraft or there are people around.

from looking at the logs the throttle output to the failed motor always hits 100% and is held there until it crashes.

To detect a failed motor what about checking if a motor output is at 100% and des pitch and roll are out by a number of degrees in the failed motors direction

motor 1 failure would roll to the right
motor 2 to the left
motor 3 would be forward and to the left etc

then lower the yaw headroom and imax automatically to keep pitch and roll at all costs, you could even have parameters for failed motor that could be set this would put the machine into a fast spin in order to keep pitch and roll but it could be landed with some degree of control.

[lvale]

Assigned to @rmackay9 because he has already done some study on this.

Perhaps this PR might also be relevant https://github.com/ArduPilot/ardupilot/pull/3672, ie detecting unexpected saturation of outputs for some time and adjusting the other outputs ?

[csriddel]

We’ve lost two s900 helis from what appears to be single ESC failures. The failed ESC seems set to 100%, and the opposite ESC gets 0%. That leaves 4 motors to maintain altitude and stability and it just doesn’t seem to work. Perhaps I’ve diagnosed incorrectly. I’d be happy to forward .log and .tlog files.
Steve

[gerbertkuiper]

Very interesting discussion here. I would like to add the following;

Mounting the motors tilted on the arms increases the yaw authority drastically and allows for a much lower yaw headroom (thus greater chance of survival after motor failure).

Tilting a motor 5 degrees around the arms length axis (so that the thrust reinforces the yaw momentum of the motor torque) makes that 8.7% of the thrust is working as a yaw moment on the copter's frame. 99.6% is still working to keep the copter airbone. So it does not reduce the flight time significantly.

Another advantage of motor tilt is a more stable vertical descent because props are less suffering from their own propwash.

I agree with geofrancis that in the end the best solution is that if the flight controller can not hold the desired roll and pitch angles, it should automatically put yaw headroom to zero and enter (super) simple + Return to Launch mode to safe the hexacopter.

[R-Lefebvre]

Yes, but keep in mind, that motor tilt will also result in the reverse action. Lateral airspeed will result in more or less lift on that motor, which can then make roll/pitch stabilization more difficult.

This effect has not been well characterized, but theoretically, it's worth considering. Just pointing it out.

[wjax]

Given the general poor quality of ESC all around, this item of UAV surviving a motor loss should keep all our attention.
Realiability and safety should go before new features IMHO!

[R-Lefebvre]

If you use high quality ESC's, there is no issue.

[geofrancis]

i have seen expensive escs and motors fail, just less often,

[wjax]

No matter how good is. I WILL fail.

In aeronautics a single failure should never imply a catastrophic safety event.

Having an autopilot that can handle a motor failure and land with minor damage is just a crucial feature and much more important than any other.

[csriddel]

I have two logs in less than two years that seem to indicate an esc/motor failure on two DJI s900s. It's an expensive and dangerous problem.

On Nov 19, 2016, at 09:03, Jesus <[email protected][e-mail]> wrote:

No matter how good is. I WILL fail.

In aeronautics a single failure should never imply a catastrophic safety event.

Having an autopilot that can handle a motor failure and land with minor damage is just a crucial feature and much more important than any other.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/ArduPilot/ardupilot/issues/838#issuecomment-261718858, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AP181mOJGs73n5HmFiTOV5XqbS43zYIPks5q_w-6gaJpZM4BhVd9.

[tabascoz]

Hello, I would like to inform a crash with one motor failure. http://discuss.ardupilot.org/t/hexa-crash-one-motor-failure/13576?u=tabascoz Willing to test and help on this matter!

[peterbarker]

Here's a simple patch to simulate a motor failure in SITL: https://github.com/ArduPilot/ardupilot/compare/master...peterbarker:sim_mot_fail?expand=1

The parameter SIM_MOT_FAIL is a bitmask of motors to fail.

[AndKe]

First:
@rmackay9 's test patch,(disable a motor midair) should be a standard part of ArduCopter - triggable by CH*_OPT
Any user would be more aware of actual behavior , and could test/train more easily then.

Then:
We have one clear-as-day sign of motor failure: , the failed motor's RC.OU is max, the opposite motor is commanded to idle.
In addition, CTUN.ThO tells us throttle demand is maxed out.

In such situation, remaining motors should be mixed so they were unaffected by the failed/ and idle motor.
If 6 failes and 5 is commanded to idle, then 1,2,3,4 should be scaled up to 100% or until thrust demand is met.

[magicrub]

In most cases, throttling to 100% means you've lost the ability to control
since you have no headroom for feedback.. but in this case doing 100 should
probably be allowed and made an exception for in the motor lib.

On Dec 27, 2016 5:38 AM, "André Kjellstrup" [e-mail]
wrote:

First:
@rmackay9 https://github.com/rmackay9 's test patch, should be a
standard part of ArduCopter - triggable by CH*_OPT
Any user would be more aware of actual behavior , and could test/train
more easily then.

Then:
We have one clear-as-day sign of motor failure: , the failed motor's RC.OU
is max, the opposite motor is commanded to idle.
In addition, CTUN.ThO tells us throttle demand is maxed out.

In such situation, remaining motors should be mixed so they were
unaffected by the failed/ and idle motor.
If 6 failes and 5 is commanded to idle, then 1,2,3,4 should be scaled up
to 100% or until thrust demand is met.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ArduPilot/ardupilot/issues/838#issuecomment-269327375,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEj7G-Y9VIxN-bq1x7gtZ6ZhEgFWVZnYks5rMRTlgaJpZM4BhVd9
.

[tabascoz]

It could be less than 100% enough to allow small pitch/roll movement.

[AndKe]

@magicrub @tabascoz Of course, I did not meant it like that.
I am well aware of the need for control. I just assumed everybody here would understand my meaning, not take it literally.
Let me rephrase; "remaining motors should be scaled up till one of them (depending on attitude) hits 100% , or required total thrust is met"

Or: to put it in another way; the post with a log @tabascoz posted a link to some days ago. illustrates my point pretty well:
One motor failed, the opposite was set to idle. The vehicle drops altitude fast, yet, there is no sign of the remaining four good motors working any harder. - There's plenty more thrust to get, but it simply not used, due to the way motor throttle is mixed. It seems to me that if - in that situation, motor throttle for 1,2,3,4 were scaled up , until any of those, in any moment, reached 100% - or thrust demand was met, then the hexa would just fine.

The sharp minds here, can - as I described earlier, easily detect such situation, and make AC handle it.

[magicrub]

I was literally saying that in this case going to 100% may be OK. I'm
thinking for a quad though, not a hex.

On Dec 27, 2016 11:27 AM, "André Kjellstrup" [e-mail]
wrote:

@magicrub https://github.com/magicrub @tabascoz
https://github.com/tabascoz Of course, I did not meant it like that.
I am well aware of the need for control. I just assumed everybody here
would understand my meaning, not take it literally.
To put it correctly , would be: "motors 1,2,3,4 should be scaled up till
one of them (depending on attitude) hits 100% throttle requirement, or
required total thrust is met"

Or: to put it in another way; the log @tabascoz
https://github.com/tabascoz posted a link to some days ago. illustrates
my point pretty well:
One motor failed., the opposite is set to idle. The vehicle drops altitude
fast, yet, there is no sign of the remaining four good motors working any
harder. - There's plenty more thrust to get, but it simply not used, due to
the way motor throttle is mixed. It seems to me that if - in that
situation, motor throttle for 1,2,3,4 were scaled up , until any of those,
in any moment, reached 100% - or thrust demand was met, then the hexa would
just fine.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArduPilot/ardupilot/issues/838#issuecomment-269371347,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEj7G1S6B5Tk1IlJYGVcaliRlpVQFHRRks5rMWa_gaJpZM4BhVd9
.

[tabascoz]

Randy's patch ported to Copter-3.4 branch: https://github.com/tabascoz/ardupilot/commits/Copter-3.4

[El-Mango]

https://www.youtube.com/watch?v=w2itwFJCgFQ#t=6m21s

[AndKe]

@El-Mango that's not very relevant, those quads are surrounded by cameras , and powerful image processing is telling them what to do, so they are not bothered by gyro and accelerometer limits like "real" /independent machines.
Also, most quads may not have enough power , or asymmetric payload , and be unable to achieve yaw rates needed to fly that way before crashing.
-So while it's fancy, not a practical solution.

Anyway - this thread is about much simpler handling of such situations for hexa/octo multirotors. (which is simpler)
A solution for a quad would be very different.

[geofrancis]

why is the yaw headroom so high by default? it sounds like that could be the cause of most of the problems of loosing controll due to motor failure? i would rather it was spinning on the spot than flying sideways.

[AndKe]

@rmackay9 What's your opinion about this ?
Should't it be fairly easy to detect/handle a problem in hexa and octo, as I described above ?

[El-Mango]

@AndKe My opinion is that, even if those quads are surrounded by cameras, that doesn't effect the laws of physics... Fancy or not. The only way to keep up those things is exactly with gyros and accelerometers as we do.
Spinning is a reaction, considering that the two elements of the remaining working pair of propellers are spinning in the same direction. So, you let the system spin in aim to make the two remaining propellers spin at the the speed that you need.
In this condition, it's possible to stabilize the altitude. (remaining pairs are keeping up the system)
Even if the system spins, you need of course to keep track of its position and orientation in every moment.
At that point, applying an extra sinusoidal power, synchronized with system's spin phase, you can determine a movement of the system in the direction you want, keeping a constant altitude.
Condition for the system to be stable while spinning with remaining working pairs, is that point of application of force of the props must be higher than the Center of Gravity, otherwise it would flip upside down with propellers pulling towards hearth.

[AndKe]

@El-Mango I am not going to argue with you here, it's just so much more to "promise" people with quads that already fly at 50% thrust to quickly get into the velocity needed to achieve that.

While possible - it's still a very different situation (and solution) than what's needed in this case. (see issue title)

Yes, I would love to AC do quad recovery in real life (actual useful machine with payload do that autonomously when needed) - imagine ArduPilot retrofitting Solo with such ability - that would impress a lot of people...

[El-Mango]

I just pointed out this case study because a strong theoretical solution, if proven working, would provide a more consistent support to all possible implementations, rather than empiric ones.
It would be nice to have a "Recovery Mode", entered after fail detection, valid for all copters in general and verifiable with some sort of auto-tuning...

[geofrancis]

the problem el-mango is that very few machines are perfectly balanced on all 3 axis like the ones in the video, most quads would just wobble out the air if they tried to spin that fast.

[El-Mango]

I don't thin balance is the key.
If case of a not balanced system, meaning Center of Gravity not coincident with Center of Force, the centrifugal force of the spinning itself, would keep the system in some kind of stable periodic status.
The wobble would be then the combination of a spin around an axis passing through the object plus an orbit around Z axis.
Key points are, in my opinion:

• Center of Gravity lover than Center of Forces (It depends on how the system is built) preventing the flip of the system
• Sensors and processor must keep up updating (EKF) algorithm, to estimate vehicle position, velocity and angular orientation, while the system is wobbling and spinning...
  Anyhow, I beg your pardon if i appeared too prolix on this matter...

[rmackay9]

@geofrancis, Re the MOT_YAW_HEADROOM default of 200... it's very possible it should be lower. I think it was an educated guess by Leanard(@lthall) and I. We can make it smaller, it's just that I don't personally know how low it should go. I think there will be a level below which we will start seeing complaints from users that vehicles change their heading after a large disturbance.

[Ivosch]

You are focusing to high level solution, when copter has not enough power to keep attitude and altitude together. But now ArduCopter can not manage this situation even when copter has enough power to fly with 4 motors instead 6. I crashed (lost altitude and yaw control) with 5 motors because CTUN.Th0 was maxed out and whole potential of running motors wasn't used. AndKe had good comment about it here before 20 days.
I propose to solve this in first step.

[peterbarker]

I'm pretty sure that SITL modification I made shows the same behaviour people are seeing on real vehicles. If someone wants to test bumping the power up to see if that fixes the problem it might show one way or the other...

[El-Mango]

I think solution should be found at high level. Adding a special "Faulty Mode".
If a motor breaks but the system isn't aware of that, it will try to stabilize itself, still counting also on the effect of the broken motor, obviously not receiving the expected feedback, due to the weakened relationship between cause and effect.
Once the fault is detected, instead, the system should update itself. To a new mode, moving from quad-to-tri or from esa-to-penta, a completely new state or model.
This way, every action will correspond to a proper expected reaction, resulting in a controlled and stable system.
Special case for quads: a broken motor would also preclude the use of the opposite one, to avoid flipping, loosing thus the whole couple. Meaning or the Clock-Wise or CCW pair. That means the system won't have the ability to control yaw anymore, at all.
The remaining pair though, should still have plenty of power to keep up the system, while yawing like crazy of course...

[AndKe]

I hope we all soon can have CAN bus ESC's , (and AP knows RPM and current) :)

[geofrancis]

@rmackay9 would that not only become a problem if the vehicle was very underpowered and using a lot of throttle?

[rmackay9]

@geofrancis, I think having a low MOT_YAW_HEADROOM would mostly cause problems for vehicles that have a motor imbalance between the clockwise and counter-clockwise motors. For example, a copter that has one motor leaning slightly can lead to this. In these cases I think that under stress (i.e. the vehicle is working hard to maintain roll/pitch) it would lose yaw control. That's what we want to have happen in severe cases.. but in more regular flight with just momentary loss of roll/pitch control, losing the heading control could be disorientating for the pilot.
You know though, I'm really not the expert. I'll have a chat with @lthall.

[geofrancis]

It would be something worth testing.

[Kiwa21]

Hey all,
Could we sum up this long-going discussion and have a plan of actions to make, at first, Y6 and X8 with enough power, safe to land with a motor failure ? I would like to see what need to be done in details :)
Thanks for your answers !

[rmackay9]

@Kiwa21,
I had a chat with @lthall and there's two things we need:

• single motor failure detection. Leonard suggested we modify the AP_Motors_Matrix class so that it looks for a single motor >95% while all other motors are <75% (those percentages are guesses and could be modified based on testing). We should probably use a counter and/or bitmask so that the situation must continue for some amount of time (0.5sec?) before we decide the motor has failed and avoid false positives.
• based on the above checks, exclude the "failed" motor from the saturation handling code. So for example, we should exclude from the calculation of "rpy_low" and "rpy_high" here in AP_MotorsMatrix.cpp. https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_Motors/AP_MotorsMatrix.cpp#L231

According to Leonard, doing this will theoretically allow the overall throttle of the motors to climb to 66% instead of the current 33% while still maintaining attitude control.

This work is a bit tricky and I have a lot on my plate at the moment so it would be great if someone else could try to make some progress on it.

[Kiwa21]

Thanks Randy,
The motor failure detection might get a bit tricky without a clear feedback from ESC rpm / current drawn.
I mean when you go forward or upward very fast can't you get some scenariis where all motors are < 80% and one was almost at 100% to correct for bad weight distrib and/or yaw correction ?
I guess on a push-pull frame it will be easier since the output throttle distribution will be more even between push/pull motors.

Might go into this matter sometimes in the next 6 months after tweaking the precision landing a bit more :) (by the way I'll be interested to talk about this with you to improve the last 0-50cm)

[AndKe]

@lthall 's suggestion should work fine, please remember that another good clue is throttle demand being 100% while only one PWM output is 100% over some time (and the opposite is usually idle.)

[Ivosch]

@rmackay9
to avoid false positives detections, maybe you could add more conditions, like increased errors over a limit in yaw/roll/pitch navigations. Yes, it won't detect failed motor everytime, like when copter is overpowered and can handle situation - but in this case it does't matter I think.

[Juzzle1]

@rmackay9
Another thing that may need to be considered here is how this all fits in with parachute deployment (which currently looks for a pitch / roll angle error of >20 degrees.)

If a motor fails it could be possible that the copter would tilt by this much before the motor failure detection has had time to take effect.

[hsteinhaus]

@rmackay9 The code for detecting motor saturation as well as the motor saturation bitmask is basically already contained in my PR #3672 .

[tuloski]

I have an idea to keep full controllability matrix also in case of one motor failure, that is roll, pitch, yaw and thrust are still controllable with 5 rotors. More details soon, because it might be worth a publication in robotics ;)

[Kiwa21]

@tuloski Are you talking only for hex ? or anything more than 5 rotors ? X8 ?

[tuloski]

For hex...for X8 is way easier.
BTW it seems they did it (theoretically): https://arxiv.org/pdf/1403.5986.pdf

Basically changing the standard configuration of hexa with alternating propellers rotation PNPNPN, using a configuration PPNNPN can ensure that in 4 failures you keep the controllability, in 2 failures you don't. It is very interesting to implement probably. And btw in the two cases where you don't have full controllability, you can still loose yaw control, but land safely having the thrust, roll and pitch.

[gerbertkuiper]

I did some controlled tests with my hexacopter. Here is a very short video of the most recent flight. Unfortunately I was stopped by local autorities short after take off, but you see a couple of second of motor failure;

https://www.youtube.com/watch?v=nWKshZ_FVBA
The settings and characteristics of my hexacopter are;

Weight; 3990 gram
Max thrust (based on measurements with only one motor connected to battery);
6 x 1700 gram = 10200 gram
Throttle at hover; 360
MOT_YAW_HEADROOM; 0 (in normal flight yaw authority is good, but after motor failure no motor capacity is wasted on trying to keep yaw control, and attitude control is prioritized)

I will try to make a longer video in a couple of weeks. Here are the log files of the previous (longer) flight with motor failure;

1 13-4-2017 19-28-46.bin.zip

[tuloski]

@gerbertkuiper which software was running?

[gerbertkuiper]

APM V3.3.3 (acf2e10c)

About the log file;
A low value of RCIN/C14 = failure of motor 5 (RCOU/Ch5)

The first couple of times the ch 14 value lowers gradually because I wanted to test carefully the behaviour of the hexacopter. After line nr 300 you see some abrupt changes in Ch 14 from high to low.

[gerbertkuiper]

And RCIN/C8 high = simple mode (necessary for controlling the direction of the spinning hexacopter)

[rmackay9]

By the way, @tridge recorded a video test of a simulated test of a motor failure on a hexacopter.
https://www.youtube.com/watch?v=ICPqm2v6Cic

In this test the vehicle descends quite rapidly and loses yaw control but doesn't lose roll/pitch control. The result depends upon the various things discussed above including power to weight ratio.

[rmackay9]

Here's a real-life report of a hexacopter motor failure that didn't end well during the AC3.5 beta testing. http://discuss.ardupilot.org/t/hexacoopter-3-5rc4-crash-with-single-motor-failure-why

[Kiwa21]

Update as well :
http://discuss.ardupilot.org/t/y6-copter-3-5rc7-rc-out-saturation-logs-analysis/18795

[gerbertkuiper]

I finally had time for making a new video. Now you can see clearly what happens after motor failure;

https://www.youtube.com/watch?v=ap8ngQqt7s8

[RJBeckwith]

Great video and explanation of your methodology. Thanks for taking the time to pull this together.

[geofrancis]

very interesting, perhaps super simple should be engaged automatically if motor failure is detected?

[highvoltageDC]

Hi,
I want to share my hexa hard landing details, I hope it might help for hexa redundancy improvements.

My hexa has hard landed because of motor #3 failure. Chain of events in order as far as I see:
1) 19:42:47 Motor 3 ESC wire break (4:58 on the video)
2) It was able to be controlled in loiter mode for pitch, yaw and roll except altitude. It was continuously loosing altitude, I was unable to understand what is going on so I was trying to move it at full speed to safe area.
3) Last seconds I switched to RTL mode as a last hope.
4) It has hard landed on the battery pack and gimbal at 8m/s vertical speed, landing gears were retracted.

BR
Mert

Hexa details:
4s 16Ah Multistar battery
Tarot680pro frame
Tarot 4006/620kV motors
1355 carbon fiber propellers
30A KissESC v1.02
Pixhawk 2.4.7 clone runs AC 3.5.2 FW (It has all critical components like MIC5332, L3GD20H, MS621FE, LTC4417 and Revision 3 STM32F427)
OrangeR1220XLR receiver with one R110XL sattelite
Px4flow 1.3.1 with sonar
Lidarlite V3 with i2c connection
Two M8N gps and one external compass
2W 5.8GHz VTX
MinimOSD
Feiyutech mini 3d pro gimbal with xiaomi yi cam

Video link: https://youtu.be/W9cj7H0gqQA
Log file: log.zip

[geofrancis]

now that BLheli 32 speed controllers have been released and support telemetry, inc rpm and current ardupilot could be a lot smarter about fault detection, it could even predict a failure if the motor deviates from its ususal operating ranges.

i know this has been talked about with CAN speed controllers but those are so rare and expensive that its hard to see it gaining traction in the wider community, with the blheli32 escs, they are cheap, they are here and using Dshot with telemetry they can do almost everything a CAN esc can.

[KirkKirk]

HI guys,
I would like to share my experience with with this issue so maybe will be of help to someone else.
I've been on holidays and I've had a lots of time for experimenting around the gimbal and I haven't had luck to make the Pixhawk1 follow ROI with precise controlling Tarot gimbal's Pan and Tilt.
So I decided to make a video of the big rock in the sea with FC control of the gimbal's tilt and drone heading to the ROI.
The mission was: flying around the rock, 2100m in total flight path on altitude of 50m and the ROI(alt 20m) was on the rock. If I ignore a casual "jelo" the videos are great. The "jelo" is present only when the wind is gusting (~35km/h). As I was testing consistency of the drone behaviour, I made two videos with the same mission and consistency has been confirmed.
Unfortunately that was my last video because I lost my drone.
About the reason, I will only assume based on the .tlog from MP because the drone is on the bottom of the ocean.
In short, in the new ROI mission the drone lost one motor and couldn't recover because of the strong wind (~35-45km/h). I was not aware of the wind speed out of the cove until I went on the "crash site" to look for it.
My personal opinion: if the FC was controlling the pan/tilt and if was the drone flying without pointing towards ROI, then it will manage the loss of one motor and with lots of hard work of the other 5 motors it will come back safely.
It is my opinion only, and I will appreciate an opinion from someone who is experienced/professional in it.
Anyhow, lesson 1: wind in the cove and wind in the open sea is different :(
lesson 2: Drone must have thrust/weight ratio more than 2:1 (mine was hower on 47% thrust).
lesson 3: take with you anti-stress pills when you executing an autonomous mission (in case if you lose the drone)
At the end, I would like to have advice what is the best combination of the parts for my new Hexa. This is my second lost drone with Pixhawk 1 FC and I would like to support the Pixhawk project, so I believe someone will give me confidence to buy the Pixhawk 2.1.

The configuration was:
Frame: Hexa F550
FC: Pixhawk 1, AC 3.5.2
Power Module: 3DR Power Module 80A
Battery: 4S 10C 10000mah
ESC: LittleBee BlHelli opto 30A (w/o BEC)
Motors: Gartt 700kv
Propellers: 1045 carbon fibre
Radio: Radiolink T10
Receiver: Radiolink 12CH
Telemetry: 3DR
Video: 2W 5.8Ghz
Gimbal: Tarot T3D_III
Camera: Gopro 4 black
LGear: Retractable
Lights: 7x Led

Thanks,
Kirk

[KirkKirk]

I am sorry for my impatience but I am desperate to find a reason for the crash so in the future to avoid the same mistake with my new build.
All examples of motor failure (what I found on the net) are not in auto mode but under pilot control and I am not an expert/developer but in the next FW I believe it is good to have the simplest way for drone to survive in Auto Mission would be (I think):
after the malfunction is detected (one motor 100% and the opposite <20%),
the mission to be canceled
and RTL to be triggered in simple mode (WP_YAW_BEHAVIOR=2 (Face next point except RTL)).
Everyone have brilliant ideas about solving this issue but let's start with something.

Attached is the .tlog file is from the last flight. I know there is not much data but I hope it will help for future development.
(Drone Crash in the Ocean.zip)

Kirk

[AndKe]

@KirkKirk I see no evidence of motor/esc/prop failure in your logs. The vibrations were increasing in the time before/when throttle demand increased.
You staret at 16.7v ( so it's a 4S back) , and were at 3,3v/cell when the throttle demand climbed, whileloosing altitude, 12,8v was logged (3,2v/cell) , assuming the cells are perfectly balanced.

So your crash was mostly due to empty battery, not related to this issue, or AP's handling of what happened.

[KirkKirk]

HI @AndKe
I am not sure why that is happened. I never noticed before that high oscillations in the voltage and usually I was in the air 15-20min with the same buttery.
About the motor failure, I think it's visible on the graph (below) where the motor no4 is 100% and the motor no3 is 0%. (the rear right failed and the front left compensated)

Please correct me if I am wrong.
Thanks

[AndKe]

@KirkKirk 7 seconds before you see those motors max out, you can throttle demand go from your typical 25% and up, 5seconds before they max out, I see voltage drop a lot as throttle demand continue to climb to 100%.
Flying at 25% - you should have been able to maintain altitude with such config, ..if those other motors had power while descending, the total power (W) is not increased compared to when you climbed.

Remember that power is voltage * current, as voltage drop, ESC/motors may not be able to produce enough power anymore.
Yes I do think there were some sort of failure, but ultimately, it should have been able to fly, not only reach 100% and descend.

[KirkKirk]

@AndKe the same battery I had used before for similar missions and there wasn't any issues with the dramatic voltage drops. I can post logs if it is necessary.
Anyhow, the battery was fully charged and you are right about the power and voltage drops because the battery pack was only 10C (10000mah), but if you can see the complete picture of the FC requirements from motors you'll see that from the other 4 motors it was never required 100% because motors were producing enough power and looks like on the last 2 sec "the drone was a bit more stabilised".
What I saw from the distance and what the .tlog shows, the drone flipped, wobbled, yawing, lost orientation and all of that happened because it was trying to maintaining its nose towards the ROI with no abilities for Yaw with 5 motors and because of that it was losing altitude. Maybe if it was on higher altitude then it would have had have more time to "recover" or something but all that happened on 40-50m altitude and the "fight" lasted less than 10 sec which means the diving speed was less than 5 m/s .
This symptoms are well known to everyone who lost (or turned off) one motor and tried to fly in stabilise mode without simple or super simple control. Good example https://www.youtube.com/watch?v=ap8ngQqt7s8.

So I am still confident about a very simple solution for this issue (in manual or autonomous missions) with triggering RTL in SuperSimple mode when the motor/propeller lost is detected.

BTW, last Wednesday I got my new Pixhawk 2.1 (with Here). The rest of the parts (esc, motors,lidar, 3DR telemetry, ....) will arrive in the next 2-3 weeks and I am very confident developers will create a task for this in the next FW. I will be the first one who will test it.

[gerbertkuiper]

@KirkKirk
I would anyway put MOT_YAW_HEADROOM to 0 (in normal flight yaw authority is good, but after motor failure no motor capacity is wasted on trying to keep yaw control, and attitude control is prioritized)

See also https://www.youtube.com/watch?v=ap8ngQqt7s8
At 1:00 you see one of my motors stop......

[KirkKirk]

@gerbertkuiper
In the beginning, on my behalf and the behalf of many others, I would like to thank you for contribution of your tests which allow us to learn and develop.
Following your example, I made MOT_YAW_HEADROOM=0, but as I explained earlier it did not help in my case.
As I am still in the building stage of my new drone, and as you already have a functional setup for the controlled on/off of one motor, can you please do another test with a motor failure, but this time during autonomous mission with ROI. In that case, the drone will try to keep it's nose to the one point with 5 motors and minimised ability for yaw.
I believe that the result of the test will totally clear this treat/question and after almost 4 years the #838 will be closed.

Thanks in advance,
Kirk

[gerbertkuiper]

@Kirkkirk

Thanks and apologies for my late reaction. I have been looking to the log file you submitted, but it is difficult to deduct exactly what happened and why. I wrote my ideas down below.

About the tests; I will keep your request in mind to perform a motor failure test during autonomous mission/ROI. However, soon after the previous tests in early june, I removed the setup from my hexacopter and continued with mounting the gimbal and camera control equipment. So, it might take quite a while before I will re-rig my hexacopter back to the motor failure configuration.

Also, if you want to be sure your next hexacopter configuration does continue flying after motor failure, I think in the end you have to test your configuration yourself as well.
If I would test my hexacopter during ROI and it would fly, it would mean that your previous hexacopter did not have the basic requirements for continuing flying after motor failure. If you had MOT_YAW_HEADROOM set to 0, I think the most likely cause was lack of thrust (hovering at 47% versus my hexacopter at 36%).
If I would test my hexacopter during ROI and it would crash (not possible to test ROI while hanging on a rope) it would mean that Ardupilot cannot handle motor failure during autonomous missions. Still your hexacopter may or may not have the capability to continue flying during manual flight after motor failure and you would still have to test this yourself to make sure.

Concerning your flight;
Can it be that propeller nr 4 got loose? The increasing vibrations at 55:11 could indicate this. Was this a not-self-tightening clockwise prop with one hole (no T-style with 3 holes)?

At 55:18 the propellor slips completely. Probably because of lack of thrust (I suspect hovering at 47% is just not enough) of the remaining 5 props, your hexacopter was not able to maintain attitude and altitude.

This got probably a lot worse because of the strong wind. Was the wind from land to sea and were you flying at the lee side of a cliff? If that was the case your hexacopter might have been suffering from heavy turbulence as well.

The more I think about it, the more I doubt if the hexacopter was trying to point its nose towards the ROI if you had MOT_YAW_HEADROOM set to 0.

I think it is only the high level flight controller (that takes care of position) dictating the yaw direction to the low level flight controller (taking care of attitude). For the low level flight controller, it does not make a difference whether the desired yaw comes directly from the pilot or from the high level flight controller. If MOT_YAW_HEADROOM is set to 0, the low level flight controller will always prioritize pitch and roll over the desired yaw. I am not sure how this works in Ardupilot and I will try to find this in the code (if I can with my limited programming knowledge) when I have more time. Or maybe someone more familiar with the Ardupilot code can answer this.

About assuring the maximum amount of thrust on you next hexacopter; You could build a test setup for this, so you can test maximum thrust and thrust versus power demand for a range of motor-prop combinations. You buy some different motors and props and after testing all combinations, you order the best ones for your hexacopter. Concerning the propellor; besides efficiency and maximum thrust, also choose on basis of propeller weight; the lighter the more direct the flight controller keeps the desired attitude. And watch out with cheap plastic propellers. They can scatter under power. I had to pull a blade out of a wall in my house one time. It takes some money, time and effort, but then you know exactly the capabilities of your setup. If you are going to do this, I can send you some photo’s of my setup.

[lthall]

I am going to close this as there is no problem with the hex configuration in particular when it comes to attitude control in the case of a motor lost condition. It currently behaves as expected with loss of yaw control.

Attitude control in a hex can be enhanced by setting MOT_YAW_HEADROOM to 0 and ensuring the aircraft has enough additional thrust and is properly tuned.

Detection of lost motors and recalculating the motor mixer matrix is a separate issue/enhancement that is applicable to all redundant configurations.

[lthall]

This issue is a subset of this issue:
Copter: Detect and handle motor loss for improved redundancy #7525

[r-ryabyy]

Does the feature of compensating the lost of 1 motor in hexa/octo already developed ?
I saw a video from 2015 year https://www.youtube.com/watch?v=FNQgdcImpKs which shows excellent landing with lost 1 motor

[AndKe]

Depending on how much thrust you have left, it never been a problem.
Could be optimised tho...

On Fri, 20 Apr 2018, 17:28 Roman Ryabyy, [e-mail] wrote:

Does the feature of compensating the lost of 1 motor in hexa/octo already
developed ?
I saw a video from 2015 year https://www.youtube.com/watch?v=FNQgdcImpKs
which shows excellent landing with lost 1 motor

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ArduPilot/ardupilot/issues/838#issuecomment-383132834,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABZpqqjoIUiOTrOwDL0LQfWE9oP_lKXuks5tqf6NgaJpZM4BhVd9
.

[lthall]

Yes, as long as you have over 50% thrust available you should maintain attitude control. We have some ideas that will further improve this. We should be able to increase the thrust limit to 60% if we can detect the motor. A 20% increase is significant but it also has to be weighed against the risks of false positives and their possible consequences.

[xiongmaoer]

I finally had time for making a new video. Now you can see clearly what happens after motor failure;

https://www.youtube.com/watch?v=ap8ngQqt7s8

hey

During a single motor failure with S900 and DJI FCU the pilot It can govern the hexa where it wants using the "Course or Home Lock" (Simple/SSimple in APM:Copter):

https://www.youtube.com/watch?v=HQ7wa5cBT_w

It's not difficult to move the drone with "S" and "SS" during yaw turn and I often shown:

https://youtu.be/RF0wnQv3aAc?t=334

So it's just a matter of code and payload, ArduPilot can and should do.
I asked for a long time to have the possibility to turn on/off a motor through a switch for testing, upon request ignored... :-)

The payload is the necessary condition that must be satisfied , how much payload can a copter afford and ensure the controllability firstly when single rotor fails is crucial. as for the code i hope to see some
clear explanation too : )

[xiongmaoer]

I finally had time for making a new video. Now you can see clearly what happens after motor failure;

https://www.youtube.com/watch?v=ap8ngQqt7s8

Hey your work is amazing ! Have you written how you solve the problem down in a blog ? hope to see more information about it .