Inav: GPS Inconsistent behavior

[stronnag]

Spoiler : Somedays, adequate humber of satellites and reasonable HDOP doesn't meant nav functions will work as expected.

Today I flew 85fe245 and f440ead on my usually utterly reliable quad. This machine has not failed to execute PH, RTH or missions since we fixed SBAS months ago ---- until today.

Result. Completely random PH behaviour (if PH fails, then I don't even try any other nav function). Randomly, on hard reset, nav functions will either work or fail. Some log files at
http://seyrsnys.myzen.co.uk/inav_ph_woes/. All the log files were created in a short time frame with good satellite coverage (16-19) and good HDOP (1.1 - 1.3).

• LOG00279.TXT: 85fe245. PH fails miserably, repeated.
• LOG00280.TXT: f440ead (having persistent mag/acc makes checkout / build / flash / apply config in the field so easy:). Works OK.

... does some betaflight stuff on the minis for 30 minutes

• LOG00281.TXT: f440ead: nav functions fail miserably, repeatedly (multiple arm / disarms)
• LOG00282.TXT: f440ead: _powercycle_ nav functions work again !!!!
• LOG00283.TXT: f440ead: _powercycle_ nav functions still work. Out of lipos, otherwise I'd have tried 43eaf10 (which behaved perfectly across multiple lipos earlier in the week).

Note 0. LOG00281 Index 1, PH attempt 4 is a good example of 'fly off'

Note 1. 281-283 were executed in quick succession, it is unlikely that there was some environmental change that affected the results.
Note 2. If PH is not (obviously, visually working), it is rapidly aborted.

Exec Summary : nav post 43eaf10 seems pretty random to me at the moment. Other experiences solicited.

[digitalentity]

Seems like a GPS-related problem. GPS (and estimated navVel) velocity doesn't seem to be correlated with UAV attitude and stick input:

It might also be a compass issue:
At log 281 there is a moment when UAV is flying at only forward pitch - this is usually flying straight forward and GPS heading should be roughly equal to magnetic heading, however they are opposite:

However, I doubt it's compass - UAV heading strongly correlates with yaw stick input. I'd blame GPS reception that reported invalid coordinates/velocities/course for some reason.

@stronnag was there strong wind that day, if so - what direction?

[stronnag]

@digitalentity, thanks for the analysis.

There was less than 8 m/s wind; I often fly in more. My initial thought (from the field) was that it was just a bad GPS day. I guess that happens from time to time. I did not suspect the compass, much of random flying / drifting around was manually correlating the observed heading with the LTM reported heading (which seemed consistent), whereas the mwp calculated 'range from home' often seemed incongruous.

It was an object lesson in 'try PH before engaging more advanced nav functions'.

I'll do some more missions today before closing.

[digitalentity]

@stronnag
Interestingly GPS glitch detection did fire an alarm a few times. We definitely need better glitch detection logic.

[stronnag]

A day later .... updated the firmware to abf1015 (so no significant change). Starts with similarly less than stellar sat statistics (13-15 sats, 1.3-1.5 HDOP); couple of hours later its back to normal, 19-20 sats and 1.1 HDOP.

PH and other nav functions back to their awesome performance and consistency, even at the lower than normal end of the sat coverage range. Someone in the DoD really didn't like me yesterday.

Maybe we should use some of those spare X-FRAME bits for a glitch detection alert or counter?

[digitalentity]

@stronnag thanks for your report, indeed something must be wrong with GPS reception. We can use a byte in X-Frame to indicate INAV internal status flags, however we still don't have a good logic of detecting GPS failures...

[Linjieqiang]

The solution to the problem is using EKF algorithm although it looks messy and complex.

[digitalentity]

EKF is nothing close to a glitch detection and protection, it's merely an algorithm to blend data from available sensors.

[Linjieqiang]

If large difference between receiving from gps and predicting form EKF,program will move the data from gps.Maybe I'm wrong.

[digitalentity]

It's not the EKF itself, it's a supporting code logic that does the detection.

[Linjieqiang]

Oh.Sorry.It's my wrong.

[xdigyx]

Today I had a situation when at pos hold mode my hexacopter rapidly started to fly away. Not sure whether it was caused by loosing some sats or... Anyway maybe it would be a good idea either to take an average of 2-3 readings or ignore for pos hold mode position change grater than... (say 10cm/sample depending on model speed from the previous cycle/ significant sat.num loss)?

[stronnag]

GPS glitches are very rare, my report was a once a year occurrence, for a frequent flyer. Your problem is more likely symptomatic of mag interference. Have you verified that the mag works perfectly at all throttle settings?

[xdigyx]

Not sure what's telling you it's mag problem. If it was, then I believe the model would make slowly bigger and bigger circles, but it would not rapidly fly away. Do you agree?
Although I have used pos hold max for 1h, it happened only once till now, apart this time pos hold works great.
Unfortunately I had no BBox enabled...

[digitalentity]

Hard to diagnose w/o logs. Circles happen when heading is slightly wrong (up to maybe 30 deg) - this result in quad moving in slightly wrong direction. Bigger error will result in quad going in significantly wrong direction and a case of fly away.

[xdigyx]

There were no circles (for sure if it were then I would see it every time when using pos hold for a while). But there was just sudden pitch inclination increase by approx 30 deg. Now I know how to enable onboard dataflash bbox, so hope to catch it next time.

[digitalentity]

It's either a magnetic anomaly or compass failure, or real GPS glitch. If compass wiring is not good the sporadic bug in connectivity to compass chip may cause the chip to freeze and give out same heading over and over again. Please also check the wiring to the compass.

[digitalentity]

453

[xdigyx]

I am using onboard compass, so no wiring, can try to threat it with hot air but low chance. The FC is shielded (50x50mm 35um Cu PCB) from the bottom and grounded.

[DzikuVx]

@xdigyx shielding like that would not block magnetic field from power cables. It might shield oscillating magnetic field, but frequency would have to be > 100kHz.
Interference from power cables is almost not oscillating, grounded Cu PCB or even full Faraday cage is ineffective.
The only solution is to move power cables from battery/ESC further away from FC and (better option) use external magnetometer on a mast

[xdigyx]

Got your point, I was not thinking about this as the yaw heading reads were changing only 1-3deg depending on THR level.
Firstly I will try to catch the event with blackbox to see what's the root cause and maybe discuss what can be done to avoid such situation . Then I will use an external mag.

[stronnag]

@digitalentity. I no longer think that this is an external GPS issue.

• I've been flying this quad for months without nav issue; I have 100s of logs to prove it;
• The problems _coincidently_ only happened after major I/O changes late in the 1.2 cycle.

Today.

• Arm, PH fails miserable (no log alas, straight line 'fly away').
• Land. Power cycle. PH works LOG00292.TXT /1.
• Land. Disarm (still powered), fly a mission perfectly. LOG00292.TXT /2
• Land. Power cycle. PH Fails miserably (straight line 'fly away') LOG00293.TXT
• Land. Power cycle. PH works perfectly. LOG00294.TXT

(all logs at http://seyrsnys.myzen.co.uk/inav_ph_woes/). There was perhaps 30 seconds in the land / power-cycle sequence. I find it hard to believe the satellite performance is varying in that short time period. The whole sequence described above was within 7 minutes, with consistently between 17-19 satellites and c. 1.1 HDOP.

That I can have nav functions randomly work / fail on power cycle within very short time periods looks to me like a firmware issue rather than a celestial GPS issue. I'm encouraged in this theory by your recent CC3D sensor woes.

[martinbudden]

I'm certainly willing to believe this is a firmware problem. The IO changes were pervasive, and although I was very careful, there is certainly a possibility that I've introduced a bug somewhere.

[stronnag]

It's all circumstantial at the moment, but definitely a regression since 2016-7-30, the last flawless nav experience (with this hardware).

I should add that so far the Dodo has behaved OK, whilst the SPRF3 has not (same firmware). Tomorrow, it's the Dodo.

[xdigyx]

Today I did catch same issue. For sure there was some yaw drift, but it seems like the model was all time trying to face the starting position. After approx 2 min from start num of sats dropped to 0 just for one read cycle and my model rapidly moved.
I've got the log file. Some lines here:
time (us) GPS_fixType GPS_numSat GPS_altitude GPS_speed (m/s) GPS_ground_course GPS_hdop GPS_eph GPS_epv
105125924 2 16 123 0.06 181.4 129 88 132
105326384 2 16 123 0.06 181.4 129 88 132
105526783 0 0 123 0.07 181.4 9999 95 132
105727183 2 16 124 0.07 181.4 129 95 132
105927635 2 16 124 17.76 171.4 131 95 132
106128086 2 16 124 0.56 171.3 131 95 132
106328488 2 16 124 0.11 171.3 131 95 132
106528932 2 16 124 0.31 127.9 131 94 132
106729408 2 16 124 0.59 22.1 148 94 132
106929832 2 16 124 0.86 359.3 148 94 132
107130275 2 16 124 0.86 359.3 116 94 132
107330739 2 16 124 0.83 356.9 122 94 132
107531103 2 16 124 0.82 357.4 119 94 131
107535112 2 16 124 0.82 357.4 119 94 131
107731575 2 16 124 0.65 355.4 119 94 131
107931951 2 16 124 0.44 355.3 122 93 131
108132427 2 16 124 0.34 357.9 122 93 130
108332876 2 16 124 0.20 0.2 129 93 130

The full log file I can send on email to whom it may concern (pm me).

Just for clarification: I am using ver 1.1.

[digitalentity]

@stronnag, I agree, this is very likely software issue. However I'm
thinking it was there before the IO changes - I remember having odd mixer
issues a while ago before that major change happened.

Which direction was your machine facing on powering up? Maybe something in
firmware is messing up the mag...

[digitalentity]

This is going to be a tough one - changes to firmware will affect the
memory layout and may seemingly "fix" the bug...

[digitalentity]

@stronnag I have a strong feeling that something is wrong with either IMU of magnetometer code.

From your latest logs:
LOG 292
heading 31 - PH works

LOG 293
heading 22 - PH fails
heading 28 - PH fails

LOG 294
heading 200 - PH works

Interestingly in LOG292 and LOG293 machine does the same (correct) tilt to corrent for error however actual correction differes. This can only happen when heading is incorrect.

Can you send exactly the same hex/dump file you've been using during these tests so I can check it on my SPRF3 board?

[digitalentity]

@xdigyx you likely experienced an ordinary GPS glitch, might be GPS wiring as well.

[digitalentity]

I have a feeling that recent CC3D woes and this issue are related...

[xdigyx]

Yes, I agree. Checked the wiring and no loosen wires or connectors. Anyway, glitch was not detected and caused model to fly away, so just a proof that better glitch detection is still needed.
Thanks

[stronnag]

@digitalentity. I tend to do my PH tests with the craft 'beam on', so c. N / S heading is expected. If failed earlier on c. 200°, and I've had failures on other orientations. I (almost) always power up at 110° - 120° (facing away from 'my' bench), which is my first mag pre-flight check.

The hex (gcc 6.1 compiled) is at http://seyrsnys.myzen.co.uk/inav_ph_woes/inav_1.2.0_SPRACINGF3_8cfc74b.hex

From my records (I keep all my logs), it seems https://github.com/iNavFlight/inav/commit/43eaf10db2170633cf424baa71a6bf0082c8061f was the last non-affected build; I will fly that today.

[oleost]

Maybe an announcement on forum regarding this?

[stronnag]

Yes, I will do one immediately.

.... done

[digitalentity]

@stronnag, @oleost yes, thats a good idea, thanks!

@stronnag can you also provide a dump so I'll be testing exactly the same fimrware setup?

[stronnag]

cli dump http://seyrsnys.myzen.co.uk/inav_ph_woes/nav_sprf3_fp.txt

[digitalentity]

@stronnag thanks. Now I should try to find a GPS module...

EDIT: In the following days I'll try to reproduce the issue on my test-quad.

[digitalentity]

From my records (I keep all my logs), it seems 43eaf10 was the last non-affected build; I will fly that today.

@stronnag it would be awesome if you could pinpoint the commit that started to glitch. I suspect that this particular issue might be related to a buffer overflow or maybe a race condition in some interrupt handler since it's intermittent. I suspect UART - if an interrupt handler somehow gets invoked before buffers are properly initialised it might result in overwriting arbitrary memory (please correct me if i'm wrong).

[stronnag]

I'm going out this afternoon armed with all my big quad LIPOs and

inav_1.2.0_SPRACINGF3_43eaf10.hex
inav_1.2.0_SPRACINGF3_4bbd176.hex
inav_1.2.0_SPRACINGF3_8cfc74b.hex
inav_1.2.0_SPRACINGF3_f440ead.hex

I expect 43eaf10 to work and f440ead not to work, based on previous experience.

The great thing about having mag and acc saved in the dump file is that reflashing / return to known config in the field is so, so easy.

[digitalentity]

@stronnag thank you so much for testing this out! I'm struggling to figure out what's going on with CC3D - it's very repeatable glitch as I have a feeling it's related to GPS issues.

[stronnag]

27822a5 is the last reliable commit. 6+ power cycle / PH successful attempts. All later builds will fail PH with 2 attempts. more details & logs when I get home.

[stronnag]

Here's the full bisect test results.
Methodology:

1. stm32flash the required firmware
2. use mwptools cf-cli tool to install known configuration
3. Perform PH
4. Land (auto if PH works)
5. Powercycle
6. repeat step 3 until PH FAIL or 6 consecuitve PASS

The results being:

| Commit | Date | Result |
| --- | --- | --- |
| 43eaf10 | 2016-08-02 | FAIL : LOG0295.TXT |
| 742f429 | 2016-07-31 | PASS : LOG0304.TXT |
| 742f429 | | FAIL : LOG0305.TXT |
| 0489eb8 | 2016-07-31 | PASS : LOG0296.TXT |
| 0489eb8 | | FAIL : LOG0297.TXT |
| 27822a5 | 2016-07-29 | PASS : LOG0298.TXT |
| 27822a5 | | PASS : LOG0299.TXT |
| 27822a5 | | PASS : LOG0300.TXT |
| 27822a5 | | PASS : LOG0301.TXT |
| 27822a5 | | PASS : LOG0302.TXT |
| 27822a5 | | PASS : LOG0303.TXT |

Several successful WP and RTH mission were subsequently flown with 27822a5.

All log files under http://seyrsnys.myzen.co.uk/inav_ph_woes/

I trust this is conclusive. It also supports the tentative conclusions relating this fault to the CC3D fault and the new IO subsystem.

[xdigyx]

stronnag, I have just compared my dump and yrs settings and noted one strange value:
set mag_declination = -130, should it not be approx 110 POSITIVE?

[stronnag]

Not if (a) one habitually flies in the New Forest area of southern England, where the declination is c. 1° 30' W and (b) you have set inav_auto_mag_decl = ON set.

May I also suggest that this thread is for trying to fix a serious regression in iNav, not provide basic support to beginners; please use the RC Groups topic http://www.rcgroups.com/forums/showthread.php?t=2495732 for basic support.

[digitalentity]

@stronnag thanks for narrowing this down. There still is a possibility that in 27822a5 and earlier this bug is still present - there was a bug reported with frozen servos on SPRF3 which I also suspect to be a result of memory corruption. Coincidentally that bug also happened after beginning migration to new IO.

@martinbudden, @ledvinap if you have the time please have a look at the code to see if there's anything wrong with new IO code review in INAV.

[digitalentity]

We are lacking any protection from wrong indexes, incorrect pointers etc. I think we should start putting ASSERT(x) all over the code with three possible compile-time options:

1. ignore (tested production code)
2. record a line to a given location in memory
3. same as 2 but with additional hardfault

Thoughs?

[digitalentity]

For one:

ioRec_t* IO_Rec(IO_t io)
{
    return io;
}

this assumes io is correct but what if it's not? then pointer to non-existent ioRec will be returned with a possibility of writing owner, resource and index to arbitrary memory localtion causing corruption of data.

Adding the following will help protect from this:

ASSERT(io);
ASSERT(io >= &ioRecs[0]);
ASSERT(io < &ioRecs[DEFIO_IO_USED_COUNT]);

[xdigyx]

Do not be so rude, I just wanted to help. Sometimes even basic things can be overlooked by pro users... 110 or 130, but I believe it should be positive (unless FW does not look at it at all and it really does not matter).

EDIT:
checked again the location. it's negative....

[digitalentity]

@xdigyx I think @stronnag didn't ment to be rude, mag declination indeed varies greatly between different places. However this issue is related to latest 1.2 code and tests on 1.1 are probably not relevant, but they create noise that destract attention from the topic.

[sambas]

Tested current master (8cfc74b) with revo. Didn't have any notisable issues. 5 short flights (PH spin+ RTH) with power cycle between. Only three fitted in the bb log. Build with gcc-arm-none-eabi-4_9-2015q2-20150609-win32 using command "make -j24 TARGET=REVO OPBL=yes". Included map file if you could provide same with gcc 6 to compare.
https://www.dropbox.com/sh/tucne4xbeanyqax/AABw0wqCcqdb0lyaBVCyD7Aha?dl=0

[digitalentity]

I don't have any GPS-related issues on my Y6 built with GCC-4.9.3 as well, although CC3D won't work at all...

[stronnag]

Strangely, I don't (think I) have this issue on my Dodo either --- I will test it rigorously later today (the problem manifests itself on an acro SPRF3 with external I2C BMP280 + M8N (I2C compass)).

@sambas : I've uploaded gcc 6.1 maps for SPRF3, REVO, NAZE and CC3D to http://seyrsnys.myzen.co.uk/inav_ph_woes/

[xdigyx]

Comming back to the topic of GPS glitch...

In my case there were a difference in position of 4.5m between one and following GPS pos reading after a glitch, but this was not caught as a glitch.
I am not an expert, but just a simply thought:
At the moment prediction of location is a radius of a constant value (INAV_GPS_GLITCH_RADIUS + 0.5f * INAV_GPS_GLITCH_ACCEL * dT * dT) this probably is enough for nav, but for PH the value is quite huge.
Maybe (INAV_GPS_GLITCH_RADIUS * INAV_GPS_GLITCH_ACCEL * dT * dT) would be a better estimation?

[martinbudden]

Looking at the work @stronnag has done:
https://github.com/iNavFlight/inav/commit/27822a59cb849266427084508146fab425d6a037 was the last reliable commit. The next significant commit after that was https://github.com/iNavFlight/inav/commit/0489eb8b0840719277450a8d15bb6f1d5e6f62be - Converted I2C, EXTI and SONAR to new IO, which didn't work.

However there was a problem with that commit in that I forgot to add EXTIInit in main.c, which I corrected in a subsequent PR. So, unfortunately narrowing it down to that PR does not really help us - it didn't work because of an unrelated issue.

@digitalentity , I think there is some merit in adding asserts, but I think that it's unlikely the problem is in the driver code, since no-one has reported a similar bug in betaflight. On the other hand, perhaps no-one has built betaflight using 4.9.3. Can you try building betaflight with 4.9.3 and see if the CC3D issue occurs - if it does then we know there is a problem with the underlying code, rather than the port.

The device drivers are virtually the same between betaflight and iNav, so my thoughts are that it could be a problem with initialisation or in the sensor code.

Also, can you try building PR https://github.com/iNavFlight/inav/pull/456 with 4.9.3 and seeing if the problem exists there - this PR is further convergence of the device drivers between iNav and betaflight.

[digitalentity]

@martinbudden the problem is that the CC3D bug goes away even if I add debug[0] = gyroADC[0] line in the gyro.c, so that PR seemingly might fix the issue without actually doing so.

[martinbudden]

Yes, you're right. We can't say this bug is fixed until we actually understand what the problem is.

[stronnag]

@martinbudden. No build after https://github.com/iNavFlight/inav/commit/27822a59cb849266427084508146fab425d6a037 works consistently on my previously super-reliable quad (SPRF3 / BMP280/ M8N), including builds that fixed EXTIInit. The failing logs from Friday https://github.com/iNavFlight/inav/issues/431#issuecomment-239489749 were from 8cfc74b (current master).

However, it's deeper than that. Today I tested the F3/Dodo/M8N/TRI on 8cfc74b; of course that works perfectly, including my arbitrary "6 strikes till you're out'" power cycle / PH test.

To my simple mind,it appears something post 27822a5 causes random initialisation failures on some hardware. Not a lot of help in narrowing the problem down. I appreciate there have been no similar reports on Betaflight; this does not really surprise me, as the use cases (and use of sensors) are quite different. I fly βF a lot; just not with many sensors enabled.

[stronnag]

It appears to me that the new IO subsystem is less tolerant of I2C daisy chaining than the previous implementation

| Configuration | Average failure of PH post powercycle |
| --- | --- |
| Dodo FC. BMP280 on FC, HMC5883 on GPS via I2C | No failures |
| SPRF3 Acro. BMP180 + HMC5883 on external board, via I2C | 1 in 4 attempts fails |
| SPRF3 Acro. BMP280 on I2C, daisy chained to HMC5883 on GPS | 1 in 2 attempts fails |

i.e. the I2C with two daisy chained items is the worst performer with 8cfc74b; all configurations perform equally well (no PH failures) with 27822a5.

The latter two examples are the same machine, reconfigured. The BMP280 configuration (worst case) was the subject off the previous tests.

And if the explanations are not clear, here's some confusing ascii art ....

+-----------------+
|  Dodo FC        |
|  +--------+     |
|  | BMP280 |     |     I2C               +------+
|  +--------+     |-----------------------| 5883 |
|                 |                       +------+
+-----------------+                     on serial GPS

 +---------------+
 |               |                     external board
 |               |                       +--------+
 |   SPRF3       |      I2C              | BMP180 |
 |               |-----------------------| 5883   |
 |               |                       +--------+
 +---------------+

 +---------------+
 |               |
 |               |      I2C     +--------+         +------+
 |  SPRF3        |--------------| BMP280 |---------| 5883 |
 |               |              +--------+         +------+
 |               |                               on serial GPS
 +---------------+

And inav_auto_mag_decl on or off makes no difference (just in case anyone suspected a bogus auto declination value).

[digitalentity]

@stronnag interesting finding. I now remember solving I2C overclock issues when doing some I2C-related changes in INAV quite a while ago. Since now we have I2C drivers from Betaflight we might be hitting the overclock issue again.

The fact that problem occurs more frequently with longer wires only confirms the thory.

I'll hook up my logic analyser to SPRF3 and verify today.

[digitalentity]

I think I've found something.

With new IO drivers SCL and SDA pins are initialised as IOCFG_AF_OD. On both F1 and F3 boards this result in GPIO port output speed register being set to zero (2MHz clock). On longer lines this may cause unacceptable rise/fall times as seen by CPU which in turn may cause data corruption.
I'm not sure how this affect sensor readings and why I didn't notice anything significant in the logs, but prioir to new IO I2C pins were clocked at 50MHz.

465 intends to fix the issue.

[digitalentity]

@stronnag can you verify the latest master on your problematic setup?

[stronnag]

I'll test this evening.

[digitalentity]

Thanks!

[digitalentity]

Ok, it's not clock frequency, it's controlled rise and fall time.
2MHz mode (default) - 125ns @ 50pF load
50MHz mode (new) - 20ns @ 50pF load

i2c bus is very sensitive to rising/falling edge quality, so when working with long line things get even worse. With nothing connected to external i2c (using only onboard sensors), even 2MHz mode is probably fine.

[stronnag]

465 makes no difference, I still get c. 50% PH failure for the SPRF3 / BMP280 / GPS mounted 5883.

[digitalentity]

Damn! I wonder if it's I2C communication or BMP280 baro driver causing havoc in memory?

[digitalentity]

@stronnag let's try to narrow out the problem. I've created a branch gps-failure-debugging that will log into navDebug the following values:
0 - PosHold target acceleration in North direction
1 - PosHold target acceleration in East direction
2 - NAV rcAdjustment[ROLL]
3 - NAV rcAdjustment[PITCH]

Let's check if UAV heading known to Navigation subsystem is correct.
If it is - then we have a problem that leads to incorrect heading when airborne.
It it is not - then something corrupts NAV internals and forces it to believe that UAV is facing in wrong direction.

Please try to catch the failed PH on the logs with the debug enabled. Also, please after PH fails, don't land and disarm immediately, make a fly-by using only pitch forward (no roll, no yaw) to see if there is strong correlation between mag heading and GPS CoG.

Sorry for putting this task on you, I'm unable to reproduce the issue - PH/RTH is rock solid on my SPRF3 equipped quad (Full board less mag + 5883 on GPS)

[stronnag]

I almost thought you had provoked a heisenbug --- the first five attempts at PH / power cycle were succcessful.

Then it failed at 6 (and 7 after soft reboot, OK at 8 again soft reboot). I've found that a soft reboot (via MSP) will usually clear the condition.

There are a load of logs at http://seyrsnys.myzen.co.uk/inav_ph_woes/2016-08-16/

| File | Image | Comment |
| --- | --- | --- |
| LOG00323.TXT | LOG00323.png | Perfect - how dare it ! |
| LOG00324.TXT | LOG00324.png | Again perfect |
| LOG00325.TXT | LOG00325.png | Perfect |
| LOG00326.TXT | LOG00326.png | Perfect -- why |
| LOG00327.TXT | LOG00327.png | Perfect. Begining to doubt my sanity |
| LOG00328.TXT/1 | LOG00328-idx_1-FAIL.png | Hurray! Fails |
| LOG00328.TXT/2 | LOG00328-idx_2-FAIL.png | Fails after soft reboot |
| LOG00328.TXT/3 | LOG00328-idx_3.png | Perfect again (soft reboot) |

Hope this is useful. The images may help assess 'straight' flight.

My Dodo (on board bmp280) + 5883 on GPS is also rock solid, as I hope will be the EVO that arrived today.

[digitalentity]

@stronnag thanks! Preliminary results:

1. Navigation does calculate required acceleration in N-E axis correctly as well as roll/pitch (with respect to known heading). What a relief!
2. Executing the calculated roll/pitch however yields incorrect motion.
3. In failed PH attempts accelerometer seem to measure more articulated motion

In successful PH attempts GPS seem to return heading data that is correlated with heading as estimated by IMU:

However in failed PH situation is different. This part is particularly interesting:

There is no yaw rotation involved and there is only forward pitch, but GPS speed drop from 6m/s to zero, accelerates in the opposite direction to 9m/s, drops back to zero and again accelerated in the initial direction to 8m/s.
This happens with only forward pitch involved and GPS heading changes direction twice which is a highly unlikely possibility.

[digitalentity]

Similarly here:

We clearly see deceleration and acceleration in the opposite direction and again - deceleration from 6m/s to 2m/s in under 2 seconds while pitch andle doesn't change, there was no yaw rotation but GPS heading changed by 150deg (almost opposite direction).

[digitalentity]

I wonder how different are the GPS modules on your three setups described in https://github.com/iNavFlight/inav/issues/431#issuecomment-239684466 ? I have a feeling that it's not the same GPS module and all failed attempts happened only on one particular machine with one particular GPS unit.

[stronnag]

Same gps for all successful and failed logs in this thread beitian m8n. Same machine just swapped out baro & mag for a set of tests. The rock solid dodo + tricopter is a similar banggood m8n. I can swap out the beitian for an older neo6m if that would help. Strange how sometimes it works though.

[stronnag]

I.e. there is only one machine with sprf3, and its gps has not been changed.

[digitalentity]

Hm... This is odd. I wonder what is wrong - GPS heading or compass heading. I wonder what's different in the code. Just to rule out the GCC issue, can you compile with 4.9.3 and re-test? I can send you a compiled hex for SPRF3 if you need.

[stronnag]

I have 4.9.3. That was my next move anyway.

[stronnag]

I tried with 4.9.3. No difference. On average 1 in 3 PH / power cycles results in failure. It's not a compiler issues.

I also tried #456 on top of master. This was unflyable, so no data on that (detail to follow in #456)

[digitalentity]

@stronnag can you remove this line and re-test please:
https://github.com/iNavFlight/inav/blob/master/src/main/drivers/bus_i2c_stm32f30x.c#L118

The main different thing between 27822a5 (good) and 8cfc74b (bad) is I2C clock stretching that changed in F3 I2C driver. I suspect it might be the cause.

[digitalentity]

I2C clock stretching by itself is not a bad thing, but I'm afraid it's casing i2c timeouts if slave holds the clock for a long time - and if it happens in the middle of the transaction it may cause the sensor to be read only partially. Not sure how it affects heading in your case, just an idea.

[stronnag]

Removing I2C clock stretching makes no difference, PH continues to fail (5b8b978).

I then applied #456 on top of master (with I2C Clock Stretching still removed). Same (PH fails).

Just to verify my sanity, I then flashed 27822a5 --- perfect.

[digitalentity]

I'm going to attempt two options:

1. Carefully review the 0489eb8b0840719277450a8d15bb6f1d5e6f62be - apparently that's the commit when things started to fail. It's fairly large though so I would appreciate any help
2. Merging in #476 and start adding ASSERTS everywhere in the code in attempt to find something that causes corruption/invalid results.

[digitalentity]

I have a feeling that the bug has been lurking in the code for long time, latest changes only provoked it - I don't see any inconsistency in the logs except for the fact that GPS velocity and course is sometimes uncorrelated with UAV tilt angles.

[martinbudden]

@digitalentity , I've read through that commit at least twice and haven't been able to see any issues in it. I'll look through it again, but another pair of eyes may be able to spot something I've missed.

I think you are probably right though - we've probably provoked a bug that has been lurking.

[digitalentity]

I've went through the relevant parts of the commit - EXTI, I2C, MPU6050 and HMC5883 code - don't see anything that might be causing this. My thoughs on this one:

1. The bug is triggered by some external event otherwise we would be seing it each reboot (like with recent NAZE and CC3D issues)
2. Based on experience, it's either uninitialised variable/array, memory corruption/buffer overflow or a race condition. All three are extremely hard to pinpoint.
3. There is an extremely small possibility but non-zero possibility that issue is completely external to firmware and all success/failure tests are a mere coincidence, but I definitely wouldn't bet on that.

I'll probably start scattering ASSERTs all over the code (they don't hurt as they compile to nothing when disabled)

[digitalentity]

@stronnag next test is for N-E acceleration and GPS N-E velocity, let's check if they correlate. Can you record a failed PH with d8727a7?

EDIT: If possible please keep quad oriented North for easier log analysis.

[stronnag]

Sure. May take a day or two looking at the weather forecast for the area.

[digitalentity]

Thanks! I'll also make a few more commits in the following days recording data to give at least a hint what's going on there. So far I don't even know what am I looking for :disappointed:

[digitalentity]

Here is another case of what appears to be this issue on an airplane and a NAZE32 - https://www.youtube.com/watch?v=AyCgWYTo_SI
EDIT: Looks like the guy was on INAV-1.1, so NOT this issue...

I'm inclining to revert all IO-related commits as they are apparently causing trouble in NAV modes.

[stronnag]

+1

My preference would be to revert and get 1.2 out, then we can work this on a separate branch

[martinbudden]

Well, it would be a bit of a pity to revert, but sometimes you need to.

If you want to revert and get out a 1.2 release, I'm OK with that.

Have you seen @ledvinap 's comments at https://github.com/iNavFlight/inav/commit/0489eb8b0840719277450a8d15bb6f1d5e6f62be#commitcomment-18700695 - it might be good to try those out before a revert though.

[digitalentity]

@martinbudden reverting would be one hell of a challange as there were lots of changes besides IO and IO is not one commit to revert.

I would like to avoid reverting such a big piece of work - it would also mean reverting F4 support which I am not a fan of at all, but if we would not be able to fix this issue in next few week I would have no choice but to revert IO and postpone it for 1.3.

[digitalentity]

Looking through @ledvinap 's comments at the moment.

[oleost]

That video is showing a 1.1 inav release.

1. aug. 2016 2.51 p.m. skrev "Konstantin Sharlaimov" <
   [email protected]>:

Looking through @ledvinap https://github.com/ledvinap 's comments at
the moment.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/iNavFlight/inav/issues/431#issuecomment-241198522,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFaHNpb0ZH5S2r6nWQo4nEBl5UfnuTcUks5qhvhggaJpZM4JeUXk
.

[martinbudden]

Yes, reverting would be a big piece of work. I didn't mean a literal revert, more like branching the code at the know last good release and selectively applying commits from that point. But even that would be a large piece of work.

So solving the underlying problem would be a much better way forward.

@ledvinap - do you have any time to help on this?

[digitalentity]

@stronnag master branch is now logging for acceleration and gps-failure-debugging branch logs GPS pos and vel (after applying local coordinate transformation). Can you record a faild flight with this one? I'm still trying to debug :smile:

[stronnag]

Just as soon as the wind drops (c. 20 m/s / 40 knots) atm. Which branch? (can do both, I'm getting quite adept at in field patching, building, flashing, reconfig ----- as well as sub 30s PH flights. Certainly won't miss the latter once we get this cracked.

[digitalentity]

_gps-failure-debugging_ is the branch that will record some information that might prove very useful. Thanks!

[wilco1967]

whoah.... That guy is quite negative..... don't let that video put you
off....

We all know this is a work in progress, and things might not always go
perfect 'out of the box'....
But this is not a commercial product, and doen't come with a warranty.

I REALLY appreciate all the efforts that go into iNav.... You will have to
tinker with it... those who are not willing to do so should go out and buy
a DJI inspire or similar, and pay huge cash.....

Testing is part of the Hobby, and yes, it might crash in the process... so
be it.....

Thanks again !

Groeten,

Wilco

On 20 August 2016 at 14:56, Martin Budden [email protected] wrote:

Yes, reverting would be a big piece of work. I didn't mean a literal
revert, more like branching the code at the know last good release and
selectively applying commits from that point. But even that would be a
large piece of work.

So solving the underlying problem would be a much better way forward.

@ledvinap https://github.com/ledvinap - do you have any time to help on
this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/iNavFlight/inav/issues/431#issuecomment-241198745,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEFaZ0tHZpJhb_cAxfX6YDViwm93IpiYks5qhvmHgaJpZM4JeUXk
.

[digitalentity]

As long as I'm flying this myself nothing will put me off from continuing the development. However what really pisses me off is providing negative reviewes without providing the feedback to devs - that's definitely not a way to make firmware better.

One of my neighbour flyers sais - if you are not ready to crash your UAV to bits - keep it on the ground.

[wilco1967]

One of my neighbour flyers sais - if you are not ready to crash your UAV to bits - keep it on the ground.

Well said.....

[ledvinap]

@martinbudden: There may be some bug in IO code, but I am quite skeptical it is primary cause of this problem.

If it is caused by memory overwrite, changing memory layout can change it's behavior - reordering files in makefile or adding dummy arrays around navigation data structures.

[martinbudden]

@ledvinap , agreed that a bug in the IO code is probably not the primary cause of the problem.

@digitalentity , I have spotted something that is wrong and seems relevant, but can't seem to see how it causes a problem. Nevertheless it's worth looking at.

In the SPRACINGF3 target we have:

#define EXTI_CALLBACK_HANDLER_COUNT 2 // MPU data ready and MAG data ready

However sonar_hcsr04.c also has an EXTI callback, so EXTI_CALLBACK_HANDLER_COUNT should be 3. So it certainly seems this could cause an array out of bounds problem, however I cannot see where in the code. I'll keep looking...

[digitalentity]

@ledvinap i also agree, IO code itself like likely not responsible for the glitch. IMO it's either a long lurking bug that was made evident or was introduced during migrating to new IO.

@stronnag as I can understand, rebooting the FC is the only thing that temporary cures the issue (please correct me if I'm wrong). It's possible that recent code somehow causes GPS misconfiguration. Can you make a test - right after successful PH test (your 6 consecutive attempts) bring the quad into gps passthrough mode and take a Ublox config dump, and also do the same after a failed PH so we can compare and confirm if this is indeed the issue.

@martinbudden as far as I can tell EXTI_CALLBACK_HANDLER_COUNT is only used in registerExtiCallbackHandler mechanism which is no longer used so it couldn't cause problems.

[martinbudden]

@digitalentity , you're right EXTI_CALLBACK_HANDLER_COUNT is unused, I thought I might have been onto something there.

One other thought: removing *(SORT_BY_ALIGNMENT(.bss*)) from stm32_flash.ld might make debugging this easier.

[martinbudden]

@stronnag , do you have the facility to use a NMEA based GPS? If you could try that we might be able to isolate the problem to the UBLOX code.

[digitalentity]

One other thought: removing _(SORT_BY_ALIGNMENT(.bss_)) from stm32_flash.ld might make debugging this easier.

Inded. If it will remove @stronnag's GPS bug then we will know for sure it's the memory corruption and will also have a hint where to look by comparing memory layouts of the two builds (although map file doesn't always show variables declared static

[stronnag]

@martinbudden: Yes. Just by setting the UBLOX into NMEA mode.

@digitalentity : Soft reboot can indeed clear the problem, however if capturing 'Ublox config dump' involves running a Windows application (u-center?), my ARM / Arch Linux chromebook is not able to support doing that.

[digitalentity]

@stronnag maybe just configure Ublox GPS once, store config into NVRAM and disable autoconfig in INAV? Lets try to rule autoconfig out.

[martinbudden]

@stronnag , when you next test this out, can you also test NMEA mode as well then?

[stronnag]

Although #define GPS_PROTO_UBLOX_NEO7PLUS is generally not defined, it is defined for ANYFC and REVO.

If you have UBLOX 3.01 firmware, this returns ** 220 ** bytes; our ublox buffer is 100 bytes. Ouch.

[digitalentity]

We should drop the remainder of the response.

[digitalentity]

I've seen GPS modules (Chinese clones) that report version > 7, but do not support PVT which will render those modules unusable if GPS_PROTO_UBLOX_NEO7PLUS is defined.

[martinbudden]

@stronnag , good spot. Although it is not good, it shouldn't actually break anything, since buffer is protected from overrun. In gps_ublox.c:

            if (_payload_length > MAX_UBLOX_PAYLOAD_SIZE ) {
                // we can't receive the whole packet, just log the error and start searching for the next packet.
                gpsStats.errors++;
                _step = 0;
                break;
            }

[stronnag]

btw, I do not have GPS_PROTO_UBLOX_NEO7PLUS defined ...

[martinbudden]

Yes, I assumed that.

[martinbudden]

We should probably increase that buffer size anyway.

[digitalentity]

We need to make GPS_PROTO_UBLOX_NEO7PLUS consistent and increase the buffer size. 256 bytes should be safe enough.

[digitalentity]

See #487

[stronnag]

On the SPRF3 quad (+external I2C BMP280 + GPS 5883 mag)

I flew gps-failure-debugging (d8727a7) and failed miserably (mea-culpa) to collect valid logs, usual pattern of failure (about 1 in 3 is a fail). I'll try it again tomorrow.

I also flew master (59e9ac3). Same results (about 1 in 3 is a fail). Sample fail log
http://seyrsnys.myzen.co.uk/inav_ph_woes/LOG00369.TXT

On the tricopter, SPRF3EVO + I2C GPS 5883 mag)

Perfect PH. (Note: the EVO was in place of the equally perfect Dodo).

So I expect I could (personally,selfishly) resolve this issue by replacing the quad FC with now spare Dodo; however I don't intend on doing that until we get to the bottom of this mystery.

[martinbudden]

@stronnag , did you try NMEA?

[stronnag]

Sorry, completely forgot. I'll try again tomorrow.

I have attached a 'postit' note to the quad with NMEA scrawled on it. That'll work.

[stronnag]

In spite of the strongly worded postit note, setting NMEA in config, disabling gps_auto*, the GPS fix led illuminated and having valid NMEA sentences in gpspassthrough mode, iNav stoically refuses to acknowledge even the existence of a GPS in NMEA mode.

So I guess I won't be testing that option.

[martinbudden]

Well, it's useful to know it doesn't work. Does NMEA work on your SPRF3EVO or Dodo setups?

[stronnag]

It does not work on the EVO (the dodo is in the spares box, inching towards the quad). Not surprising, as they are similar GPS units. NMEA appears to be non-functional, despite showing valid NMEA in gpspassthrough.

The commonality between the working cases is simplicity of external I2C (mag v. mag + baro). If it wasn't for the flawless performance up to and including https://github.com/iNavFlight/inav/commit/27822a59cb849266427084508146fab425d6a037 I'd just write it off to hardware, replace the SPRF3 with the dodo and be done with it. I have a few non-nav βF minis with F1 that would fight over the SPRF3.

[martinbudden]

NMEA is not initialised in the same way as UBLOX, see https://github.com/iNavFlight/inav/issues/494 - this could be why NMEA is non-functional.

[digitalentity]

Just checked, NMEA appears to be functional on CC3D. It's something else that prevents it from working correctly in your case. Can you share your dump so I could try reproducing the issue on my EVO?

[digitalentity]

Hm...
NMEA works on CC3D + Ublox7
NMEA doens't work on EVO + M8N
More to investigate.

[digitalentity]

It looks like M8N GPS by default doesn't send out GPGGA and GPRMC sentences. Instead it sends out GNGGA and GNRMC for some reason. NMEA driver needs to be adjusted to use them as well.

[digitalentity]

2431441 fixes the NMEA on M8N

[stronnag]

I tried three different firmwares this evening, with typical results.

LOG0370.TXT/1 : index 1 : 2431441 "gps-fixes" in NMEA mode. OK NMEA works, but PH fails in the usual way.
LOG0370.TXT/2 : index 2 : 2431441 "gps-fixes" in UBLOX mode. After reboot, PH OK.

LOG0371.TXT : 2431441 "gps-fixes" in UBLOX mode. After power cycle, PH Fail.

So the "gpx-fixes" branch fixes NMEA on M8N, but appears to not to fix the real problem.

I next tried d8727a7, the "gps-failure-debugging" branch. This generates 1 byte sized logs, so it not helpful (explains yesterdays lack of logs too). One PH OK, one PH fail, so standard behaviour.

Next up, "master" (50fa119), just because I can.
LOG0375.TXT/1 : index 1 : PH Fail
LOG0375.TXT/2 : index 2 : PH OK after reboot.
LOG0375.TXT/3 : index 3 : Fly a WP mission, because it'll keep working between power cycles.

In summary, we have coincidentally fixed NMEA mode on M8N, but the underlying problem remains, and does not appear to be related to GPS protocol.

[martinbudden]

Well, I've been looking through the code some more and I have noticed a difference between the SPRACINGF3, SPRACINGF3EVO and RMDO boards. SPRACINGF3 is the only one with MAG_INT_EXTI defined in target.h. Which means the other two targets are getting their compass readings via polling rather than an interrupt.

I think it's worth removing this #define from the target.h file and seeing if that makes a difference - even if it is only to eliminate it.

[martinbudden]

@stronnag , when you next get a change, can you try https://github.com/iNavFlight/inav/pull/499 - see if that helps

[stronnag]

Yes. It will be tomorrow now, due to rather stressful commute.

[martinbudden]

Great, thanks

[digitalentity]

@stronnag I have a wild guess - can you try disabling a blackbox feature and see if the PH works? navDebug variables shouldn't affect logging - there's something else going on in memory. Let's try narrowing it down to a subsystem. Don't compile out, just disable features.

[martinbudden]

@digitalentity , I think we should also look at stack size again. Changing the stack size did not solve the CC3D problem, but we found out that was due and ioConfigurations[] array bounds problem. So there might still be a stack size issue. The stack is only 1K and we are not short of RAM, so we could easily change it to 2K. [I'm also looking at writing some stack watermarking code, but that will be a little while.]

[digitalentity]

Stack watermarking code will definitely help. Let's try increasing to 4K on all targets and see if it solves the issue.

[digitalentity]

@martinbudden please also add calls to ASSERT() where you see fit.

[digitalentity]

See #503

[stronnag]

Flew 2524736 (i.e. #499). No difference (usual 1:2 / 1:3 PH fail).
Turned off blackbox. No difference (usual 1:2 / 1:3 PH fail).
Applied/ flashed /configured #503 (4k stack). No difference (usual 1:2 / 1:3 PH fail) [bbox back on]
As a sanity check, back to 27822a5; perfect over 5 attemps (PH, RTH, WP).

I'm more than happy to continue testing new builds in an effort to track this down; I always have a hardware fix (spare dodo) should we draw stumps on it.

[martinbudden]

@stronnag , thanks very much for doing this testing. You efforts are much appreciated.

[hydra]

@stronnag if you or anyone else needs a spare SPRF3 ACRO or DELUXE board to help with this issue let me know as I have few repaired customer returns here that need a good home.

[stronnag]

@hydra, swapping out the broad would at least eliminate whether the existing ACRO board has marginal I2C. You should have have my address from the recent EVO order 000001173 (nice bit of kit, works really well for iNav).

[hydra]

@stronnag ok, i'll get one shipped out to you.

[martinbudden]

@hydra, yes could you send me one as well. That would mean I could test some of my proposed fixes myself and not have to rely entirely on @stronnag 's generosity.

[digitalentity]

@hydra thanks for dropping in!
@stronnag it would be great to rule out marginal hardware issue, but still zero sized blackbox logs is something that is not normal and I have a feeling it's related to flacky GPS performance.

[hydra]

@martinbudden of course! email me your address this afternoon and I'll get one in the post along with @stronnag 's one tomorrow. Indicate in the email if you need a MINI and/or EVO too please.

[martinbudden]

@stronnag ,

Could you try out PR https://github.com/iNavFlight/inav/pull/509 ?

This attacks the problem from the other end. It puts in the missing EXTI init into the first PR that added the new EXTI handling.

[stronnag]

Sure. I'll try it tomorrow, weather permitting (which It should be).

[stronnag]

or maybe not. This patch does not apply to master, which (a) has the changes to main.c, albeit with a large offset, and (b) none of the changes to SPRACINGF3/target.h. So just the SPRACINGF3/target.h changes, which I can do locally?

[martinbudden]

This is a branch from wayback. Perhaps the easiest thing is for me to attach a .hex file?

[stronnag]

No, I prefer not to fly random hex files. I can clone your branch.

[martinbudden]

OK

On 26 August 2016 at 19:59, stronnag [email protected] wrote:

No, I prefer not to fly random hex files. I can clone your branch.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/iNavFlight/inav/issues/431#issuecomment-242822499,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAL4GukkzVgCPs3ZEX40MWryWV0hqR62ks5qjzeZgaJpZM4JeUXk
.

[digitalentity]

@stronnag, if possible please also try disabling everything not absolutely
necessary for flight - blackbox, telemetry. You are using a ppm receiver
right? Worth looking if we have a problem in that part of code.

[stronnag]

Disabled blackbox and telemetry : PH still fails 50% of the time.
Disabled blackbox and MSP on the telemetry device : PH still fails 50% of the time.

This machine and the ex-dodo / now EVO machine are both PPM.

[martinbudden]

@digitalentity , @stronnag , I've found a bug in the handling of the compass read function. Namely the return value of the read function is ignored. So if the read failed potentially erroneous mag data values could be used.

This could possibly be causing our problems, the erroneous reads will effect the hmc5883lInit function and could be causing problems in the updateCompass function.

See https://github.com/iNavFlight/inav/pull/511 for the fix.

The code now is certainly wrong, but whether it causes our problem is another matter...

Anyway, @stronnag , could you test when you next fly?

[stronnag]

Damn it! I've just rebuilt the troublesome quad with the spare dodo to eliminate the GPS/mag as the problem source (which I will try tomorrow), so it will probably be Monday, unless I can escape a three line whip on supervising my wife whilst operating power tools.

[martinbudden]

Well, it will be good to eliminate the GPS/mag as the problem source. I think I've found a genuine bug, but whether it is our bug is another matter. One of the useful side effects of this investigation is that we have found and fixed some other problems.

[stronnag]

precisely my thoughts, we've done a great deal of cleanups under this issue. Even if it turned out to be bad hardware, the cleanups are highly beneficial.

And going back to the sprf3 (which I fully intend to do) only takes a few minutes

[digitalentity]

Yes, even if we will confirm it's marginal hardware we've done great job finding and fixing a bunch of issues including that elusive memory corruption bug in PWM code.

[stronnag]

So I flew Dodo on the recalcitrant quad. Some PH behaviour. Were it not for the flawless 27822a5 I'd blame it on some environmental factor.

Firmware was ebadf7d6 (the sd fix branch) with #511 patched onto it. Same behaviour with or without #511. Unfortunately. I failed to get any logs. Should it stop raining I'll try again later.

[digitalentity]

Wait a second! The Dodo that flew perfectly on the Tri fails on the quad? This is something really odd...

[stronnag]

Yes, that's what I thought. Then I come back to 27822a5. Also tried with and without velned (no change).

And slightly over 50% of the time, it's perfect on any firmware.

[digitalentity]

What's the difference between quad and Tri (apart from airframe)? Now we might have some sort of a hint where this bug lurks.

[stronnag]

Other than the obvious, very little. Both PPM, both serial loggers, both 3DR / LTM. M8N (different vendors and firmware). The quad has the later 3.01. I thought of reverting that, than I come back to 27822a5.

[digitalentity]

Brainstorming possible causes:

• Somehow related to hardware (noise) to which newer version is less tolerant
• Serial UART code
• GPS code
• PWM output code
• Mixer code

Other ideas? @martinbudden what do you think?

[digitalentity]

BTW, are DODO and SPRacingF3 are running the same firmware, right?

[stronnag]

Yes, exactly the same firmware.

And here's one further hint that there's maybe still something 'wild' happening. The phantom motor

1. This happens on both the Dodo and EVO, lipo power applied. No phamtom motors on the quad. The servo is on M7.

[digitalentity]

The Servo #6 IS the tail servo on the Tri, so nothing wrong here, although I'm not sure how it ended up on M7 output.

[martinbudden]

So, can you clarify what you have done. As I understand it you have taken your Dodo FC off your tricopter (which does PH fine) and put it on the quadcopter which previously had the SPRACINGF3. The quadcopter didn't do PH before and now still doesn't do PH. Is that correct?

Also, can you put the SPRACINGF3 on the tricopter and see how that flies?

[digitalentity]

I'd say it's a hardware issue, however we have one single weird thing here - issue is not present on older commit (from before complete IO migration). I'm confused.

@stronnag do you, by any chance, have a SerialRX receiver (S.Bus or something like that)? I'd really like to rule out the PPM receiver code. 0489eb8 did some changes to timer handling which may be the thing that's haunting us now. Also, try disabling gyro sync (just to make sure EXTI is not responsible).

[digitalentity]

If I understand correctly, the difference between Tri and Quad is the airframe and GPS module, the rest is the same. I'd say it's something wrong with GPS code or GPS processing code in NAV system.

Also worth trying - SPRacingF3 EVO on a quad. I know it's not that simple but it will rule out the specific firmware issue.

Facts we have:

1. GPS modules are different on Tri and Quad
2. Issue is not related to the FC hardware (to be confirmed)
3. Issue is sometimes resolved by soft-resetting
4. If PH works - it works perfectly w/o issue, it never fails between flights if machine wasn't rebooted
5. In failed PH attempts we usually have uncorrelated heading and GPS course

Did I miss something?

[stronnag]

I have a spektrum satellite that I can steal off another machine (moving tele off or soft serial). I can also disable gyro sync. I could put the SPRF3 on the tri, but that's my last resort. I hope to get out later today with spek sat and gyro off (and working logging).

Summary:

| Setup | Result |
| --- | --- |
| TRI + Dodo | No failure with any firmware |
| TRI + Evo | No failure with any firmware (only since last 10 days) |
| Quad + SPRF3 / BMP280 / GPS mag | PH fails 50% post 27822a5 |
| Quad + SPRF3 / BMP180 +5583 | PH fails 50% post 27822a5 |
| Quad + Dodo | PH fails 50% (not tested with 27822a5) |

[stronnag]

Did you miss something. No
To confirm:

| Vehicle | GPS |
| --- | --- |
| Quad | Beitian BN880 + V3 firmware |
| Tri | popular 'banggood' M8N (in black plastic cover) + V2 firmware |

[digitalentity]

@stronnag thanks for testing my crazy ideas :smile:

I think the issue is isolated to your specific setup, however since it wasn't evident with earlier revisions, we should try hard and fix it as it's clearly a regression in the latest code.

[stronnag]

And thanks for all the developer effort that has gone into this. I'm more that will to agree that it could well be setup specific post 27822a5, on the other hand, we have a very small testing community.

[stronnag]

Further testing / option elimination

| Test | Effect |
| --- | --- |
| gyro_sync = off | No difference |
| gyro_lpf = 98HZ (vice 188) | No difference |
| Raise GPS tower by 25mm | No difference |
| Use serial RX | No difference |
| move 3DR further away from GPS | No difference |

However, one thing you mentioned did seem worthy of further experimentation.

Most of time, after a hard reset PH will initially fail; a single soft reboot will iirc always fix the condition.
Once PH is working, it will not fail again after subsequent soft reboots (tested with about 6 PH / land / reboot cycles).
Is this indicative of a cold start timing or race condition that did not exist in 27822a5 and is mitigated during a soft reboot as the hardware is ready?

[digitalentity]

@stronnag now this is something!

Quite possible it's a hardware not ready condition. Do you know if it's HMC5883L or HMC5983 chip on BN880 GPS? They use the same driver but HMC5983 is usually taking longer to initialise.

In the sensorsAutodetect() compass is detected last, so we have this sequence of initialisation:
PWM -> Serial -> GYRO -> ACC -> BARO -> MAG

If during migration to new IO we saved some time on initialising all of them then the delay for compass to initialise on power-up is no longer enough.

Another possibility is that we start initialising GPS too early, not giving it much time to boot properly.

Simply looking at the diff output I see that we changed a delay in bmp085 detection from 200ms to 20ms, maybe some other code is working faster as well. If so - simply adding a pre-init delay when detecting a power-up condition (not a soft-reset) will solve the problem.

I'll prepare a debug branch to test as soon as possible!

[digitalentity]

I've implemented two extra delays on cold-booting in #517. One is imposed prior to sensor detection (500ms), the other delays GPS initialisation to start at least 2sec from power-up.

You get PH working some time, so missing delay is probably marginal. I suspect that reduced BMP085 detection delay (less 180ms to boot time) is what caused the problem. If so, extra half-second delay should put things back to perfect state.

[stronnag]

In the field. 7 successful pH cycles on boot-delay and naze32. Big grin. Changing fc to dodo for more tests.

[stronnag]

10 consecutive PH cycles on dodo with boot-delay. We're fixed. Time for another RC?

[digitalentity]

Yee-haw! We finally did it!
Cleaned up lots of stuff on the way and finally tracked to mere needing a bigger boot delay.
Everybody involved, great job!

[martinbudden]

Fantastic!

Thanks again @stronnag for all your help in testing.

An lots of good cleanup resulted from this.

And I am SO GLAD that it was not a problem with the conversion to the new IO!

[digitalentity]

It's definitely time for another RC. I'll merge the following and release an RC3
Improved gyro initialisation #456
Add extra delays when cold-booting the UAV #517
Changed mag.init to return false if failed #515 (after testing)

[martinbudden]

@digitalentity can you also merge https://github.com/iNavFlight/inav/pull/521 - updated targets.

It just adds some BARO and MAG #defines to some targets, adds the REVO_NANO and REV OPBL targets and makes a minor tidy to the makefile.

[bk79]

Also please merge ibus telemetry

Il 29/ago/2016 11:11, "Martin Budden" [email protected] ha scritto:

@digitalentity https://github.com/digitalentity can you also merge #521
https://github.com/iNavFlight/inav/pull/521 - updated targets.

It just adds some BARO and MAG #defines to some targets, adds the
REVO_NANO and REV OPBL targets and makes a minor tidy to the makefile.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/iNavFlight/inav/issues/431#issuecomment-243073047,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALSWigfjIyn7_N128DxnUJcxg16azV3cks5qkqIugaJpZM4JeUXk
.

[stronnag]

And EVO SDCARD ....

[digitalentity]

@bk79 did anybody do a PR for iBus telem?
@stronnag ofcourse, forgot that one, already merged.

[hydra]

@stronnag I _really_ appreciate all the time and test flights and willingness to put your aircraft at risk in order to help solve this problem.

Well done to everyone else involved too, of course!

[randyoo]

I know this bug is marked closed, but I'm pretty sure I'm experiencing it (or something closely related), because engaging PH seems to always result in the aircraft flying away, but after a soft reboot is performed, PH works fine. (still gathering data, but haven't observed any occasions where PH works on hard reset, or doesn't work on soft reboot)

This is on an Omnibus clone (f4 v5pro) on 1.7.2 with a Beitian BN-880 GPS/compass. There's no blackbox capability on this FC, or I'd share a log. Willing to perform any steps that might help narrow down the cause of the issue.

EDIT: After some more research, it appears very likely that this is issue #1791 that I'm experiencing.