Px4-autopilot: jMAVSim SITL ekf2 starting/stopping navigation continuously

[julianoes]

Recently in SITL, I've been seeing ekf2 starting and stopping navigation continuously every few seconds.

This seems like a regression somewhere. I see this on macOS with jMAVSim on Java 8.

Here is log where the drone just sits there.
https://logs.px4.io/plot_app?log=8a5d658f-46d1-47a6-845d-44dc40247da6

Edit: this seems to happen back in April (8c9b941e604e6ed971059e5b5fce12b95e9020fb)

Init MAVLink
INFO  [simulator] Got initial simulation data, running sim..
INFO  [pwm_out_sim] MODE_16PWM
ERROR [sensors] no ADC found: /dev/adc0 (9)
INFO  [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 14556 remote port 14550
INFO  [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14557 remote port 14540
INFO  [mavlink] MAVLink only on localhost (set param MAV_BROADCAST = 1 to enable network)
INFO  [logger] logger started (mode=all)
pxh> INFO  [logger] Start file log
INFO  [logger] Opened log file: rootfs/fs/microsd/log/2018-05-25/15_07_36.ulg
INFO  [Unknown] EKF aligned, (pressure height, IMU buf: 22, OBS buf: 14)

pxh>
pxh>
pxh>
pxh> ekf2 status
INFO  [ekf2] local position OK no
INFO  [ekf2] global position OK no
INFO  [ekf2] time slip: 2 us
pxh> ekf2 status
INFO  [ekf2] local position OK no
INFO  [ekf2] global position OK no
INFO  [ekf2] time slip: 158 us
pxh> INFO  [Unknown] EKF GPS checks passed (WGS-84 origin set)
INFO  [Unknown] EKF commencing GPS fusion

pxh>
pxh> WARN  [Unknown] EKF GPS data stopped
WARN  [Unknown] EKF stopping navigation
INFO  [Unknown] EKF commencing GPS fusion

pxh>
pxh>
pxh> ekf2 statusWARN  [Unknown] EKF GPS data stopped
WARN  [Unknown] EKF stopping navigation
INFO  [Unknown] EKF commencing GPS fusion

INFO  [ekf2] local position OK yes
INFO  [ekf2] global position OK yes
INFO  [ekf2] time slip: 29 us
pxh>
pxh>
pxh> ekf2 status
INFO  [ekf2] local position OK yes
INFO  [ekf2] global position OK yes
INFO  [ekf2] time slip: 59 us
pxh> gps status
INFO  [gps] not running
Command 'gps' failed, returned -1
pxh> uorb top

update: 1s, num topics: 79
TOPIC NAME                        INST #SUB #MSG #LOST #QSIZE
actuator_controls_0                  0    6  242   653 1
battery_status                       0    6   89   301 1
ekf2_innovations                     0    1  242   237 1
ekf2_timestamps                      0    1  242    22 1
estimator_status                     0    4  242  1114 1
geofence_result                      0    1    4     0 1
rate_ctrl_status                     0    1  242   215 1
sensor_accel                         0    1  242     0 1
sensor_accel                         1    1  249    55 1
sensor_baro                          0    1  100     0 1
sensor_bias                          0    4  242   405 1
sensor_combined                      0    4  242   231 1
sensor_gyro                          0    2  242     0 1
sensor_mag                           0    1  100     0 1
sensor_preflight                     0    1  242   235 1
vehicle_air_data                     0    5  100    49 1
vehicle_attitude                     0    6  242   698 1
vehicle_attitude_groundtruth         0    1  167    73 1
vehicle_attitude_setpoint            0    4  242   650 1
vehicle_global_position              0    5  242   994 1
vehicle_global_position_groundtruth  0    1  167   155 1
vehicle_gps_position                 0    5    9     8 1
vehicle_land_detected                0    7    1     0 1
vehicle_local_position               0    7  242  1107 1
vehicle_local_position_groundtruth   0    1  167   155 1
vehicle_local_position_setpoint      0    4  242   844 1
vehicle_magnetometer                 0    4  100    49 1
vehicle_rates_setpoint               0    4  242   627 1

[Antiheavy]

I noticed the ekf start and stop navigation in SITL too. I started noticing it a month ago or so. I’m running SITL in docker in Windows.

[Antiheavy]

this is a post I made about it a while ago in PX4 Discuss:
http://discuss.px4.io/t/getting-ekf-gps-datra-stopped-in-sitl/6441

[LorenzMeier]

@julianoes Do you have the cycles to look into it?

[julianoes]

@julianoes Do you have the cycles to look into it?

Not right now, sorry.

[LorenzMeier]

@julianoes Checking in - any chance you could have a look?

[LorenzMeier]

I can reproduce this. Performance on MacOS:

[LorenzMeier]

Looked into this on the weekend, but no final verdict yet. I suspect that we have a busy loop somewhere or run something at a too high rate and hence get the high load.

[bkueng]

Found the root cause, it's described here: https://github.com/PX4/ecl/issues/459