Px4-autopilot: Memory and mavlink problems pixhawk v2

[tuloski]

I noticed that my pixhawk v2 was having some problems, so I hooked up the console in telem 4/5.

What I see at random are different problems:

• Memory problems: this warning comes out "WARNING WARNING WARNING!
  Revision A has a silicon errata:
  This device can only utilize a maximum of 1MB flash safely!
  https://docs.px4.io/en/flight_controller/silicon_errata.html" but I havn't found any way to fix it.
  And it's not only a warning because later this happens:

"nsh: mount: mount failed: Out of memory
nsh: mkfatfs: mkfatfs failed: Out of memory"

• Mavlink problems: I have a custom "extras" script to launch mavlink instances. In particular the script is the following:

mavlink stop-all
#ESP
#mavlink start -r 20000 -b 921600 -d /dev/ttyS0
#USB
mavlink start -d /dev/ttyACM0 -b 1000000 -m custom
mavlink stream -d /dev/ttyACM0 -s HEARTBEAT -r 2
mavlink stream -d /dev/ttyACM0 -s ATTITUDE_QUATERNION -r 50
mavlink stream -d /dev/ttyACM0 -s ATTITUDE -r 10
mavlink stream -d /dev/ttyACM0 -s HIGHRES_IMU -r 50
mavlink stream -d /dev/ttyACM0 -s GPS_RAW_INT -r 5
mavlink stream -d /dev/ttyACM0 -s GLOBAL_POSITION_INT -r 50

what happens sometimes at random is this error:
"INFO [mavlink] all instances stopped
INFO [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B
ERROR [mavlink] could not open /dev/ttyACM0
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN [mavlink] mavlink for device /dev/ttyACM0 is not running
"

I really don't know where to start fixing it. Can it be all related to memory issues? Why isn't it able to open /dev/ttyACM0 sometimes?

PS: I tried to update the bootloader with https://docs.px4.io/v1.9.0/en/advanced_config/parameter_reference.html#SYS_BL_UPDATE without any success.

[mhkabir]

You're running out of RAM. You need to either stop some modules (unused mavlink instances for example), or get newer hardware with more RAM (e.g Pixhawk 4)

The Silicon Errata warning is about Flash, and is unrelated to the issues you're seeing.

[tuloski]

But it's standard px4 1.9.2 code.
There should be a warning that it is unusable on pixhawk v2.

[tuloski]

BTW "free" returns:

nsh> free
total used free largest
Umem: 238064 170320 67744 64576

It seems there is some free RAM.

[mhkabir]

Can you post logs from before and after your extras script runs?

@dagar FYI

[tuloski]

Case 1 (happening at random, can't open ACM0):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255                          204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

[hardfault_log] Fault Log is Armed
INFO  [tune_control] Publishing standard tune 1
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [370:100]
INFO  [px4flow] scanning I2C buses for device..
Addons script: /fs/microsd/etc/extras.txt
INFO  [mavlink] waiting for instances to stop
.INFO  [mavlink] exiting channel 0

INFO  [mavlink] all instances stopped
INFO  [mavlink] mode: Custom, data rate: 50000 B/s on /dev/ttyACM0 @ 1000000B
ERROR [mavlink] could not open /dev/ttyACM0
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
WARN  [mavlink] mavlink for device /dev/ttyACM0 is not running
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)

NuttShell (NSH)

Case 2 (happening at random, out of memory):

FMUv2 ver 0x8 : Rev 0 V2
[boot] Fault Log info File No 4 Length 3177 flags:0x01 state:1
[boot] Fault Log is Armed
sercon: Registering CDC/ACM serial driver
sercon: Successfully registered the CDC/ACM serial driver
HW arch: PX4_FMU_V2
HW type: V2
HW version: 0x00090008
HW revision: 0x00000000
FW git-hash: 106905871d374c125730fe675266d67c7f869469
FW version: Release 1.9.2 (17367807)
OS: NuttX
OS version: Release 7.28.0 (119275775)
OS git-hash: 423371c7d4012e725ac4ca51323a18df64e581b3
Build datetime: Jul  9 2019 13:47:06
Build uri: localhost
Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
PX4GUID: 000100000000313235333432470a00280019
MCU: STM32F42x, rev. A

WARNING   WARNING   WARNING!
Revision A has a silicon errata:
This device can only utilize a maximum of 1MB flash safely!
https://docs.px4.io/en/flight_controller/silicon_errata.html

nsh: mount: mount failed: Out of memory
INFO  [tune_control] Publishing standard tune 16
nsh: mkfatfs: mkfatfs failed: Out of memory
INFO  [tune_control] Publishing standard tune 2
INFO  [param] selected parameter default file /fs/mtd_params
Board defaults: /etc/init.d/rc.board_defaults
WARN  [dataman] Could not open data manager file /fs/microsd/dataman
ERROR [dataman] dataman start failed
rgbled on I2C bus 1 at 0x55 (bus: 100 KHz, max: 100 KHz)
nsh: rgbled_ncp5623c: command not found
nsh: rgbled_pwm: command not found
nsh: blinkm: command not found
Board sensors: /etc/init.d/rc.board_sensors
HMC5883_I2C on I2C bus 1 at 0x1e (bus: 100 KHz, max: 400 KHz)
nsh: lis3mdl: command not found
nsh: ist8310: command not found
WARN  [hmc5883] no device on bus 2 (type: 1)
WARN  [mpu6000] no device on bus #3 (SPI1)
MPU6000 on SPI bus 1 at 4 (1000 KHz)
WARN  [mpu6000] no device on bus #5 (SPI4)
INFO  [mpu9250] Bus probed: 3
WARN  [mpu9250] probe failed! 104
WARN  [mpu9250] no device on bus 3
INFO  [mpu9250] Bus probed: 5
WARN  [mpu9250] probe failed! 0
WARN  [mpu9250] no device on bus 5
L3GD20 on SPI bus 1 at 1 (11000 KHz)
LSM303D on SPI bus 1 at 2 (11000 KHz)
MS5611_SPI on SPI bus 4 at 3 (20000 KHz)
WARN  [ms5611] no device on bus 4
MS5611_SPI on SPI bus 1 at 3 (20000 KHz)
nsh: bst: command not found
ERROR [param] Parameter SENS_EN_BATT not found
ERROR [param] Parameter SENS_EN_MB12XX not found
ERROR [param] Parameter SENS_EN_PGA460 not found
ERROR [param] Parameter SENS_EN_SF1XX not found
ERROR [param] Parameter SENS_EN_TRANGER not found
ERROR [param] Parameter TRIG_MODE not found
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [mavlink] mode: Config, data rate: 800000 B/s on /dev/ttyACM0 @ 57600B
Starting Main GPS on /dev/ttyS3
Starting MAVLink on /dev/ttyS1
INFO  [mavlink] mode: Normal, data rate: 1200 B/s on /dev/ttyS1 @ 57600B
ERROR [mavlink] DM_KEY_MISSION_STATE lock failed
ERROR [mavlink] offboard mission init failed (-1)
INFO  [px4io] default PWM output device
INFO  [init] Mixer: /etc/mixers/quad_x.main.mix on /dev/pwm_output0
INFO  [init] Mixer: /etc/mixers/pass.aux.mix on /dev/pwm_output1
ERROR [param] Parameter CAM_CAP_FBACK not found
ERROR [param] Parameter WEST_EN not found
px4flow [350:100]
INFO  [px4flow] scanning I2C buses for device..
ERROR [param] Parameter UAVCAN_ENABLE not found
INFO  [logger] logger started (mode=all)
INFO  [logger] log root dir created: /fs/microsd/log

NuttShell (NSH)

[dagar]

What's on you SD card? Can you format it and try again?

[tuloski]

What's on you SD card? Can you format it and try again?

There are only extras and logs.
Formatted it now with no luck.

[dagar]

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

• nsh: mount: mount failed: Out of memory
• mkfatfs: mkfatfs failed: Out of memory
• WARN [dataman] Could not open data manager file /fs/microsd/dataman
• ERROR [dataman] dataman start failed

[tuloski]

All the errors involve the sdcard. What size is it? How is it formatted? Do you have another you can try?

• nsh: mount: mount failed: Out of memory
• mkfatfs: mkfatfs failed: Out of memory
• WARN [dataman] Could not open data manager file /fs/microsd/dataman
• ERROR [dataman] dataman start failed

It's a Kingstone 4Gb that ships with pixhawk.
BTW I switched it with a newer one 16 Gb, and it seems solved.
I'll keep you updated if it happens again. Thanks for the input.

[tuloski]

mmm I said hurray too early. The problem is still there. It worked 3 reboots and now again at random.
I'm starting to think it's some hardware failure on the pixhawk.

[julianoes]

@tuloski can you see the log output when it fails?

[tuloski]

@tuloski can you see the log output when it fails?

In the SD?

[julianoes]

No, that would be with a console connected: https://dev.px4.io/en/debug/system_console.html

[tuloski]

Yes, actually what I wrote in Case1 and Case2 are console outputs. Also when it fails I see it.

[julianoes]

Oh, sorry I missed that.

[tuloski]

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue.
Still not clear what the real issue is.

[julianoes]

Adding some usleep of 0.5s here and there in the extras.txt seems to solve the issue.
Still not clear what the real issue is.

That's odd though because when it fails it doesn't even mount the SD card (from what I see in the log above).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.