Px4-autopilot: PX4IO not enough RAM left for VTOL models like convergence

Created on 6 Feb 2020 · 8Comments · Source: PX4/PX4-Autopilot

@sfuhrer found that sometimes RC did not work on the convergence and that the ACT LED on the Pixhawk 4 IO side would be always on instead of blinking.

We could also see a segfault using GDB via Dronecode Probe:

Program received signal SIGSEGV, Segmentation fault.
mm_free (heap=0x20000438 <g_mmheap>, mem=0x20001e38) at mm_heap/mm_free.c:130
130   if ((prev->preceding & MM_ALLOC_BIT) == 0)
(gdb) bt
#0  mm_free (heap=0x20000438 <g_mmheap>, mem=0x20001e38)
    at mm_heap/mm_free.c:130
#1  0x08009132 in NullMixer::~NullMixer (this=0x20001e38, 
    __in_chrg=<optimized out>)
    at ../../../../../src/lib/mixer/NullMixer/NullMixer.hpp:47
#2  0x080077d4 in List<Mixer*>::clear (
    this=this@entry=0x20000e3c <mixer_group>)
    at ../../../../../src/include/containers/List.hpp:168
#3  0x08007872 in MixerGroup::reset (this=0x20000e3c <mixer_group>)
    at ../../../../../src/lib/mixer/MixerGroup.hpp:74
#4  mixer_handle_text (buffer=buffer@entry=0x20000c65 <dma_packet+4>, 
    length=62) at ../../../../../src/modules/px4iofirmware/mixer.cpp:601
#5  0x08005d0c in registers_set (page=<optimized out>, 
    offset=<optimized out>, values=values@entry=0x20000c65 <dma_packet+4>, 
    num_values=<optimized out>)
    at ../../../../../src/modules/px4iofirmware/registers.c:464
#6  0x08006446 in rx_handle_packet ()
    at ../../../../../src/modules/px4iofirmware/serial.c:176
#7  rx_dma_callback (handle=<optimized out>, status=status@entry=2 '\002', 
    arg=arg@entry=0x0)
    at ../../../../../src/modules/px4iofirmware/serial.c:234
#8  0x080065be in serial_interrupt (irq=<optimized out>, 
    context=<optimized out>, arg=<optimized out>)
    at ../../../../../src/modules/px4iofirmware/serial.c:325
#9  0x08002fe2 in irq_dispatch (irq=irq@entry=54, 
    context=context@entry=0x20001c5c) at irq/irq_dispatch.c:176
#10 0x08001d88 in up_doirq (irq=54, regs=0x20001c5c) at armv7-m/up_doirq.c:86
#11 0x08001a7a in exception_common () at armv7-m/gnu/up_exception.S:213
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

According to git bisect the bytes free go from 568 to 88 (sometimes 24) anywhere here, presumably due to the NuttX upgrade:

The first bad commit could be any of:
58799dc7d17b01e4e586463b23cb83692b678b77
694ae56afea4fdcd7d8ef665ab7a516e758dd7df
e847698c9f0ec950310eea30313340bf98b37df0
36a75b79a418129913936bd987717afa8914a911
645b6c3fbcdeb2b5f9d3e56807fbaa1fa74811ec
0f2a7ad2b7d2acea354f1b91771512f2a1f76eee
522ed2d99ba4b0ed6f749a6c2218a6d4ec0c9bef
78944a70c345d1da51e906e6225352607675ac21
d965b928f8ef4985c0799e8ce64df4d9c4912a2b
69694ab000f923106931612fa34f8d689b1732fc

Steps to reproduce:

Set SYS_AUTOSTART to 13012 for convergence airframe.
Boot and do px4io status.
You should see:

88 bytes free

Or sometimes garbage values when it crashed.

FYI @bkueng @davids5.

bug fmu-v5 priority-critical

Source

julianoes

All 8 comments

For comparison.

PX4IO: starting
MEM: fre` 1160, Hargest 1160  <-- stable v1.10
MEM: free 622, largest 632    <-- master

dagar on 6 Feb 2020

@dagar What is that comparing? And what airframe?

julianoes on 6 Feb 2020

It's the SYSLOG output in px4io.c before the mixer is even loaded.

https://github.com/PX4/Firmware/blob/b399698306e5afa068903adda094adedf0423691/src/modules/px4iofirmware/px4io.c#L359

dagar on 6 Feb 2020

👀1

@julianoes - it was the last upgrade of NuttX, when the H7 came in. FD/streams became required. We could set it to 0 in the past and that saved memory. I adjusted the stack and free mem to boot. with out police lights, but Not all the mixers were tested. :(