Device-os: Feature Request: Decrease RAM usage of listening mode

[elcojacobs]

Listening mode needs too much RAM

On the photon, with system thread enabled, a minimal app consumes 35.7kb of RAM.
The maximum available is 81.4 kb.

Triggering listening mode consumes an extra 32.3 kb of RAM, leaving the free available RAM at only 13.4kb.

Here is a table from the test app below:

  | Free | Max | Used | Total | Used %
-- | -- | -- | -- | -- | --
Before | 45640 | 35724 | 35724 | 81364 | 43.91%
During | 13408 | 68052 | 67956 | 81364 | 83.52%
After | 39416 | 68052 | 41948 | 81364 | 51.56%

What this means is that when listening mode is triggered while running an app, the user should ensure that he uses less than 13kb RAM! Otherwise, memory allocations will fail when listening mode is triggered and I have seen this causing all kinds of problems, including the thread management getting confused and not returning to normal state at listening mode timeout.
After listening mode, WiFi would not reconnect and the device would be stuck in breathing blue.

I know you can set up a hook with System.on for setup_begin, and you could try to free all the user memory before entering listening mode. But this has a few downsides:

• 13kb is not much memory. You would have to tear down your entire app.
• This makes static memory use impossible. This is subtracted from the available heap. Requiring to free user memory for listening mode forces the developer to dynamically allocate all RAM.
• This out of memory trap is a very easy trap to fall in. I think most apps will use over 13kb. And most developers writing simple apps don't even think about freeing memory. Listening mode will not function properly, unless the developer stops all tcp servers, mdns servers, any hardware interface classes with larger buffers. Many of these are not even designed to be deallocated.
• Unless you start digging into this, you don't know that memory allocation is failing and you just see random errors. I didn't realize this was happening until I started showing RAM use on my LCD screen continuously.

SoftAPApplication

On the photon, listening mode starts SoftAP with a multi-page HTTP server. This partly causes the high memory usage. It seems to only be disabled when re-compiling the system layer with MINIMAL=y.
Listening mode starts:

• SoftAp
• A HTTP server (not on all platforms, but default on the photon)
• A serial protocol
• A simple tcp protocol

How many people actually use the HTTP server provided by the softap? It is not required by the phone app or when configuring WiFi over serial I think?

Snippet from softap.cpp:

/**
 * The SoftAP setup application. This co-ordinates the various dispatchers and the soft AP.
 */
class SoftAPApplication
{
    SoftAPController softAP;
    AllSoftAPCommands commands;
#if SOFTAP_HTTP
    HTTPDispatcher http;
#endif
    SimpleProtocolDispatcher simpleProtocol;
    TCPServerDispatcher tcpServer;
    SerialDispatcher serial;

public:
    SoftAPApplication(void (*complete_callback)()) :
            commands(&softAP.complete_semaphore(), complete_callback),
#if SOFTAP_HTTP
                    http(commands),
#endif
            simpleProtocol(commands),
            tcpServer(simpleProtocol, WICED_AP_INTERFACE),
            serial(simpleProtocol)
    {
        softAP.start();
        serial.start();
        tcpServer.start();
#if SOFTAP_HTTP
        http.start();
#endif
    }

Possible fixes

• Only start softap HTTP server when listening mode is not started from a user app. This would still make it available for first time setup of bare photons.
• Make it an optional (default false) flag, set at runtime, instead of a compile time define that requires recompiling the system layer.
• Make it a compile time flag, but one that doesn't require recompiling the system layer. So pass it to the system layer as an argument.

My preference would be to be able configure all of these separately. I would be happy if I could configure listening mode to only use serial.

Other observations:

• Base memory use is pretty high.
• Memory use after listening mode exit higher than before.

Test App

/* Includes ------------------------------------------------------------------*/
#include "application.h"
#include <cinttypes>

SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);

void
printHeapUse(char* dest, uint8_t maxLen)
{
    runtime_info_t info;
    memset(&info, 0, sizeof(info));
    info.size = sizeof(info);
    HAL_Core_Runtime_Info(&info, NULL);
    snprintf(dest, maxLen, "%" PRIu32 ", %" PRIu32 ", %" PRIu32 "", info.freeheap, info.max_used_heap, info.total_heap);
}

void
setup()
{
    WiFi.connect(WIFI_CONNECT_SKIP_LISTEN);
    Serial.begin(115200);
}

char loopStart[32];
char duringListening[32];
char afterListening[32];

void
loop()
{
    printHeapUse(loopStart, sizeof(loopStart));
    delay(5000);

    WiFi.listen(true);
    delay(5000);
    printHeapUse(duringListening, sizeof(duringListening));

    WiFi.listen(false);
    delay(1000);
    printHeapUse(afterListening, sizeof(afterListening));

    WiFi.connect(WIFI_CONNECT_SKIP_LISTEN);
    while (1) {
        Serial.print("before: ");
        Serial.println(loopStart);
        Serial.print("during: ");
        Serial.println(duringListening);
        Serial.print("after: ");
        Serial.println(afterListening);
        delay(3000);
    }
}

Results with system layers recompiled with MINIMAL=y

This excludes the http server by setting SOFTAP define.

  | Free | Max | Used | Total | Used %
-- | -- | -- | -- | -- | --
Before | 45640 | 35724 | 35724 | 81364 | 43.91%
During | 24144 | 57316 | 57220 | 81364 | 70.33%
After | 39352 | 57316 | 42012 | 81364 | 51.63%

Closing thoughts

The app doesn't crash completely when memory is full during listening mode. The app will return to lower memory use when listening mode exits. But I think including http by default is a mistake, not just because not many people use it, but because it will not even work properly for many users because the memory limit is hit.
SoftAP http should be optional and come with a huge disclaimer and perhaps even a runtime check for available RAM before it is even started up. The current implementation is a trap.

[elcojacobs]

Another option worth considering:
Don't run listening mode parallel to the user app. Load it as a separate app, bootloader style load.
Give the user a hook to neatly shut down the app, load the SoftAP app instead, restart the user app afterward.

The SoftAP app is an app on its own. It is not designed to be run in parallel:

• It takes over WiFi
• It takes over Serial
• It uses 70% of user available memory.

I have to adapt my app to detect listening mode and to take down all communication.
Yes, I can still interface with hardware and pins with the 13kb memory I have left. But perhaps saying listening mode is a separate config mode in which the app does not run is easier.

[elcojacobs]

I just discovered that the situation is even worse.

When listening mode is triggered, the SoftAP constructor is called first!
Only after constructing SoftAP, the user callback is called.

So there is no way for the user to even try to free memory for listening mode!
In other words: never use more than 13k memory in your app at all times if you want listening mode to work. :confused:

[No1089]

@elcojacobs SoftAP is part of the WICED stack, and as such, outside of Particle's control.
Your ask is thus considered a feature request and will be addressed as such.

I will update you in due course when a way forward has been decided upon.

[elcojacobs]

This assessment is wrong.
The SoftAPApplication class that implements starting a http server is not in wiced. It is in your own file, softap.cpp.
You even have a compile time #if SOFTAP_HTTP to check whether it should be included.
You have a compile flag MINIMAL=y that excludes it but requires rebuilding system layers.

I even included a snippet of that class in my issue.
The part that is in WICED is only a small part of what that class starts. And not the biggest memory hog.

Now can someone REALLY look at the issue before dismissing it?

This is a major bug in your platform. Give it a serious look! Jesus. I'd be happy to take some time to explain it to a competent developer. Shouldn't take much time at all and I am certain he will agree with me within 5 minutes.

[elcojacobs]

More discussion here: https://community.particle.io/t/listening-mode-on-the-photon-cannot-work-reliably-in-current-implementation/55789/21?u=elco

[elcojacobs]

So... Has anyone looked at this issue yet?

[No1089]

@elcojacobs Yes, as I mentioned in the ticket, we will send you a response once the way forward has been decided upon.

[elcojacobs]

And I am still waiting for that response.

[No1089]

As mentioned before, we are working on it, and you will receive a response once a way forward has been determined.

[No1089]

Good news, our LTS DeviceOS will include a fix that will decrease per-thread RAM usage. Exact numbers are not yet available - I will update once we know more!

[elcojacobs]

That will definitely help and is very welcome. The design to allocate 32kb of memory without checks, without any possible user customization, and before the user callback hook is still broken though, so I hope that will be addressed separately.

Just had another bug report from one of my biggest customers that he couldn't configure wifi credentials in any way.

[brahma-dev]

After months of development we start flashing new batch of devices and turns out we simply can't configure wifi after flashing our code.

[No1089]

@brahma-dev Have you tried using Device OS 2.0-rc.1? The RAM usage has dropped significantly and users having trouble with previous versions are able to use listening mode without issue.

[elcojacobs]

This issue has been closed, but it is not resolved in my opinion.
With device-os 2.0-rc.1, if you use a lot of memory and start listening mode, a hardfault still occurs.

In the current implementation, the application memory use has to be kept low at all times for listening mode to work. We cannot detect listening mode in time to prevent an out of memory crash.

Listening mode tries to allocate a lot of memory before the application receives an event. If the event would be sent to the application before listening mode starts up all its handlers, it would give application developers the chance to free memory. The current event is useless for custom memory handling.

setup_begin: signals the device has entered setup mode

I think we would need an additional event (or change the implementation):

setup_triggered: signals the device is about to enter setup mode

[brahma-dev]

Agree with @elcojacobs. The core problem is still there. We're using the workaround from here : https://community.particle.io/t/listening-mode-on-the-photon-cannot-work-reliably-in-current-implementation/55789/27?u=brahma-dev