Arduino: Sketch size and memory usage after #5376 commit

[reaper7]

Basic Infos

• [ ] This issue complies with the issue POLICY doc.
• [x] I have read the documentation at readthedocs and the issue is not addressed there.
• [x] I have tested that the issue is present in current master branch (aka latest git).
• [x] I have searched the issue tracker for a similar issue.
• [ ] If there is a stack dump, I have decoded it.
• [ ] I have filled out all fields below.

Platform

• Hardware: [ESP8285]
• Core Version: [latest git https://github.com/esp8266/Arduino/pull/5376]
• Development Env: [Arduino IDE]
• Operating System: [Windows]

Settings in IDE

• Module: [ESP8285]
• Flash Mode: [default for esp8285]
• Flash Size: [1MB]
• lwip Variant: [v2 Higher Bandwidth]
• Reset Method: [ck]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [OTA]
• Upload Speed: [] (serial upload only)

Problem Description

what happened with the final sketch size and dynamic memory usage on commit https://github.com/esp8266/Arduino/commit/6280e98b0360f85fdac2b8f10707fffb4f6e6e31?

I try to compile latest sonoff-tasmota for ESP8285; 1M/noSPIFFS;
and on previous commit: https://github.com/esp8266/Arduino/commit/49417115058e7c74a1ca1f033a9259a369e057a0
I getting:
Sketch size: 491564 (48%), max 1023984
Global variables: 50984 (62%), 30936 free for local variables

but with this commit:
Sketch size:575192 (56%), max 1023984
Global variables: 78044 (95%), 3876 free for local variables

and info about low level of available memory (possible stability problem)

MCVE Sketch

sonoff-tasmota sketch
https://github.com/arendst/Sonoff-Tasmota

Debug Messages

Debug messages go here

[devyte]

The price for exceptions is supposed to be about 19KB of flash size and about 800 bytes of heap.

[d-a-v]

@reaper7 can you upgrade the toolchain with tools/get.py ?

[earlephilhower]

That's a good sketch for stress testing things, @reaper7. Do you just put the libs in ~Arduino/libraries, and build sonoff.ino with any configurations?

You will need to update your toolchain (manually rerun tools/get.py) and apply #5425 to get it to build, but I'm seeing an iram explosion in linking that needs looking at before a good binary can appear.

[earlephilhower]

I checked out #5376~1 but using the latest toolchain (i.e. just before the exceptions merge, but exceptions are not enabled in the core/arduino build flags) and can't build w/nodebug/noassert due to IRAM full errors, so I think I'm building a different config than you. That said, I expanded the IRAM linker space (i.e. it won't run, but will give me a binary).

Pre-exceptions build usage
FLASH: 550292
RAM: 50448

Post-exceptions build use (I could build normally w/#5425 as long as I did NDEBUG+NASSERT config):
FLASH: 685812
RAM: 49828

So in my build I'm seeing flash usage is up significantly at 150K, RAM is down by ~600 bytes.

My guess is that there are news which are now capable of throwing exceptions, and so have exception strings.

[reaper7]

I think I have latest toolchain:

r7@reaper7 MINGW64 /c/PROGRAMY/Arduino/hardware/esp8266com/esp8266/tools (master)
$ python get.py
Platform: i686-mingw32
Tool esptool-0.4.13-win32.zip already downloaded
Extracting dist/esptool-0.4.13-win32.zip
Renaming esptool-0.4.13-win32/ to esptool
Tool i686-w64-mingw32.xtensa-lx106-elf-2c41c41.zip already downloaded
Extracting dist/i686-w64-mingw32.xtensa-lx106-elf-2c41c41.zip
Tool mkspiffs-0.2.0-no_magic_length-windows.zip already downloaded
Extracting dist/mkspiffs-0.2.0-no_magic_length-windows.zip
Renaming mkspiffs-0.2.0-no_magic_length-windows/ to mkspiffs

I see where is my mistake with RAM...for a long time I have been building with VTables in HEAP,
from this commit I must switch to VTables in flash :)

I still have to disable some sonoff modules for ota functionality (sketch size <50%)

[earlephilhower]

Did some looking in generated executable and you've got 87K worth of gcc_except_table + eh_frame (basically the data structure G++ uses to handle exceptions) with this codebase.

There's an add'l ~19K of code to actually process those bits on an exception (i.e. new of an object hits OOM, etc.)

This points to the need for exceptions to be user configurable...ugh, but good to catch now!

[earlephilhower]

See #5434 for a menu to disable exceptions completely.

[Jason2866]

So the default will be disabled exceptions?

[earlephilhower]

Exception support only adds ROM and has ~0 impact on free heap. It catches when you try to do a String("This is a test") which can't allocate memory and prints both the exception info (std::bad_alloc === OOM) and gives you a stackdump to catch probably 1/2 of the "my program doesn't work" type bugs we've been seeing. With this, and no changes to anyone's apps, if any new fails, you'll know immediately where and get a nice debug dump.

ROM size will be an issue for some stuck on 512M or 1M flash devices looking to do OTA, I fully understand.

The default now is enabledbut if enough users have an issue we can definitely adjust!

[Jason2866]

Thank you for explanation. ROM size matters for all TASMOTA users ;-)
We are now already on the boarder to 512M. So every kbyte counts

[devyte]

Closing via #5434 .

A word of caution to Tasmota developers, as well as others who are near the 512KB bin size limit: Exceptions are configurable for now, but it is planned to eventually enable them permanently. This is because Exceptions are the only mechanism that allows reacting correctly to error conditions in some cases, e.g.: oom while allocating in construction, oom during concat of Strings, etc. This drastically improves code robustness.
In addition, as I explained elsewhere, we intend to replace some custom code with standard libc/libc++ calls, or based on STL calls and containers, and there is some reliance on Exceptions there as well. This reduces our own code and maintenance overhead, and again improves robustness. Finally, there is a plan to bring in a subset of boost, which also relies somewhat on Exceptions. There is already some work underway about this.
It is not expected to make Exceptions part of the Arduino user-facing API, but it is planned to add Exceptions to handle cases like the above within our core implementation and catch them out of sight, e.g.: in loop_wrapper().

As a result, I suggest planning for bin size increase. There are several things you might do to slim down your build result, here are some ideas (without having looked at your code):

• drop old code (we saw some busted pwm and wiring functions in your bins while looking into the bin size increase, which aren't in our core anymore)
• use templates instead of a bunch of overloads for similar types with almost duplicated code (templates get instantiated only as needed according to what is called, so no bloat for unused overloads)
• migrate big strings such as embedded html to files for loading into a SPIFFS image
• don't use macro-functions (defines that get used like functions, but then get expanded inline).