Arduino: SDFS/SD make F() strings appear in RODATA, not flash

[TD-er]

Basic Infos

• [x] This issue complies with the issue POLICY doc.
• [x] I have read the documentation at readthedocs and the issue is not addressed there.
• [x] I have tested that the issue is present in current master branch (aka latest git).
• [x] I have searched the issue tracker for a similar issue.
• [ ] If there is a stack dump, I have decoded it.
• [ ] I have filled out all fields below.

Platform

• Hardware: [ESP-12|ESP-01|ESP-07|ESP8285 device|other]
• Core Version: [Core 2.6.0]
• Development Env: [Platformio]
• Operating System: [Windows]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [dio]
• Flash Size: [4MB/1MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [OTA|SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

In my project, we have had log to SD present, but since it was disabled for a long time, I now made it compile again.
At first there is this strange thing when including SD.h, a lot of __FlashStringHelper related code would no longer compile.

For example I could not compile code like this anymore:

   static const __FlashStringHelper *gpMenu[8][3] = {
      F("⌂"),   F("Main"),          F("/"),              // 0
[...]
      F("🔧"), F("Tools"),         F("/tools"),         // 7
    };

Had to change it into separate functions with a switch statement returning F() macro defined strings.

After changing all of these, I ended up with this error:

c:/users/gijs/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: address 0x3fffe008 of .pio\build\debug_custom_ESP8266_4M1M\firmware.elf section `.bss' is not within region `dram0_0_seg'
c:/users/gijs/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: address 0x3fffe008 of .pio\build\debug_custom_ESP8266_4M1M\firmware.elf section `.bss' is not within region `dram0_0_seg'
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\debug_custom_ESP8266_4M1M\firmware.elf] Error 1

Only when throwing out almost all code from my sketch, I was able to compile it wilt SD support enabled.
But I doubt it is useful, since you don't have any memory left.

Without SD support:

DATA:    [=====     ]  52.0% (used 42572 bytes from 81920 bytes)
PROGRAM: [=======   ]  69.0% (used 616020 bytes from 892912 bytes)
Creating BIN file ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.bin" using ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.elf"

With SD support:

DATA:    [========  ]  84.7% (used 69356 bytes from 81920 bytes)
PROGRAM: [=======   ]  70.9% (used 632760 bytes from 892912 bytes)
Creating BIN file ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.bin" using ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.elf"

That's an increase of over 26 kByte RAM usage.

I'm not entirely sure the issue with the __FlashStringHelper related code has anything to do with this, but maybe it does somehow prevent strings to be stored in flash?

[TD-er]

Just an update.
Simply including the file SD.h is enough to run out of RAM.
A user of my project also did try compiling it in ArduinoIDE and got the same issue (running out of RAM)

[earlephilhower]

Can you provide a simple MCVE, @TD-er , which shows this? That would let me reproduce it locally and make fixing things much faster.

[TD-er]

OK, took a while to come up with a proper MCVE.
But I think I got one and you need to do a bit yourself ;)

#include <Arduino.h>

#include <SD.h>

void setup() {
//  String foo = F("Lorem ipsum ...., euismod tortor.");

}

void loop() {

}

I used 5 paragraphs of Lorum Ipsum as one-liner in the flash string. (3229 characters, used 2 spaces between the paragraphs)

What happens is this: (core 2.6.0)

#include <SD.h>
String foo 

DATA:    [====      ]  38.1% (used 31216 bytes from 81920 bytes)
PROGRAM: [===       ]  26.5% (used 277132 bytes from 1044464 bytes)

#include <SD.h>
//String foo 

DATA:    [===       ]  34.2% (used 27984 bytes from 81920 bytes)
PROGRAM: [===       ]  26.2% (used 273892 bytes from 1044464 bytes)

//#include <SD.h>
String foo 

DATA:    [===       ]  34.0% (used 27844 bytes from 81920 bytes)
PROGRAM: [==        ]  24.3% (used 253744 bytes from 1044464 bytes)

//#include <SD.h>
//String foo 

DATA:    [===       ]  34.0% (used 27844 bytes from 81920 bytes)
PROGRAM: [==        ]  24.0% (used 250320 bytes from 1044464 bytes)

It looks like all flash strings are now in memory, as soon as you include SD.h

[earlephilhower]

Cool. That's very weird and I'll definitely look into it!

[TD-er]

I just updated the order of the outputs, to make it more logical.

[TD-er]

Just one thing that looks strange while browsing through the includes from SDFS.h.

The file assert.h does not have #ifndef... #define... #endif wrappers.

Probably nothing to do with this, but it also doesn't help compilation either I guess. (although it does include pgmspace.h )

[earlephilhower]

Ah, able to repro and narrow it down.

The issue is actually caused by including the real file from SdFat library!

For some reason he has this snippet in src/syscall.h
````

ifdef ESP8266

// undefine F macro if ESP8266.

undef F

endif // ESP8266

````

I'll pull it out (we already have some local changes for SPI compat) and do some testing...

[earlephilhower]

Updated the title, @TD-er , just to make it clear when the PR comes in what the root issue was.

[earlephilhower]

Also, about assert.h: That's coming from the GCC toolchain, basically, and I only added the pgmspace.h include and did minor tweaks to the #define. I think it's fine as-is, because that's what all other OSes and chips are doing (well, anything using GCC...)