Nodemcu-firmware: Missing documentation for LUA_FLASH_STORE

[rvalle]

Expected behavior

During firmware configuration
Should be easy to select the value for LUA_FLASH_STORE in user_config.h
Options and/or units should be clearly documented so that the setting can be used.

Actual behavior

current default setting is:
//#define LUA_FLASH_STORE 0x10000

Comments

0x10000 corresponds to 64kb in hex, What are the other possible options? Documentation reads that up to 256Kb of code can be run from the LFS, but screenshots show up to 128K.

With 4Mb of Flash I would definitely reserve maximum possible amount for LFS. But I just cannot find the right setting.

Datasheet specifies a theoretical limit of 16Mb.

Hardware

ESP12f
ESP8266

[rvalle]

I am happy to submit a PR for this docs, if someone can point me to the information.

[rvalle]

Oppps! just realized the 0x for hexadecimal number... little bit embarrassing.

Anyway, still don't know what the maximum I can use is. I am currently testing a 256kb LFS partition... see what maximum we can achieve.

[rvalle]

according to the ESP8266 datasheet:

_An external SPI flash is used together with ESP8266EX to store user programs. Theoretically speaking, up to 16 Mbyte memory capacity can be supported_

However I am hitting this limit when try to generate the img file:

#define LUA_MAX_FLASH_SIZE  0x10000  //in words

Which with 4 bytes words it is 256kb, which is consistent with a mention in the doc:

_The Lua Flash Store (LFS) patch modifies the Lua RTS to support a modified Harvard architecture by allowing the Lua code and its associated constant data to be executed directly out of flash-memory (just as the NoceMCU firmware is itself executed). This now allows NodeMCU Lua developers to create Lua applications with up to 256Kb Lua code and read-only (RO) constants executing out of flash, with all of the RAM is available for read-write (RW) data._

Why 256Kb, is it a hard or soft limit? That leaves more than 3Mb for filesystem, seems disproportionate.

[marcelstoer]

Why 256Kb, is it a hard or soft limit?

Sorry, don't know. One for @TerryE I guess.

That leaves more than 3Mb for filesystem, seems disproportionate.

That depends on your use case. Are you saying you'll be building a >256Kb Lua app?

[TerryE]

The 256 Kb was because the relocation scheme in the first LFS version replace any 32 bit address reference by an in place 16 bit offset + 16 bit link to next addr to be relocated where the offset was in words hence 64K × 4 bytes max offset or 256Kb. This is historic, but the ICache mechanism only maps a 1Mb window and the rest of the firmware is typically over 512Kb, so we couldn't relax up to 512Kb. At the time we only had ~40+Kb code _and_data, so I reckoned that 256Kb code + 40 Kb data would be a good step.

I've got a bunch of commits that I want to do into the next week or so, and if I get through the current world crisis, I can take this to the next stage.

[rvalle]

@TerryE thanks for the clarification, I noticed there was a cache mechanism in the chip, but missed the 1Mb limit.
I have reserved 512kb for LFS, but I share your point that 256kb are probably enough.
I have also seen that there seems to be now a partition table.
For OTA I was thinking of 2 LFS partitions one with Factory Init firmware and another with last OTA. Although I dont know if it is possible. We like to implement a mechanism to revert to Factory firmware, something like turning on the device with a switch pressed... and then it will go back to Factory firmware.
Anyway I am still getting familiar with everything here...
thanks for the info.

[VladESP]

Marcel, it would be very nice to have 256 kb LFS size and more (512 is a dream) on the site https://nodemcu-build.com. 128 kb is not enough for projects. Please add this option if possible.

[pnkowl]

Marcel, I would like to second the request for 256kb LFS size option on nodemcu-build. Also, out of curiosity, is it possible to reflash the firmware without requiring a reload of LFS and SPIFFS files. My attempts to flash a smaller firmware fail. Only flashing the same firmware is successful.

[marcelstoer]

There you go. I also updated the table at https://nodemcu.readthedocs.io/en/latest/lfs/#selecting-the-firmware accordingly.

[VladESP]

Excellent! Marcel, thank you very much!

[pnkowl]

Works great so far. I only had several LFS images slightly over 128k to test with. Unfortunately, the LFS Cross Compiler Web Service (https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/) has not been producing usable images for me since 26 April of any size (see ticket#3090, A Lua Cross-Compile Web Service Generates Unusable Images). Once again, thanks.

[VladESP]

Yes, https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/ produces a * .img file that does not work. Unfortunately.

[TerryE]

I'll check this out. Sorry.

[matzl]

However I am hitting this limit when try to generate the img file:

#define LUA_MAX_FLASH_SIZE  0x10000  //in words

Which with 4 bytes words it is 256kb, which is consistent with a mention in the doc:

Awesome, thank you for clarification. So am I right that:

#define LUA_MAX_FLASH_SIZE  0x2000  //32 KB
#define LUA_MAX_FLASH_SIZE  0x4000  //64 KB
#define LUA_MAX_FLASH_SIZE  0x6000  //96 KB
#define LUA_MAX_FLASH_SIZE  0x8000  //128 KB
#define LUA_MAX_FLASH_SIZE  0x10000  //256 KB

?

In general, you guys are doing a really great job, but so many times the documentation has many gaps (like in this case where the units are missing and no commented example is provided) which makes many things practically useless for people who don't already know what they are doing anyway. I hope you keep up the good work and also work a bit more on the documentation. <3

[marcelstoer]

Your list is correct. Please create a PR to improve the description in user_config.h.

so many times the documentation has many gaps

I hope you realize what you got yourself into now? Please list the many gaps (in separate issues) so we can address that.

[matzl]

Your list is correct.

Thanks for confirmation.

I hope you realize what you got yourself into now? Please list the many gaps (in separate issues) so we can address that.

Haha, yeah. I will, not today though.
One thing is the LFS manual. It took me days to figure out how it works even though I read all the related thing in the Manual and tried Terry's website (which unfortunately didn't work) but I think I got it now and I already started to write down some notes on what I did and which problems I had.

[KT819GM]

Exactly for these like things (hex / whatever) related I always tell to use modern IDE's like vscode, because integrations like C/C++ Intellisense can show decimal value when mouse is over it, though it does not make you smarter :) About LFS, current dev Getting started I find very informative.

[TerryE]

Your list is correct. Please create a PR to improve the description in user_config.h.

@marcelstoer, On this occasion, I disagree. LUA_MAX_FLASH_SIZE is a define constant internal to the Lua core component flash.c. It isn't intended to be changeable by end devs, because doing so could break stuff. The user configurable parameter is LUA_FLASH_STOREand this is already documented in getting-started.md though I don't think that this includes a statement about the current 256Kb limit, though the main documentation page does here. I am not sure how many times we need to repeat this.

@matzi, the "public" stuff -- that is generally available to Lua devs building custom firmware comprise the #define statements in the user includes in app/include. These are documented inline. We also have various parameters that can be latched in the _Reboot Control Records_ (RCR) at runtime such as to change the partition sizes and the startup command.

In general anything in the app/lua* hierarchy is more a case of _Beware! There lies demons!_ This applies to any of the 800 odd defines and other code in this directory tree. Actually our project additions are better documented inline than Roberto's standard Lua code, because standard Lua adopts a zero documentation policy, and anyone changing it is just expected to retro-engineer the bare source to understand what is going on. I've been doing this for about 5 years now and am still discovering nuances as you can see from some of the discussions on my recent RFC issues such as #3176.

Now to your specific challenge. What you seem to need is an LFS region of larger that 256 Kb to compile your applicaiton. Please remember that when I first developed LFS, we had to fit both code and data into ~40kB RAM so moving _any_ code into flash was a major step forward. The 256Kb limit came from a constraint of the address relocation scheme in encoding LFS images. Whatever the LFS region size is, the ESP8266 only maps 1Mb of flash memory into ICACHE buffered address space and both the firmware and the LFS has to fit within this 1Mb window, so at best we might be able to grow this by 50% with the Lua 5.3 engine.

To be honest you are the first developer to report running out of code space for an ESP8266 app. So :trumpet: :mega: :clap:

What you are discussing isn't so much a build issue, but a constraint of our Lua 5.1 architecture. What you really need to do is to raise a feature request saying "How can we have LFS regions larger than 256Kb?" or something like that.

[marcelstoer]

LUA_MAX_FLASH_SIZE is a define constant internal to the Lua core component flash.c. It isn't intended to be changeable by end devs, because doing so could break stuff. The user configurable parameter is LUA_FLASH_STORE and

Of course...my bad. I didn't pay attention to _which_ constant he actually changed/documented. Sorry. I have a feeling @matzl actually wanted to document LUA_FLASH_STORE as well. I don't see any problem documenting the supported values in hex and prose in user_config.h?

this is already documented in getting-started.md though I don't think that this includes a statement about the current 256Kb limit, though the main documentation page does

Yes, https://nodemcu.readthedocs.io/en/latest/lfs/#selecting-the-firmware lists the supported values which also implies what the current maximum is.

[TerryE]

Any multiple of 4Kb _up to a maximum of 256 Kb_ This last phrase could be added to the comment in user_config.h. Big boys and girls can use hex notation. This is standard Lua and C. The rest can use a calculator or do the multiply in their head.