Nodemcu-firmware: Additional documentation for Lua 5.3

[TerryE]

We have quite a few gaps in our documentation set that moving from Lua 5.1 to Lua 5.3 underlines. We really need to plug these before we can even consider moving the Lua 5.3 from Beta to being formally recommended. These include:

• The Lua 5.3 Reference Manual (LRM) is a pretty concise yet formal and complete reference for the Lua language and its implementation. (It's still around 7,500 lines long). We don't have any parallel formal definition of how NodeMCU Lua varies this LRM. IMO, we need one and the easiest thing to do is to mirror the same style and structure. I've started drafting a NodeMCU LRM. This gist is a rough cut work in progress, and which I hope to finish tomorrow.

This plus the standard module documentation is the minimal set that should provide any developer with what is needed to write Lua code for the ESPs and develop C libraries. That being said, I have been going through the LRM in detail for what must be over 5 tears and yet I am still discovering subtleties that I have missed on previous readings.

• Our ReadTheDocs online material contains a lot of Lua 5.1 build-specific content. We need to go through this and either remove the specifics or put in a comparison table of 5.3 vs 5.1 changes.

- Lua 5.3 is really quite big change so we could do with some detailed release notes which emphasise what has changed.

Lastly we have a longer term documentation objective that we need to make progress on. I don't think that it is practical to consider this as a precursor to formally recommending our 5.3 engine.

• The LRM is supplemented by a more informal Programming in Lua [PiL]. The 1st edition is available free online. The forth edition is aligned to Lua 5.3 and available in paperback and eBook and this goes far more into techniques and worked examples. This is a lot more developer friendly that the LRM. Again, I think that we need to consider how we provide the additional context of how to use NodeMCU Lua, but this is a non-trivial task for a professional author -- which I am not.

I suggest in the first instance that we try to provide this material more as a community effort within the NodeMCU wiki as a collection of specific articles. Maybe when they are sufficiently mature, all or a subset of this material could be promoted to our ReadTheDocs online material.

Thoughts? Reactions?

[KT819GM]

@TerryE small example list of Lua 5.1 build-specific content would help to identify magnitude of work needed to be done and maybe, it would be worth to freeze current 5.1 and go with 5.3 only (without looking at backwards compatibility)?

Only mine, as end-user opinion.

[TerryE]

Actually doing this documentation exercise has proved used and I've realised that the NodeMCU additions need a little more clean up. For example the public C API is contained in lua.h and luaxlib.h (which itself includes lua.h) so the _only_ lua header that is needed. The other lua headers are really internal to the runtime and should not be avoided. We do have existing modules that break these guidelines and need cleaning up.

I continue to work on this documentation as a background activity over this next week.

[TerryE]

In drafting this reference, I've realised that I need to decide: do we look forward, or do we face backwards. By this I mean that there are subtle differences two between the versions of the APIs in the two LRMs 5.1 and 5.3. An example is that in 5.1 all table access routines push the value onto the stack with a void return; the 5.3 equivalents return the type of the value pushed, so in Lua 5.1 compatible code you might write:

   lua_getglobal(L, "LFS");
   if (lua_type(L, -1) == LUA_TTABLE) {

This is a very common pattern in modules, so Lua 5.3 enables you to fold these into a single

   if (lua_getglobal(L, "LFS") == LUA_TTABLE) {

and halving the API calls needed. Yes, the former still works in 5.3, but I don't want to say to developers who are developing new modules "use LRM 5.3, but don't use \ if you want the module to compile in Lua 5.1", or have to say "this module only compiles in Lua 5.3" when there isn't a strong reason for this.

As it turns out, adding the type return to the 5.1 API calls involves about 8 × 2 line changes and is backwards compatible, so I will just do this as part of this PR.

Another example is that the strip debug function is a NodeMCU addition to the public API, but I called it luaG_stripdebug(). No, it is public and so needs to be renamed to lua_stripdebug().

[TerryE]

The gist draft has been updated.

[TerryE]

@nwf @HHHartmann @marcelstoer, this draft NodeMCU reference is probably now at a stage where it is worth an independent read and review if anyone cares to. The format and style mirror the Lua Reference Manual. Thanks guys.

[nwf]

I think this is in keeping with what you've been doing, but:

• I'd vote for us to use 5.3 features whenever possible, because we think that we are going to eventually ditch 5.1, barring some catastrophe.

• We have, as we all know, very limited person-power here, so "write for 5.1 and then update to 5.3" is probably not a great plan (though it could, of course, work).

• Wherever minor divergences like this arise, perhaps we should back-port deltas to our 5.1. The getglobal in particular seems sufficiently straightforward to emulate, even if the result is that the 5.1 modules are slightly less efficient from spurious calls to lua_type() internally,

[TerryE]

I think this is in keeping with what you've been doing, but:

Only comment is that _but_ should read _and_ :smile: in that we agree on this. See specifically the section New Lua 5.3 features back-ported into the Lua 5.1 API, and my thinking here is that we can write new modules largely in 5.3 style.

I've already gone through the existing modules and made the minimum changes to ensure that they compile clean in both. With this back port work done, then we can also have the option as we clean up old modules of moving them to a more 5.3 style without breaking 5.1 compatibility.