Nodemcu-firmware: `node.stripdebug()` has no effect on `node.compile()`

[devsaurus]

As discussed in #2429 - IMO an inconsistency between doc and code behavior.

Expected behavior

As stated in the doc for node.stripdebug():

Controls the amount of debug information kept during node.compile()

Actual behavior

Whatever setting I choose for node.stripdebug(), the compiled .lc file doesn't include debug info. I.e. it unconditionally strips (all?) debug info from the .lc file.

Test code

Simple script.lua:

print("line 1")
print("line 2")
print("line 3"..ko)

When run as Lua source while ko is `nil:

script.lua:3: attempt to concatenate global 'ko' (a nil value)
stack traceback:
    script.lua:3: in main chunk
    [C]: in function 'dofile'
    stdin:1: in main chunk

When compiled to .lc, ko is nil:

?:0: attempt to concatenate global 'ko' (a nil value)
stack traceback:
    ?: in main chunk
    [C]: in function 'dofile'
    stdin:1: in main chunk

The error message doesn't change for whatever stripdebug setting is applied before node.compile().

We should either change the API doc and remove the relation to node.compile() or fix node.compile() to comply with the documented configuration effect.

NodeMCU version

Current dev.

Hardware

Some ESP8266 NodeMCU clone.

[TerryE]

Yup, we should make the compile option strip-aware

[ryanplusplus]

@HHHartmann watch this issue for a solution

[ryanplusplus]

In case it is helpful, I confirmed that for all values of LUA_OPTIMIZE_DEBUG (including undefined) all debug symbols are stripped and the binary that results is identical to the size when compiling with luac.cross -s.

[ryanplusplus]

This applies to both dev and dev-esp32.

[TerryE]

@ryanplusplus, LUA_OPTIMIZE_DEBUG is part of my LCD patch (see our online documentation) and defines the default value for the node.stripdebug() setting. To understand how the two debug stripping mechanisms work, you need to understand how the Lua compilation process works.

• All of the Lua loaders ultimately call ldo.c:f_parse() with the Lua stream; this checks the head of the stream and either calls the binary loader, lundump.c:luaU_undump() if this contains the Lua binary signature ("\lparser.c:luaY_parser() to compile a text stream.

• The standard parser by default adds local, upvalue and lineno metadata to facilitate debugging. These take up RAM and aren't needed for execution, and in fact the reduced debug library used by NodeMCU _only_ uses lineno metadata and then only for error tracebacks.

• The node.compile() patch and the luac.cross compiler use luaY_parser() to load the Lua into RAM in binary and then immediately call luaU_dump() to serialise the RAM structures for the function with the stripping parameter set to true (this is also what luac.cross option -s also does). This strips out any remaining Lua debug metadata.

The objective of the LCD patch is to remove (most of) the debug RAM overhead during the compile process. It does this by immediately stripping out the local and upvalue metadata and using byte run-length encoded method of storing line-no info. This last uses 1 byte per source non-blank source line rather than 1 word per Lua VM instruction, reducing the debug overhead typically by over 10×. (Since this info is only used during error conditions to generate line numbers, adding a few µSec to decode the packed line number info isn't really an issue.)

What I need to do is to make a small change to node.c:node_compile() to add an optional parameter to set the stripping parameter and update the node.compile() documentation accordingly. :smile:

[ryanplusplus]

@TerryE thanks for the info and thanks for all the work you've done!

[TerryE]

In fact there is an interesting side effect of the LCD approach. luac.cross (and luac) call the listing function (the -l and -l -l options) on the loaded function. So if you use both to compile and list a Lua source then the former show the number of locals and upvals as zero and the latter doesn't which is a bit of a PITA, but not worth redoing, IMO. These counts are only used for debug info. I'll sort this out with Lua 5.3.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TerryE]

I've fixed this with Lua 5.3 but won't bother with 5.1 as we've only had this one report.

[ryanplusplus]

@TerryE thanks for the update. I've been away for a bit and didn't know so much progress had been made on 5.3. Where is the Lua 5.3 work available? A quick check seems to indicate that the latest master and dev branches still have 5.1.

[ryanplusplus]

In case anyone else is curious there's some detail here: https://gist.github.com/TerryE/646407100785fb00c2cbddbb0e8a2b61

[TerryE]

Where is the Lua 5.3 work available?

@ryanplusplus All of the related issues and PRs have the Lua53 label, and specifically PR #2912 includes a reference to my dev-lua52-2 branch which contains the Alpha code. I will be adding a couple of major commits to this in the next few days. I don't want to merge alpha code into dev just yet.

I've also got a bunch of updates to do to the gist discussion paper that you referenced.

[ryanplusplus]

@TerryE Thanks!

[TerryE]

@devsaurus Arnim, I have added the extra parameter for Lua 5.3. Is it worth backporting this into Lua 5.1 to make this behaviour consistent?

[TerryE]

Fixed in #3193