Nodemcu-firmware: RFC: Views on how to implement host-side provisioning

[TerryE]

I use my luaOTA provisioning system, but few others do. One of the main PITAs, IMO is that it requires a host-side Lua installation and for whatever reason most NodeMCU developers seem reluctant to do this. So I want to remove this requirement.

I am interested in views on possible approaches for doing this. I see there are two primary alternatives to my doing a minimal change and leaving this largely as a personal tool:

• Add the Lua File System and MD5 libraries to the app/lua/luac_cross build so that you can use the luac.cross -e option to execute the provisioning system. Having these has other uses: for example if you want to move small text files such as js, html, etc. into a resources module were these are string constants which can then be loaded into LFS, then having the LuaFS makes doing this as a -e script fairly straight forward as well.

• Adopt the esptool.py approach and reimplement the server in Python. It's a decade since I did any Python coding, but I am sure that I can rapidly dust this off. But I do this that this is a sort of "give up" statement if we end up writing server-side code in Python when Lua is equally as good.

I will listen to any views and close this issue in a few weeks.

Marcel, I realise that this doesn't fit the template but I think that the committers who end up closing most issues deserve a few liberties. :smile:

[jmd13391]

@TerryE, For quite some time now, I have been planning to incorporate your luaOTA into my NodeMCU / ESP8266 based product. I must confess that I got sidetracked with these plans the day you announced LFS (an absolute work of art BTW).

I have spent the last few years struggling over how to easily OTA update the thousand or so of my product out in the field and at the same time get my application+OTA code to fit on the ESP8266. Of course, LFS is the answer to my prayers - as I have reached a plateau with the overlay / module coding approach (mostly due to my coding inabilities).

For me, your luaOTA implementation would be most beneficial if it was structured in a way that allows me have units out in the field enter wifi.STATIONAP mode (units are normally in wifi.SOFTAP), connect to a predefined download repository on my company Internet website, pull update file(s) via a standard download protocol (http, ftp, etc.) and then execute the update. Now, I may be missing something here but seems to me roadblock to me utilizing your luaOTA provisioning system within my production environment model is the "host-side Lua installation" requirement (could that be one of the reasons "most NodeMCU developers seem reluctant to do this"?).

For what it's worth, my vote would be to structure luaOTA in a way that allows a device in the field to pull and execute the update autonomously from a website that is hosted by any one of the many mainstream hosting services (wix, squarespace, etc.).

Let me take this opportunity to thank you and the other contributors for all you have given to the NodeMCU project. I look forward to seeing LFS and the LFS version of luaOTA land in dev.

Cheers!

-Joe

[TerryE]

@jmd13391 Joe, feedback from active users like yourself is very useful in helping the developer group focus their priorities.

IMO, a real concern with IoT devices is security -- especially if they are provisioning from cloud-based services over the Internet. My personal approach to this is to isolate my own IoT devices on a control WiFi service and to use a locked-down RPi3 (clone) acting as a local host and bridge to any public services. Hence, my IoT devices cannot directly access the Internet, and conversely any Internet access to them or their data is through properly authenticated and trusted connections.

I realise that this might not be a viable approach for specialist IoT suppliers, but policies such as the Internet Society's IoT Trust Framework are essential, IMO, and yet still seem to be ignored by many mainstream IoT developers.

At its most basic, any provisioning service should use a properly authenticated and secure trusted path from the individual IoT devices to any cloud-based provisioning service which will involve some TLS-layered transport.

At a personal level, my work on this would be a non-trivial commitment, and what concerns me is that the ESP8266 is now a mature SoC, and modules based on its successor ESP32 part are now readily available, far more capable, and such ESP32 modules are only 3× the price of ESP8266 ones and this gap is rapidly closing. I am not sure how long ESP8266 parts will continue to be available mainstream. I had to solve a lot of fairly complex technical issues to realise LFS, and my instinct is that my next priority should be to finish my work on an LFS-enabled Lua 5.3 implementation for shared deployment on the two platforms.

So the scope of any LuaOTA work should be very limited.

[jmd13391]

@TerryE, I try to follow @marcelstoer's mantra of "1) Make it run. 2) Make it secure. 3) Make it fast. 4) Make it pretty. - IN THAT ORDER". Your points regarding Step # 2 are spot-on. Unfortunately, with OTA, I'm still stuck on Step # 1 :-(

Margins are tight enough that I am forced to wait until supply & demand does its thing to lower the cost of an ESP32 to the current ESP8266 neighborhood before I can justify it replacing the ESP8266 in our current products. Besides, I still see wet paint when looking at the ESP32... much like the ESP8266 was back in early 2016 when I first starting using it in production products. Needless to say, there were a LOT of challenges I had to overcome back then to produce a stable, fail-safe, marketable product based on the ESP8266 - and in doing so, most of the wet paint got all over me. This is not an exercise I wish to repeat. ;-) I recall reading somewhere that Espressif committed to a 12 year availability window for their ESP8266 module. Hopefully, for my ESP8266 based products, that's true.

"_my instinct is that my next priority should be to finish my work on an LFS-enabled Lua 5.3 implementation for shared deployment on the two platforms._"

Follow your instinct... it serves you well. ;-)

Cheers!

-Joe

[TerryE]

Given that that the compile and rebuild cycle for LFS is _so_ short (a couple of seconds to compile and reload and reload an absolute LFS region + the new LUA_INIT_STRING config directive:

#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)"

or a function variant with a SPIFFS fallback if the LFS isn't loaded such as

function() local f,a=node.flashindex'_init'; return a and dofile'reinit.lua' or f() end

so that if the LFS isn't loaded, then the startup falls back to executing the SPIFFS file reinit.lua.

I am now planning replacing my prod apps by a fully LFS packaged one, where SPIFFS is only used for volatile configuration and other application data and a recovery lua fallback in case there is a powerfail during LFS update.

In this case the application update now reduces to downloading a new LFS image and executing a flashreload. The one wrinkle that I was considering adding to the reload was to make the image file compressed (as this typically shaves ~40% off the image size to be downloaded and stored) and to uncompress this during reload. This would allow us to do an integrity / hash check so that only valid LFS would be flashed. You could pull the image from any suitable HTTPS cloud service if you want.

[joysfera]

Not sure if my amateur OTA implementation can add anything useful here but I'll explain it anyway. I prepare the *.lc files on one of my development ESP8266 (using esptool that uploads *.lua and compile on the ESP), then download them from the dev ESP and upload them to the provisioning server with apache web server. Later I'll let the ESP8266s in the field download the *.lc files using plain http get. This might sound cumbersome at first but I basically just mirror a working setup from one device to all others. Thus I don't need any Lua on the server, and I can use any public web hosting that offers downloading plain files (all do).

When I switch over to LFS I'll simply replace the bunch of *.lc files (that often fail to download completely) by a single LFS image that I am again prepared to prepare separately on/for my development ESP8266 first (and test it there) and then move on the public webserver.

So maybe this can inspire @TerryE a bit. Lua on the server might not be such issue...

[TerryE]

@joysfera, I assume that implement your own http stack in Lua rather than use the http library as the latter can't cope with large content gets. Have a look at my ftp example where I do flow control to prevent the server overrunning the ESP8266.

As I said, I am coming around to a keep-it-simple-stupid approach to provisioning lfs.img files, but with the luac.cross compressing the binary image with the loader automatically reinflating on the fly. So you'd use whatever your preferred poison to get the image from the server: http, https, ftp or whatever and load it into SPIFFS, then issue the node.flashreload() to reimage it. I might add an option to do a two pass reload: the first is just a dummy run to make sure that the img file is complete and the hash matches the content, the second can then do the reflash fairly safely before rebooting.

[nwf]

I also have a KISS "provisioning system" (if one will be generous with their vocabulary) that uses the little mockup of a telnet server I wrote to write files (often, lua source rather than cross-compiled byte-code) to the spiffs. It's all driven with expect around netcat host-side: https://github.com/AcmeTensorToys/esp-lua_core/blob/master/host/pushvia.expect, which, because it's waiting for prompts from the nodemcu device, is sort of naturally rate-limited. The server itself is https://github.com/AcmeTensorToys/esp-lua_core/blob/master/telnetd/telnetd.lua but loads https://github.com/AcmeTensorToys/esp-lua_core/blob/master/telnetd/telnetd-file.lua to do the actual fs work.

I, too, have been meaning to replace this whole thing with @TerryE 's OTA world, but if another example of "what other people do" is useful, there 'tis. If not, please pardon the noise. FWIW, I have no objection to having a host-side lua environment (and, in fact, often prototype my nodemcu applications' logic under test harnesses host-side).

[TerryE]

See #2315 for discussions on luac.cross. This is all pretty easy for Linux and Mac users; a minor hoop for Win10 Pro users and a PITA for all other Windows users that don't want to run a Linux VM as we've got no WinX developer willing or with the skills to do the make on something like MinGW to build a WinX executable for those developers that want to use this.

[TerryE]

I have decided to go "thin server"; that is to use a standard webserver as the host-side delivery vehicle and drop the host-side Lua server entirely. Thanks for the discussion and feedback.

[devloperbra]

Terry,
Congrats by your great work for nodemcu.

I came from 8 bits microcontrollers and now I must to migrate to ARMs, IOTs
devices.

I work on a university, as a electornic technician.

What I am saying is, I am not a high level programmer.
Here, we ( I and other students) decided to use nodemcu and LUA.

But, after a deeper analysis, we noted if our IOT product must have OTA.
It's unacceptable to do one more step without OTA working.

So, we found your OTA example but we can't put it to work.

For now, we are loose couse we still don't have a practical solution. May
be we leave LUA cause the top priority is the OTA solution, not the
language or script.
(Personally I would like to do it with LUA, cause I already "sold" LUA for
my team and coordinator)

Once more, thanks for all nodemcu devs.
I hope become a contributor in a near future.

Em sex, 6 de jul de 2018 21:41, Terry Ellison [e-mail]
escreveu:

I have decided to go "thin server"; that is to use a standard webserver as
the host-side delivery vehicle and drop the host-side Lua server entirely.
Thanks for the discussion and feedback.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodemcu/nodemcu-firmware/issues/2408#issuecomment-403176722,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AfKNZFoRePblvReItdnWlM55dvFc1hQ2ks5uEAPNgaJpZM4U0kfK
.

[TerryE]

@develperbra, I'll be pushing an LFS based OTA solution within the next week. :)