Nodemcu-firmware: Enduser setup (EUS) - Provide more hooks for code

[sztanpet]

So my use case is this:

I want to use EUS for board set-up like it was meant to be used, but I also want extra form fields (a human-readable identifier for the board and a secret code). I know I can edit the HTML, but I also want to get at all the values entered on the form on the lua side.

For more background: I am planning a little thermostat project where the board sends temperature and humidity readings identified by the human-readable id entered on the EUS form, and the data HMAC-SHA1 signed using the secret also from the EUS form). Trying to make it as easy to set up as possible.

[jmattsson]

Hmm, not a trivial thing to add, but could certainly be useful. @robertfoss @jfollas what do you guys think?

[jfollas]

First thought is that the query string (i.e., the postback data) could be passed out on the success callback for further parsing by Lua.

Since the callback is invoked after the request has been processed, the query string would need to be retained in state. This makes the assumption that only one client at a time would be trying to submit the form data to setup the wi-fi (I think that's an assumption anyways).

[sztanpet]

it would be even more lovely if there was a validation step too, so user code could cancel the set-up because the extra form fields are invalid for some reason.

[robertfoss]

I think this sounds pretty doable, might be a bit of work though.

[davehun]

Would be handy for some other other moduels e.g. mqtt

[pastukhov]

Universal solution, i think

The idea is: to be able to store additional fields of eunduser setup form. For example with wifi_ssid=blablabla&wifi_password=123&custom_config={option1:1,option2:2}, urlencoded and stringifyed. Then the module should save it to custom_config.json or similar.

[pjsg]

How about a way for the caller of EUS to be able to provide a chunk of HTML that is inserted into the form. This could be implemented by having the form (on being rendered in the browser) calling back to the nodemcu to get the extra HTML. This would then be inserted into the form. The author of this extra HTML can add javascript onclick handlers etc to validate that the field data is correct.

In this way, the EUS code would only need to have a little extra logic (to make another http request to get the extra HTML and to insert it into the form).

[sztanpet]

I would leave the HTML building out entirely, just prepare your own HTML for the entire form (this could be made a lot easier with a simple little web service for the easy cases to assemble, minimize, compress, etc). It will never be flexible enough for everybody otherwise.

Pure client-side only validation is never the answer, but I realize there are limitations so if this is the path chosen just please make sure to have a scary warning about it in the documentation. :)

[rvowles]

OK, so I built this, should I start getting it ready for PR quality?

[rvowles]

My operational premise for the validation function callback is that a user can pass a table back, and that can hold three values (one of which is mandatory if they want to reject the configuration attempt) - [status, body, content_type]. My C is pretty rusty so this extra functionality may take another week, but its a great idea!

[rvowles]

Ok, I believe I have finished the code for this @marcelstoer (including the validation callback, allowing the lua to send a custom response back to the javascript) - I still have no confirmation that anyone is interested? The next step would be to reset the debug steps, reformat to the same format, rebase against dev, squash all the commits and write the documentation to get it ready for a hard-line review. I'd prefer to write up a file (enduser_setup-test.txt for example) indicating my testing strategy for review as well.

https://github.com/nodemcu/nodemcu-firmware/compare/master...rvowles:esp-auto-config?expand=1

[pjsg]

I'm certainly interested in this....

[rvowles]

@pjsg i can put up a link to a compiled firmware image if you are willing to beta-test it?

[rvowles]

enduser_setup.start()

Starts the captive portal.

Note: Calling start() while EUS is already running is an error, and will result in stop() to be invoked to shut down EUS.

Calling EUS start() when there is already a valid network connection will cause EUS to call stop() when
it detects a successful connection. It is recommended that you force disconnection from the
wifi (such as wifi.sta.disconnect()) before starting if you need setup to recur (such as detecting
a new field is missing if you are doing OTA (over the air) updates).

Any extra parameters (field/value combinations) passed to the esp will be saved in a file called enduser.json in json format.

Syntax

enduser_setup.start([onConnected()], [onError(err_num, string)], [onDebug(string)], [onValidation(table)])

Parameters

• onConnected() callback will be fired when an IP-address has been obtained, just before the enduser_setup module will terminate itself
• onError() callback will be fired if an error is encountered. err_num is a number describing the error, and string contains a description of the error.
• onDebug() callback is disabled by default (controlled by #define ENDUSER_SETUP_DEBUG_ENABLE in enduser_setup.c). It is intended to be used to find internal issues in the module. string contains a description of what is going on.
• onValidation() callback if there are fields other than wifi ssid and password passed. expects either no return or a table
  consisting of a status (mandatory), content_type and body (both optional). This method is optional and
  extra fields will still be stored in the enduser.json file regardless of whether they pass
  validation.

Returns

nil

Example

(no extra fields expected)

enduser_setup.start(
  function()
    print("Connected to wifi as:" .. wifi.sta.getip())
  end,
  function(err, str)
    print("enduser_setup: Err #" .. err .. ": " .. str)
  end,
  print -- Lua print function can serve as the debug callback
);

Example

(using validation)

enduser_setup.start(
  function()
    print("Connected to wifi as:" .. wifi.sta.getip())
  end,
  function(err, str)
    print("enduser_setup: Err #" .. err .. ": " .. str)
  end,
  print,
  function(t)
    print "validation called - fields passed are"
    for k, v in pairs( t ) do
      print(k, v)
    end
    if t["location"] == nil or t["location"] == '' then
      val = {}
      val["status"] = 400
      return val
    end
  end
);

To try this example, you can do the following things:

• download the default enduser_setup.html file from NodeMCU Github
• modify it so it accepts a 400 response and adds in the location field. (see below)
• upload the file to your esp (I use the Upload feature from ESPlorer)
• run the above example (forcing wifi disconnect if your esp is already wifi connected)

The HTML changes for the enduser_setup.html file are:

                <input id=wifi_password type=text autocorrect=off autocapitalize=none autocomplete=off placeholder=Password />
                <!-- add the following line in, don't waste space with this comment -->
                <input id=location type=text autocorrect=off autocapitalize=none autocomplete=off placeholder=Location />

    if (s == 400) { // note 400 check must come first
       $('#st').innerText = 'Please specify location';
      cur('#f1');
    } else if (s != 200) {
        $('#st').innerText = 'Awaiting Status (' + s + ')';

[rvowles]

I have debugging still turned on, so if people who are interested could please download and flash and try I would appreciate the feedback.

https://drive.google.com/drive/folders/0B2kixRhS_SOGc1J1QTFqQl9ZQjQ?usp=sharing

[gsker]

Thank you for working on this.
I flashed your firmware and made the modifications you indicated above, but that didn't work.

If you would include a complete html file that works for you instead of snippets I would test that.

But it did pass the extra field! I wrote a simple HTML page with a GET:

```

Connect device to your Wi-Fi

It displayed  this on the lua console:

``` 858:    enduser_setup_http_handle_credentials
875:    enduser: starting checking for parameters in query string
754:    enduser: opening enduser.json for write
794:    ssid
795:    myssid
547:    enduser_setup_http_urldecode
547:    enduser_setup_http_urldecode
794:    wifi_password
795:    mypassword
794:    emailadress
795:    gerry%40home
547:    enduser_setup_http_urldecode
547:    enduser_setup_http_urldecode
794:    submit
795:    Submit
547:    enduser_setup_http_urldecode
547:    enduser_setup_http_urldecode
832:    enduser: calling lua for validation
validation called - fields passed are
emailadress     gerry@home
ssid    myssid
submit  Submit
837:    enduser: checking if lua table was returned
1136:   enduser_setup_serve_validation

But I don't think it connected......

This is very serendipitous as I was just starting on a little project that would be really aided by the ability to pass some more fields during configuration. Thanks for your work on it.

Gerry

[rvowles]

HI @gsker I am using
enduser_setup.html.gz

I also have a "common.lua" that I use on startup to ensure we have an enduser.json file and if not, triggers the setup.

[rvowles]

If it isn't clear, load that file onto your file system on the esp8266.

[gsker]

Thanks. I'm not sure why mine didn't work, but yours did just fine. Even after I changed the name of the field and added another field.

I'd rather not have a file written by default. I would think that making assumptions about a file system would get us in trouble. But I'm glad to have the feature and will incorporate it into my custom build and see how it goes.

Gerry
edit: but now that I play with it in my custom build and include the cjson module I realize that it's exactly how I will use it anyway so I really like it. :-)

[marcelstoer]

I'd rather not have a file written by default. I would think that making assumptions about a file system would get us in trouble.

Your device won't run anything useful if there's no init.lua on the SPIFFS file system at boot time. So, I guess you can safely assume a file system. Writing to it may still fail of course.

now that I play with it in my custom build and include the cjson module

Side note: on dev CJSON was just recently replaced with SJSON. Will be on master with the next snap as well.

[rvowles]

@marcelstoer should i rebase onto it? I presume the answer is yes.

[marcelstoer]

If you're planning to contribute code in a PR then yes, rebasing is a good idea: https://github.com/nodemcu/nodemcu-firmware/blob/master/CONTRIBUTING.md#working-with-git-and-github

[rvowles]

Just to note, that CJSON (nor any json) is used in the PR. Thats only used when I was testing it in LUA.

[Jcrash29]

The last PR to attempt this code was closed without being accepted into Master.

I have attempted a similar code change based on the comments from the previous PR.

I'm looking for some people to test and review the code.

https://github.com/nodemcu/nodemcu-firmware/pull/2132

[gsker]

I modified my code which was using the old PR.

After adding my form field to the config structure and building the stringified json it worked just fine.

I was rusty since I've just been using it and not coding it for so long. The final result was the enduser_custom_parameters.json file with the json string in it.

[Jcrash29]

Is anyone here able to accept #2132?

[marcelstoer]

Finally landed with #2810.

[dama2009]

Oo:Opo