Espeasy: Create a "minimal" build for 1M versions with OTA support

[TD-er]

Based on the discussion here: https://github.com/letscontrolit/ESPEasy/issues/303#issuecomment-367985783

Just a (very) small subset of devices and perhaps also a matching SPIFF filesystem size.
Also the unneeded libraries should be stripped.

Total size of 2-step OTA flash firmware + this build + SPIFF should be smaller than 1M.

Included devices:

• switch
• some temperature sensors standard compatible with Sonoff
• some of the sensors available in Sonoff's
• VCC defined???

Builds for:

• ESP8266-1M
• ESP8285-1M

At first skip PUYA-support until it is clear OTA on PUYA will work.

[soif]

I have a working Small OTA firmware compiled + source at:
https://github.com/soif/EspBuddy

[ToltingColtAcres]

I pulled the EspBuddy repo today and attempted to build the OTA firmware. While it compiles, when I flash it to my Sonoff 4CH Pro (ESP8285-1mb) (both OTA via ESPEasyMega firmware, and via flashesp8266), it will not boot. I can pull the EspEasy repo, build, and flash fine.

[TD-er]

That was also reported on the forum
Apparently not for all 4ch pro models.

[uzi18]

Maybe EspBuddy is not compatible with actual configuration?
@soif possible??

[soif]

EspBuddy itself is just an easy to use wrapper, so there should not be any compatibility issues, while the intermediate OTA firmware which I have built MIGHT have issues.

while I compiled it only for ESP8266, I had some positive feedback from one ESP8265 user, and i also succeeded on an ESP8265 that I own (cant remember the exact device, but not a 4CH relay for sure).

But i would really prefer if the intermediate OTA firmware would be maintained by the official ESPEasy team, rather than me : i'm far from an expert in the flash parameters field :-)

[uzi18]

@soif ok, but You didn't explain how it work.

[soif]

I don't understand : Could you please detail your question ?

[uzi18]

I mean You did not explain how this EspBuddy and intermediate OTA works, but I think already figured out, by source reading.

[uzi18]

@TD-er when using intermediate OTA, ESPEasy.bin can't exceed 615kB

// 1MB-128k SPIFFS
>>> 1024/8*7
// availabe flash:
896
// - intermediate OTA
>>> 896-281
// - max size of ESPEasy.bin
615
>>> 

[soif]

ESPeasy has many commands, but i guess you're only talking about the OTA upload feature:

Here is how it works:
1) (Optionally) build the wanted firmware using the platformio command
2) if configured, upload the intermediate firmware using espota.py, wait esp is ping-able after reboot
3) upload the final firmware using espota.py

_adding the -v option will show you ALL commands involved_

BTW the espota.py feedback message is often leading you to wrong conclusions.
ie, if you upload a firmware that is TOO big, rather than saying something like "file too large", it still stay on something like "Waiting for OTA upload". 👎

_I would certainly add a size check in EspBuddy itself to fix that, if you could tell me the EXACT formula to calculate the firmware maximum size for 1M, 2M and 4M boards...
I mean what is the formula to get the free firmware memory . is it always FASH/8*7 ?_

[uzi18]

It is true for 1MB (8Mbit flash and we use 128kB spiffs), for 4M it is 4096/4*3.
To know how mutch flash available substract intermediate ota binary size.

[soif]

Thanks, I get the idea.

Are those number exactly precise to the bit, ie
$available = (4*1024*1024/4*3) - FileSize(ota_firmware) ;

or is there some extra bits reserved for other thing like boot, etc...
_I want the test to be as precise as possible_

[uzi18]

I'm not sure about extra bits, but we can try as described above.
Arduino eboot is integrated into bin file.

@TD-er @Grovkillen @psy0rz: why we build puya image as "dev" (661kB bin) and not normal (601kB)?
Why we build puya in general? With my puya_v2.patch for core/Esp.cpp it will detect flash and use puya workaround so special build is not needed anymore. Did You tested it?

[Grovkillen]

@uzi18 I haven't tested that (didn't even remember seeing it).

[TD-er]

I haven't tested it, since I've tried (and failed) on reliably building my own core lib.
I was never sure it was actually using my patched code.

We will change the builds in the near future with the defines made by @soif so we should also look into that I guess.

[Micha-he]

I've tested uzi18's V2-patch with source from 8th march. It seems that it work on PUYA too.
I'll do another test with the new source in 1 or 2 days and report here.

[susisstrolch]

What is the minimum set of plugins required in Mega-Branch?
I've tried to build for sonoff_th (ESP8285) with
_P001_Switch _P004_Dallas _P005_DHT _P013_HCSR04 _P026_Sysinfo _P028_BME280 _P033_Dummy
All Web Pages but the Device Page are working.
Hitting the Device Page leads to an immediate reboot of ESP

I've tried the mega_20180316.

[susisstrolch]

Sorry, been in a rush. More details:
last working version (upstream, custom compile) was mega-20180312.
I build by preparing the sources via a .sh script (see https://github.com/susisstrolch/ESPEasy/blob/mega/buildPluginDotH (via Arduino IDE), to reduce the num of Plugins to an absolute minimum.
With this kind of bare build, based on mega-20180312, the "Device Page" is visible w/o any problems.
Switching to mega-20180316 doesn't show the "Device Page" anymore.
For me it looks like there are some unresolved dependencies handling the "Device Page".
Interestingly enough, compiler/linker don't show any warnings/errors.
Symptom is the the same for SONOFF Devices (1MB, ESP8285) and WeMOS D1 pro (4MB, ESP8266).

[TD-er]

@susisstrolch Around that date, the ESPeasy.ino file was split.
The globals were split into this file: https://github.com/letscontrolit/ESPEasy/blob/mega/src/ESPEasy-Globals.h
In the last line of your script, you do a sed on ESPEasy.ino, but the line you want to replace has been moved to this ESPEasy-Globals.h

[susisstrolch]

Yep - found this issue meanwhile and fixed the scripts.
Looks much better now...

[Oxyandy]

susisstrolch
Can you do me a favour please..
Point me in the direction of a minimal build pre-compiled firmware
I want to use rules as timers On clock & GPIOs with simple a switch as triggers in rules on a Sonoff Basic, nothing else needed
I will flash it and see how some of my 'ongoing' issues behave
Thanks

[susisstrolch]

Not really shure which plugins you'd need...
clock is part of core, the GPIOs may be handled via "Switch". At least I did it this way with my SONOFF - defining the GPIOs as type Switch. It only gives some strange behaviour (oscillating) if you use "inverted" switches.
For this case you'd run my script "buildPluginDotH" (take the current from my repo) and simply change "// #define _P001_Switch.ino" to "#define _P001_Switch.ino".
So you'd should have ESPEasy with only a switch device.

[Oxyandy]

It only gives some strange behaviour (oscillating) if you use "inverted" switches.
Interesting, something to try today...
I have been beating my head against a wall trying to do this:
Turn on pump from internal rule for 5 minutes..
Water goes to tray, tray has a float switch
When float switch rises is goes from NC to open, rule triggers stopping timer / pump
Float switch lowers, returns to NC, takes about 2 seconds..
That's the main work..
The bonus, some data reporting so if away, via the internet I can check on things..
Now you would think this is possible with a Sonoff, a pump, a switch running ESpeasy firmware
but ahhh nope.. there are times my switch gets stuck at the end of cycle as '1'
The next cycle that '1' will mess up everything - switch is fine of course...
Going crazy here

[Oxyandy]

Ok cloned Repository, now open in Atom
buildPluginDotH open in Notepad++ (not sure how to run your script ?)
Not seeing // #define _P001_Switch.ino
tiny bit of help, please

[TD-er]

That script he used was a bash script. You typically run that in Linux.

[Oxyandy]

damn see.. windows environment here, hence 'do me favour.. "build me a firmware...."
haha

[TD-er]

Not me, it has been very late the last few nights in a row. I'm off to bed now.

[susisstrolch]

There's a pretty nice tool for Windows - called MobaXterm, which has a linux like environment. It has "bash" and also all the nifty linux tools like sed, grep etc...
Oh... and this ugly bad black window with text only input...

Tell me which plugins/controllers you need - and which base (SONOFF, WeMos...)

[Oxyandy]

TD-er, if you read my post I said float switch was not responding... not showing in log..
Well I was flashing new firmware, building the unit with case etc - final unit..
I gave up doing that.. bare circuit board hanging in the air now...
Anyway I manually started pump, manually raised float, nothing - pump kept on going didnt it..
seriously.. !
removed cables from board..
manually started pump & 'then' bridged GPIO14's header pin with ground
with a jumper cable between the 2 header pins..
omg, pump just kept on running...
Removed jumper cable between the header pins GND & GPIO14
Powered Sonoff down, tried again - now it responds properly,
-inappropriate swear words go here-
sob, sob sob...
telling ya, the code for the rules is borked somewhere..

[susisstrolch]

But therefore you should open an own issue. I'll accompany you with logs from oscillating inverted GPIOs

[Oxyandy]

Just a sonoff basic, running rules & a switch
I use MQTT Domoticz - that's all
Cheers

[Oxyandy]

Before I open an issue, I need to study the cause to death, log data & fully get my head around what is happening, because it is intermittent & because I have these Sonoffs 'trying' to do a real world job..
bench testing is torture cause it doesn't happen often enough

[susisstrolch]

@mailbox

[Oxyandy]

If I try update a sonoff with a Mega release it wipes the settings & doesn't update..
wonder how I would go with this ?
Backing up settings OTA then going to give it a try to OTA update with 'minimal' firmware
Thanks @susisstrolch - much appreciated

[susisstrolch]

You must do the two-step update...
First flash the ESPEasyUploaderMega then flash the Mega image...

[Oxyandy]

Thanks, I actually haven't done it successfully before, usually I do not mind via cable,
but my patience is getting thin & the fun is wearing off..
I will do exactly what you suggest thanks, you just prevented me from another lesson learnt the hard way
cheers

[uzi18]

@Oxyandy please don't mix problems in this issue.
If You got problem not related to this one in topic please go to forum / open new issue.
We know about some issues with inverted state of switch, just use as normal and will be fine.

[Oxyandy]

@uzi18 - it was not my intention, I have been discussing an issue with @TD-er
and had simply made a revelation that the rules are actually stopping working - except for on Clock
I wanted to get that message to TD-er once after MANY hours logging & watching and I had finally became aware of it..
I do not have TD-er on any personal messenger and considered it valid enough to get this through to him ASAP - What a shame I do not use inverted state (inverse logic)
I wasn't going to open an "Issue" until I was 110% fully aware of the issue & have all the experience, facts & data - cause I am sure people that start "User Error Issues" are even more annoying than me posting here, sorry I have upset you in any way.

[TD-er]

On the forum it was again made a bit actual again, to make a minimal version.
And looking for "minimal" led me to this topic again :)

So, maybe good to get some proper list for a "minimal" build?

Quote from Waspie:

for me, what covers most of my usage is: switch (obviously), dallas (probably add DHT?), mqttimport, sysinfo, dummy, mcp23017 (probably pca too for people that use them instead), sr04, and TSL2561
Slightly more? maybe oled and neopixel stuff

I dump all the controllers but Openhab (probably should have Domo). I also get rid of the 2 notifications.

[Grovkillen]

This one is added #1989

[khturnings]

      You must do the two-step update...

First flash the ESPEasyUploaderMega then flash the Mega image...

Thanks for the 2 step update but can the wiki be updated with instructions on how to do this - I have tried a number of ways and have yet to get it to work.

I think it is along the lines of this -

1. Connect TTL to ESP8266
2. Use Easy ESP Flasher to flash ESPEasyUploaderMega with NO post flash action
3. Disconnect and reconnect (my TTL does not have a reset button)
4. Use Easy ESP Flasher to flash ESP_Easy_mega-20181109_minimal_ESP8266_1024_OTA.bin enabling post flash
   ….. Post flash does not work for me
   …..ok use webpage config (https://www.letscontrolit.com/wiki/index.php/EasySetup)
   ….. Access point is call 'Minimal 1M OTA_0 but password 'configesp' does not work???

Thanks