Espeasy: Feedback required: New sets of plugins/controllers

[TD-er]

The last few weeks we're running out of the space available for the sketch.
The test and dev builds now approach their max size so something has to change.

I think the sets of normal, test and dev does not really make sense anymore.

We already have some builds specific for some use case, like specific hardware or just a single purpose.
For example the Sonoff POW as a specific build for off the shelf hardware and the IR builds for the IR RX/TX and AC control.

Maybe we can make more logical build sets.

For this we need some user input to get an idea of sensors often used together or so often they must be included in any build.

Other options are to disable some of the webserver parts if they are not needed.

So please let's get some proper lists of used plugins to redefine the build scripts.

[Domosapiens]

My applications are:

• Multi DS18B20 / BME280 Temperature measurement
• Energy calculation by DS18B20 delta Temperature measurement and Flow measurement plus PWM pump control
• Thermostat setting&control by Rotary encoder, MLX90614 IR and DS18B20 Temperature measurement and relay control

all supported by generic:
LCD/OLED display, Touch,IR,RCWL detection, Dummy Devices, Analog, NTP and System measurement

reporting to:
2# Domoticz controllers via HTTP, P2P, Rules

[fly74]

My plugins in use:

DS18B20 / BME280
SSD1306 /w. and wo. framed
7Segment
Dummy Device
Switch

P2P Controller
Generic HTTP controller

Rules

[uzi18]

maybe add just some kind of stats and send it periodically to server, so we will know minimal needed features, then extra stuff will be available for own/self build or with configurator for patreons

I don't use espeasy personally but help for my friends to use it :)
some improvements was dedicated for them

stats:
nettemp controller
mqtt controller
dallas
gpio
bme/bmx
pcf
lcd

[Budman1758]

My 2 cents...
A much more detailed tutorial on setting up the necessary components to self-compile the firmware with just the plugins needed for your specific use case, If it were a bit easier to understand the process to build your own versions that would reduce the need for 20 to 30 plus bin files every release. I don't think you will ever make a set of pre-built bins that will please everyone. While this would not completely solve the issue methinks it would help immensely. This way you could have some very generic builds for most setups and build a few device specific bins for stuff like Sonoffs etc.

[ghtester]

I am using or would like to start using in future these devices/features:

Analog input - internal
Display - LCD2004
Display - OLED SSD1306
Energy (AC) - Eastron SDM120C/220T/230/630
Environment - DHT11/12/22 SONOFF 2301/7021
Environment - DS18b20
Environment - BMx280
GAS - SGP30
Generic - Dummy Device
Generic - MQTT Import
Generic - Pulse Counter
Input - iButton
Keypad - TTP229 Touch
Light/Lux - BH1750
Output - DomoticzMQTT Helper
Position - HC-SR04
RFID-PN532
Regulator - Level Control
Switch Input - Rotary Encoder
Switch Input - Switch

Infrared Receive
Infrared Transmit

Controllers:
Generic HTTP
ThingSpeak
Domoticz MQTT

Notifications:
Email (SMTP)

BTW. on my primary node I am still running this (a bit older but great) ESP Easy release:
GIT version: mega-20180311
Uptime: 99 days 0 hours 43 minutes
Load: 29% (LC=11244)
Free Mem: 14696 (3176 - parseTemplate3)
Wifi RSSI: -87 dB
:-)

[2bontb]

I use the following

Devices:
Sonoff Basic (running 1M OTA)
Sonoff S20 (running 1M OTA)
Sonoff POW (running hard SONOFF POW)

Wemos + Sensors, having temp/hum/presure/light often combined with PIR and sometimes with Pulsecount (running 4M normal)

• Environment - DS18b20
• Switch input - Switch
• Light/Lux - TSL2561
• Environment - DHT11/12/22 SONOFF2301/7021
• Environment - BMP085/180

Wemos + dedicated for Itho (own 4M normal build and added itho from plugin playground [adapted to fit new web interface])

• Itho ventilation remote

Wemos + display + 3 gpio's (running 4M normal)

• Display - LCD2004
• Switch input - Switch

All Sensors use : system info

• Generic - System Info

Controlers:
FHEM

Using: Syslog, NTP

Running on all systems Release mega-20190827

[HV-NL]

I am using the following combinations:

Dummy device

OLED SSD1306
Switch

OLED SSD1306
APDS9950 (stripped version, derived from APDS9960)
Switch

md5-70f3cd84d26193e1d1270980836dcef9



DHT22



md5-70f3cd84d26193e1d1270980836dcef9



Switch



md5-70f3cd84d26193e1d1270980836dcef9



P1 WiFi Gateway

Many of these combinations use Rules
All are communicating with Domoticz HTTP and many are writing results to syslog

My devices are

• Wemos D1 mini (4M),
• NodeMCU (4M)
• Sonoff basic (1M)
• Sonoff S20 (1M or 4M, resoldered),

[micropet]

I would submit a simple personal config file where I can select the controllers and the plugins.

For people who can not compile themselves, a "normal" version should be enough.

If you want more, you have to worry about how to compile a file. Of course, a simple tutorial is very useful.

[TD-er]

@micropet
This evening I've been fixing the good old memanalyzer.py file, also with this in mind.
So now I know again how to generate a build from within Python having all setup from the command line.

The other way around is already possible, as can be seen in the config of the custom build.
That one uses a Python script to define the plugins to use.
Both could be used to read a file, but then you have to change the file again if you want to change the config.
Another option is to read from a database, where a config is stored from someone clicking away :)

I think if we just allow to read from a file -if exists- with nothing else than a list of plugins and controllers, then anyone can make its own preferred set.
We also have the custom.h file, but that's a bit harder to work with if you ask me.

[micropet]

So you mean a website that lists all plugins and controllers. Then I just do a ckeckmark to the required componenets.
That would be great, of course.

I never understood how it works with the Python script.
custem.h is also mysterious

I've always fiddled with define_plugin_sets.h.

[TD-er]

Just as an initial test, could you checkout the latest from the mega branch and go to tools/vagrant.
There is a (very short) readme about what to install and then go to that folder with a terminal/shell.
In Windows it can be CMD or Windows PowerShell and in Linux it can be any terminal.

Then type vagant up
Please let me know if anything fails.
If it is finished building something in the build folder, then you can type vagrant halt in the same window you typed vagrant up.

It is just preliminary, so I will add some support for reading TXT files defining what needs to be built, but this is not bad for a morning work I think.

[LeeNX]

@TD-er oh, I like this idea of trying to build in a VM! Almost like a single command to get a standard build tool-chain.

Current implementation looks to be possible a little broken. There were some red screen stuff scroll by, so I don't know what was broken just yet, but I did do the process in a screen recording, so I can upload what happen from my OSX machine.

I do seem to have four bins files thou

vagrant@vagrant:~/GitHub/letscontrolit/ESPEasy/dist/bin$ ls -la
total 23832
drwxrwxr-x 2 vagrant vagrant     4096 Sep  2 09:37 .
drwxrwxr-x 3 vagrant vagrant     4096 Sep  2 09:37 ..
-rw-rw-r-- 1 vagrant vagrant 16777216 Sep  2 09:37 blank_16MB.bin
-rw-rw-r-- 1 vagrant vagrant  1048576 Sep  2 09:37 blank_1MB.bin
-rw-rw-r-- 1 vagrant vagrant  2097152 Sep  2 09:37 blank_2MB.bin
-rw-rw-r-- 1 vagrant vagrant  4194304 Sep  2 09:37 blank_4MB.bin
-rw-rw-r-- 1 vagrant vagrant   276848 Sep  2 09:37 ESPEasy_2step_UploaderMega_1024.bin
vagrant@vagrant:~/GitHub/letscontrolit/ESPEasy/dist/bin$ date
Mon Sep  2 09:40:48 UTC 2019
vagrant@vagrant:~/GitHub/letscontrolit/ESPEasy/dist/bin$

Might it be worth moving the discussion to an issue of it's own, instead of high-jacking your Feedback request?

My feedback on what I use ESPEasy for/with ... Every cheap sensor that I can find that ESPEasy supports now ... ;-) .

Can you not possible pull the web site stats on what sensors people search for or click throu?

Thanks for all the awesome tools!

[TD-er]

Hmm, I think it may be interesting to see what went wrong here.
I just tested it on Windows here, so that's why I did add the sed command in the provisioning to remove the CRLF and replace then by the Unix line endings.

I guess you're right in moving it to a separate issue about Vagrant.

[LeeNX]

@TD-er I don't know if anything went wrong, just saw some red messages, as I seem to have a build. Maybe it's just warnings.

Included me in the new issue, if you interest in any feedback I could give. I have text screen recording to upload some place, if that helps?

[TD-er]

Please continue in #2594 and I would like to see the screen recording.
The red messages could be build errors, but it is best if you check the ZIP file in the (new created) build dir: tools/vagrant/build
There should be one extra bin file like this:

[micropet]

I think that will not work for me.
I have Hyper-V installed on all computers.
That seems to be incompatible with VirtualBox.
I'll keep looking.

[TD-er]

Maybe Vagrant can also work together with other virtualization environments?

I chose Vagrant since it is the defacto product for running deployment tests during development.
Can you run "Ubuntu on Windows" with Hyper-V active? (See Windows store for "Ubuntu")
If so, then we can split the scripts so you can also run the same tools only start them with another command.

[LeeNX]

@micropet vagrant supports hyper-v - https://www.vagrantup.com/docs/hyperv/

Am I missing something?

[TD-er]

@LeeNX There is a warning about HyperV on the Vagrant site

[LeeNX]

I think the warning is about VirtualBox's vs Hyper-V Hypervisors, but from the link I provided, I would believe that vagrant natively supports M$ Hyper-V Hypervisor. Could be something else too. There are warnings about Hyper-V and VMWare too.

I have not tried Vagrant on Windows, so I am very much just an arm chair suggestor ... ;-)

[TD-er]

Not all builds of the bento/ubuntu-18.04 support HyperV, but this build does.

Maybe if you just change the relevant parts of the Vagrantfile into this:

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-18.04"
  config.vm.box_version = "201812.27.0"
end

So add the box_version line right after this one:

config.vm.box = "bento/ubuntu-18.04"

I have not explicitly set the provider, so I guess it could detect HyperV then.

[micropet]

I get the following error:

d:SoftESPEasy-megatoolsvagrant>vagrant up
Vagrant failed to initialize at a very early stage:

There was an error while executing VBoxManage, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["--version"]
Stderr: Der Befehl ""c:Program FilesOracleVirtualBoxVBoxManage.exe"" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Stupid question: Does Virtualbox additionally need to be installed, or is Hyper-V sufficient?

And what should the Vagrant do?
I thought we were talking about a website with boxes to tick the required plugins.

[LeeNX]

I am guessing, as I don't use Windows nor M$ Hyper-V, but I think you are not meant to have VirtualBox installed on a Windows Computer that has Hyper-V setup. At least, that is how I read - https://www.vagrantup.com/docs/hyperv/

[LeeNX]

@micropet - I think the vagrant discussion is better moved to #2594

Vagrant build system will let you build custom firmware and plugins, without having to do more than git clone and vagrant up in the correct folder. Later some basic config files will let you build custom firmware for your ESP device with only the plugin and controllers you need ... I think ... ;-)

[micropet]

Thank you.

[TD-er]

Just a quick reply, to let you know I will reply to this in the Vagrant issue :)

[berbergh]

To begin with, I need the test version for as long as some devices are not set to Normal.
I noticed that of the mega-20190903, the 252 test core of 4M is gone.

So far, my devices are:
Mostly a Wemos with:

• OLED Framed, DHT22, DS18b20
• OLED Framed. Gases - CO2 MH-Z19, Dust - PMSx003
• OLED Framed, DHT22, BME280
• Communication Serial Server RFLINK
• Display Nextion

I would like to get MQTT working, but it cuases severe crashes on all devices.

[TD-er]

To begin with, I need the test version for as long as some devices are not set to Normal.
I noticed that of the mega-20190903, the 252 test core of 4M is gone.

If a build is not included in the ZIP file, it could very well be the binary size exceeded the limit for a sketch size. (if you flash one that's too large, it will corrupt the existing partitioning or just continue to crash)
In the latest builds, the file name of the "_4M" builds has also changed to "_4M1M" to make it clear it is using 1 MB SPIFFS on 4 MB flash.
This to differentiate the "_4M2M" builds which have 2 MB SPIFFS.
Currently there is only one _4M2M build, but better make it consistent throughout all builds

About the MQTT crashes.
Do you have any special demands?
Have you tried lowering the timeout to 100 msec?
What MQTT controller do you use?

I have MQTT running on all my nodes without issues, but that's all Domoticz MQTT connecting to Mosquito on a Pi.

[berbergh]

About the MQTT crashes.
Do you have any special demands?
Have you tried lowering the timeout to 100 msec?

No special demands at all. I have set time out to 100 msec now.

What` MQTT controller do you use?
I have MQTT running on all my nodes without issues, but that's all Domoticz MQTT connecting to Mosquito on a Pi.

I have the same setup. Domoticz MQTT and Mosquito on a Pi.

In the latest builds, the file name of the "_4M" builds has also changed to "_4M1M" to make it clear it is using 1 MB SPIFFS on 4 MB flash.
This to differentiate the "_4M2M" builds which have 2 MB SPIFFS.
Currently there is only one _4M2M build, but better make it consistent throughout all builds

Which one should I use for Wemos 4MB?

[TD-er]

Well, right now only the IR related build has the 4M2M layout. (and still the old 4M1M)
All others have the 4M1M layout, so I guess there is yet not even a choice here. Just go for 4M1M.
The 4M2M is meant to allow for more room on SPIFFS, but it has not yet been tested.
If you change from one to the other you will loose all settings, because the filesystem is recreated and formatted.

[s0170071]

back to the initial request:

I use these pluigins:
SHT25 temp/ humidity (my code, in the playground)
SMA inverter, modbus (my code, in the playground)
MT681 SML power meter (my code, in the playground)
ViessmanVitotronic optical interface (my code, in the playground)
IRTX

all with mqtt except the viessmann, that one uses http.

[TD-er]

Last night's build was somewhat of a disaster if you look at the bin files that made it into the zip.
Only 1 "test" build and no "dev" build included.
So I will split up these, but the question is: "how?"

[shaman1010]

sonoff pow2 (AC meter), BME 180/280, MQTT, Dummy.

May be linux-way? Small base core (for minimum size of flash 512kB), and overlays ( https://github.com/pvvx/esp8266web )
Overlays - added from web interface esp8266. One device (DS, MQTT, BME, etc) = one overlay.

Or server with own builder. And checkboxes. ( https://wifi-iot.com/p/main/ )

[TD-er]

Or server with own builder.

See: https://github.com/letscontrolit/ESPEasy/issues/2594

[TD-er]

I was thinking about how to split the build sets.
On the forum, it was suggested to also have a look at how Tasmota is handling build sets

What about the controllers?
Is anyone using a MQTT controller with something other than generic/advanced HTTP?
I guess Domoticz HTTP/MQTT should be in the same build set, since they share a lot of code.

My suggestion:

• All MQTT controllers + Domoticz HTTP + generic/advanced HTTP + MQTT import plugin
• All other controllers + generic/advanced HTTP (not including MQTT import plugin)

And use that including the existing normal/test/dev builds?
Maybe also have one set of "metering" plugins, like interfacing with Modbus devices.
And one for "playful " plugins like

• P038_NeoPixel
• P041_NeoClock.
• P042_Candle
• P043_ClkOutput
• P048_Motorshield_v2
• P055 Chiming
• P065_DRF0299_MP3
• P070 neoPixel Clock
• P079_Wemos_Motorshield

Those are not useful for most users. Doesn't mean a build with these should only have these, so we may also need some "standard set" of plugins included in all builds (except for "switch only" builds)

Just for some idea of plugin/controller sizes the output of the memory analyzer run.
The rather big ones are marked bold.

module |cache IRAM |init RAM |r.o. RAM |uninit RAM |Flash ROM
--- | --- | --- | --- | --- | ---
CORE |29268 |1584 |2960 |37096 |568404
MQTT_ONLY |0 |68 |356 |1192 |35936
USE_SETTINGS_ARCHIVE |0 |0 |140 |0 |9664
WEBSERVER_RULES_DEBUG=1 |0 |-8 |0 |0 |16
WEBSERVER_TIMINGSTATS |0 |-8 |4 |-8 |5216
src/_C001.ino |0 |-8 |8 |8 |5344
src/_C002.ino |0 |68 |40 |1144 |18592
src/_C003.ino |0 |0 |4 |24 |2400
src/_C004.ino |0 |-8 |0 |16 |3008
src/_C005.ino |0 |-4 |8 |1144 |10320
src/_C006.ino |0 |-4 |8 |1144 |10400
src/_C007.ino |0 |-8 |0 |32 |3056
src/_C008.ino |0 |-8 |4 |8 |5184
src/_C009.ino |0 |64 |36 |8 |7360
src/_C010.ino |0 |-8 |0 |32 |4112
src/_C011.ino |0 |0 |4 |24 |7792
src/_C012.ino |0 |0 |0 |32 |4512
src/_C013.ino |0 |-8 |0 |16 |1328
src/_C014.ino |0 |-4 |332 |1152 |19472
src/_C015.ino |0 |0 |1144 |120 |7712
src/_C016.ino |0 |-8 |0 |32 |5216
src/_C017.ino |0 |72 |40 |24 |4768
src/_C018.ino |404 |-8 |104 |72 |18928
src/_N001_Email.ino |0 |-8 |4 |-8 |4000
src/_N002_Buzzer.ino |0 |-8 |0 |0 |688
src/_P001_Switch.ino |0 |-8 |40 |8 |13696
src/_P002_ADC.ino |0 |0 |0 |16 |2496
src/_P003_Pulse.ino |220 |-8 |0 |384 |2512
src/_P004_Dallas.ino |0 |-8 |28 |112 |4352
src/_P005_DHT.ino |24 |0 |16 |0 |2064
src/_P006_BMP085.ino |0 |0 |0 |16 |2112
src/_P007_PCF8591.ino |0 |0 |0 |0 |480
src/_P008_RFID.ino |240 |0 |0 |16 |1472
src/_P009_MCP.ino |0 |-8 |16 |0 |10768
src/_P010_BH1750.ino |0 |-8 |0 |0 |2544
src/_P011_PME.ino |0 |-8 |0 |0 |4032
src/_P012_LCD.ino |0 |8 |64 |0 |5568
src/_P013_HCSR04.ino |0 |-8 |0 |48 |5856
src/_P014_SI7021.ino |0 |-8 |0 |0 |1744
src/_P015_TSL2561.ino |0 |-8 |0 |16 |4112
src/_P016_IR.ino |464 |0 |680 |120 |33648
src/_P017_PN532.ino |0 |-8 |0 |64 |2192
src/_P018_Dust.ino |0 |-8 |0 |0 |640
src/_P019_PCF8574.ino |0 |0 |16 |0 |10704
src/_P020_Ser2Net.ino |0 |-8 |8 |24 |3184
src/_P021_Level.ino |0 |-8 |0 |16 |2176
src/_P022_PCA9685.ino |0 |-8 |4 |-8 |8368
src/_P023_OLED.ino |0 |-8 |40 |-8 |5808
src/_P024_MLX90614.ino |0 |-8 |0 |0 |960
src/_P025_ADS1115.ino |0 |0 |16 |0 |3120
src/_P026_Sysinfo.ino |0 |-8 |0 |0 |3440
src/_P027_INA219.ino |0 |8 |4 |24 |2816
src/_P028_BME280.ino |128 |8 |28 |32 |7008
src/_P029_Output.ino |0 |-8 |0 |0 |512
src/_P030_BMP280.ino |128 |0 |0 |64 |3072
src/_P031_SHT1X.ino |0 |-8 |0 |0 |2704
src/_P032_MS5611.ino |0 |-8 |0 |48 |2640
src/_P033_Dummy.ino |0 |0 |0 |0 |2400
src/_P034_DHT12.ino |0 |-8 |0 |0 |736
src/_P035_IRTX.ino |76 |-8 |868 |-8 |21888
src/_P036_FrameOLED.ino |0 |0 |0 |144 |32672
src/_P037_MQTTImport.ino |0 |-4 |12 |1200 |12736
src/_P038_NeoPixel.ino |128 |-8 |20 |-8 |5968
src/_P039_Thermocouple.ino |0 |0 |12 |16 |1968
src/_P040_ID12.ino |0 |-8 |0 |0 |768
src/_P041_NeoClock.ino |128 |-8 |0 |0 |3360
src/_P042_Candle.ino |128 |0 |12 |32 |8672
src/_P043_ClkOutput.ino |0 |0 |0 |0 |1680
src/_P044_P1WifiGateway.ino |0 |-8 |28 |64 |4416
src/_P045_MPU6050.ino |0 |-8 |0 |64 |4464
src/_P046_VentusW266.ino |0 |0 |0 |0 |0
src/_P047_i2c-soil-moisture-sensor.ino|0 |-8 |0 |0 |3104
src/_P048_Motorshield_v2.ino |0 |24 |8 |-8 |6944
src/_P049_MHZ19.ino |404 |0 |44 |16 |9280
src/_P050_TCS34725.ino |0 |-8 |0 |0 |3344
src/_P051_AM2320.ino |0 |-8 |0 |0 |992
src/_P052_SenseAir.ino |404 |-8 |224 |16 |13312
src/_P053_PMSx003.ino |404 |-8 |12 |16 |7024
src/_P054_DMX512.ino |0 |0 |0 |0 |1696
src/_P055_Chiming.ino |0 |-8 |4 |-8 |3616
src/_P056_SDS011-Dust.ino |404 |-8 |12 |16 |7952
src/_P057_HT16K33_LED.ino |0 |0 |48 |0 |4192
src/_P058_HT16K33_KeyPad.ino |0 |0 |32 |0 |1376
src/_P059_Encoder.ino |432 |64 |16 |48 |3888
src/_P060_MCP3221.ino |0 |-8 |32 |0 |2240
src/_P061_KeyPad.ino |0 |-8 |64 |0 |2352
src/_P062_MPR121_KeyPad.ino |0 |-8 |16 |0 |1840
src/_P063_TTP229_KeyPad.ino |0 |-8 |4 |-8 |1808
src/_P064_APDS9960.ino |0 |0 |0 |0 |4944
src/_P065_DRF0299_MP3.ino |404 |-8 |28 |16 |4144
src/_P066_VEML6040.ino |0 |0 |16 |0 |2672
src/_P067_HX711_Load_Cell.ino |0 |0 |4 |88 |8224
src/_P068_SHT3x.ino |0 |0 |0 |0 |1776
src/_P069_LM75A.ino |0 |-8 |32 |0 |1104
src/_P070_NeoPixel_Clock.ino |0 |-8 |0 |0 |16
src/_P071_Kamstrup401.ino |404 |0 |16 |16 |7040
src/_P072_HDC1080.ino |0 |-8 |0 |0 |800
src/_P073_7DGT.ino |0 |-8 |120 |-8 |8704
src/_P074_TSL2591.ino |0 |-8 |28 |0 |4096
src/_P075_Nextion.ino |404 |-8 |76 |16 |10560
src/_P076_HLW8012.ino |432 |0 |8 |24 |8304
src/_P077_CSE7766.ino |0 |-8 |0 |0 |3280
src/_P078_Eastron.ino |404 |-8 |52 |8 |9840
src/_P079_Wemos_Motorshield.ino|0 |0 |0 |0 |2096
src/_P080_DallasIButton.ino |0 |-8 |0 |16 |2560
src/_P081_Cron.ino |0 |56 |608 |16 |8624
src/_P082_GPS.ino |432 |0 |136 |24 |17536
src/_P083_SGP30.ino |0 |-8 |0 |0 |1248
src/_P084_VEML6070.ino |0 |-8 |0 |0 |1472
src/_P085_AcuDC243.ino |532 |0 |240 |16 |12336
src/_P086_Homie.ino |0 |-8 |4 |-8 |3904
src/_P087_SerialProxy.ino |404 |-8 |12 |16 |8640
src/_P088_HeatpumpIR.ino |0 |8 |624 |320 |20752

[TD-er]

Just the log of last night's build.

Status | Reason | Env
--- | --- | ---
SUCCESS| 829456 < 1044464 Bytes. |custom_ESP8266_4M1M
SUCCESS| 1689376 < 1900544 Bytes. |esp-wrover-kit_test_1M8_partition
SUCCESS| 1689392 < 1900544 Bytes. |esp32test_1M8_partition
SUCCESS| 863296 < 892912 Bytes. |normal_ESP8266_1M
SUCCESS| 863664 < 892912 Bytes. |normal_ESP8266_1M_VCC
SUCCESS| 863248 < 892912 Bytes. |normal_ESP8285_1M
SUCCESS| 818192 < 892912 Bytes. |normal_core_241_ESP8266_1M
SUCCESS| 865088 < 1044464 Bytes. |normal_WROOM02_2M
SUCCESS| 865088 < 1044464 Bytes. |normal_core_252_WROOM02_2M256
SUCCESS| 864992 < 1044464 Bytes. |normal_ESP8266_4M1M
SUCCESS| 819344 < 1044464 Bytes. |normal_core_241_ESP8266_4M1M
FAILED|No file created. |normal_core_252_ESP8266_16M
SUCCESS| 871712 < 1044464 Bytes. |normal_core_260_sdk222_alpha_ESP8266_4M1M
FAILED|No file created. |normal_core_260_sdk222_alpha_ESP8266_16M
SUCCESS| 614016 < 616448 Bytes. |minimal_core_242_ESP8266_1M_OTA
SUCCESS| 614016 < 616448 Bytes. |minimal_core_242_ESP8285_1M_OTA
FAILED|too large 650064 > 616448 Bytes. |minimal_core_252_ESP8266_1M_OTA
FAILED|too large 650048 > 616448 Bytes. |minimal_core_252_ESP8285_1M_OTA
SUCCESS| 839584 < 892912 Bytes. |minimal_IRext_ESP8266_1M
FAILED|No file created. |minimal_IRext_ESP8266_4M1M
SUCCESS| 840944 < 1044464 Bytes. |minimal_IRext_ESP8266_4M2M
SUCCESS| 956512 < 1044464 Bytes. |normal_IRext_no_rx_ESP8266_4M2M
SUCCESS| 1012000 < 1044464 Bytes. |test_core_260_sdk222_alpha_ESP8266_4M1M
SUCCESS| 1012704 < 1044464 Bytes. |test_core_260_sdk3_alpha_ESP8266_4M1M
FAILED|No file created. |test_core_260_sdk222_alpha_ESP8266_16M
SUCCESS| 1013904 < 1044464 Bytes. |test_ESP8266_4M_VCC
SUCCESS| 1004544 < 1044464 Bytes. |dev_ESP8266_4M1M
SUCCESS| 710672 < 1044464 Bytes. |hard_SONOFF_POW_4M1M
FAILED|No file created. |hard_other_POW_ESP8285_1M
FAILED|No file created. |hard_Shelly_1_2M256
FAILED|No file created. |hard_Ventus_W266

The _"No file created."_ is reported when at build the file size already exceeds the set value in platformio.ini. (or another build error)

[jimmys01]

minimal_IRext_ESP8266_4M1M
minimal_IRext_ESP8266_4M2M
Those are practicaly the same build but the one compiled and the other not..
Also in my pc the latest code builds fine.

[TD-er]

I've noticed the same.
It could be the full path in which the build is made may just be the tipping point.
We can switch those to core 2.6.0, since that one has the brand new feature of flash string de-duplication.
That may save a few kByte in binary size. Perhaps a lot more on the IR lib.

Edit:
The "minimal" 4M1M build may be checking against 1M minimal size.

[TD-er]

I pinned the issue, since it is really important to have a proper definition of builds.
Almost all testand devbuilds no longer fit the sketch size.

[clumsy-stefan]

As I anyway use my own set of plugins, I can't really comment here, but one thing I noticed is that even when only using the one controller plugin I need (C009 FHEM) and one Plugin (P001 Switch) compiling it with Core 2.6.0 I already get over the maximum size for 1M untis for using the two-step OTA.
So therefore my vote would be to add some more stringent BUILD_MINIMAL_OTA or additional definitions which strip out everything not needed (eg. icons, infos, debugs, etc.) and so it fits again in small (1M) builds.

[Grovkillen]

The new GUI engine (which is not yet available for beta) will allow for external hosting* of the GUI so that would be ideal for the 1M units.

*Meaning you can surf to a big brother (4M) unit and from that one control the 1M unit(s).

[TD-er]

I have been ding some static code analysis and the C009 controller does appear quite high on the resources used list.
The do_process_c009_delay_queue call alone uses 7.6 kByte.
It also uses JSON, which needs an external library.

Plugin_001 does use 11.2 kB

[TD-er]

Should the minimal build also have the web-log present?

[Grovkillen]

Should the minimal build also have the web-log present?

No I'd say that's not in the scope of those small units.

[clumsy-stefan]

Should the minimal build also have the web-log present?

No I'd say that's not in the scope of those small units.

Agree, especially if it saves significantly. It's still possible to use syslog for logging/debugging though

[mackowiakp]

Since I am probably the oldest here (almost 60 years) and I started my computer education in 1980, I see that the whole ESP project has reached the border with which I have encountered many times in IT history.
Please, do not treat this as rude, but I think the question is not right.
The problem is not which device/sensor to choose for a particular distribution. The problem is how to get FW containing all the drivers needed in a particular application, ignoring all the others, even the most popular ones.
Here comes a comparison with the history of the Novell NetWare system.
Version 1.XX (based on the XNS protocol - probably no one remembers that it is the first Ethernet protocol) only supported specific types of hardware. Of course, with time new types of new equipment arrived. Therefore, version 2.XX introduced a system build/compilation at the installation stage. During this compilation, it was possible to add (from diskette) device drivers supplied by various manufacturers.
This process required some knowledge and was not universal. Any hardware change required a new build/compilation. And adding too many drivers resulted in exceeding the computer resources, for example, memory.
I think the ESP easy project is at this stage. To have your own set of sensors, we basically need to compile our own version of FW.
At the end of the last century, Novell introduced version 3.xx. It was probably the first system with dynamically loaded kernel and driver modules. Something that is common today was a revolution at the time.
So I think the future is a similar solution in the case of ESP. At the flash stage, we select from the future version of esptool, using checkboxes, a list of necessary sensors. And the ESPeasy "kernel" contains only what is directly on the PCB, for example, nodeMCU. If the number of drivers exceeds the device resources, esptool will inform us.
And besides, the story of Arduino does not end today. Device resources will increase, not decrease. The number and types of sensors will also increase. Therefore, the use of a system of dynamically loaded modules seems even more necessary.
These are my - maybe a little too long - comments on the choice of sensors.

[Grovkillen]

@mackowiakp I think you're 100% correct in your analysis of the problem we're facing. That is why we plan on making a "compile-as-you-go" (online) service. But until we have time for that we want to create a set of commonly used sets of firmware releases. And we will most likely still provide pre-compiled versions for development and Swiss-army-knife units.

[TD-er]

I totally agree with your post, but sadly there is no possibility for our setup (using C++ in the way we use it) to load components at run time.
For that we should move to another language like MicroPython. (see upyEasy project)

As you may have seen, I was working on a Vagrant setup to ease the process of self-compiling and I do think there's the future.
Let the node decide what it needs.
But we're not there yet. So until then we should make a solid set of application driven plugins.
Maybe it is already sufficient to split at controller level, like MQTT and non-MQTT, since those are the biggest controllers.
Also LoRaWAN and network related controllers will hardly ever be used together.

[mackowiakp]

@ Grovkillen .But remember. Swiss-army-knife has 23854 units, or more. Except the one You just need. Murphy is alive!

[Misiu]

As you may have seen, I was working on a Vagrant setup to ease the process of self-compiling and I do think there's the future.

I this that this will be THE biggest feature yet🎉 User picks the board (device) he has (ideally a list of devices like Wemos D1, Shelly 1, etc), he selects the features he needs and he clicks the "COMPILE" button.
This is the feature that should be available only for active supporters (patreon users only?). The goals would be completed in no time! :)

I think that there should be a list of devices that the user can select from because it isn't very intuitive what bin we should select now. Take a look at espurna releases page I think it's easier to select bin file named shelly2.bin than to know what chip the device has inside.
The configurator should allow selecting the device and based on that it would select the correct chip. What do You think?

[TD-er]

Well I tried the "build based on product" idea a while ago, see the Sonoff POW builds.
But then we would end up with so many builds.

Also it does not fix the problem with the "many sensors build" definition.

I'm now working on splitting the stuff as much as possible, to make it all definable from the Custom.h file as much as possible.
I also did make a feature request to the ESP8266/Arduino repo for an option to even let go of flash-size related build definitions. (determine SPIFF location at run time, not compile time)

Also I do think the new core 2.6.0, which will be released in about a week, will be enough for 90% of the builds, so that can reduce the build images a lot.

[Misiu]

@TD-er I agree that "build based on a product" won't be good for GitHub releases page, but I think it would be helpful to preconfigure the web-based build tool.
Let's say you select Shelly 2, then the ESP8285 will get selected (I think they use ESP8285), this way you will know that you can select plugins that won't increase the build size above 1M.

EDIT:
@TD-er there is one more benefit of selecting the device. According to this issue there will be some limitation when auto-size more will be enabled:

Warnings, when the option is enabled:

user must ensure sketch will fit in flash (cannot check)
not fully tested, especially with 512K and 1MB flash chips, may not work
(there is this "magic" word at physical address 0x0 in flash which contains "software" flash size...)
(reading it is disabled in this PR, replaced by the "physical" result ESP.getFlashChipRealSize())

So you must know how many plugins you can select to fit them all in flash.

[henkiejan1]

I am using the most:
BME280 sensor
Domoticz controller
MQTT controller
ESP Easy P2P controller
PWM functionality (no settings via webinterface. Should be great ;))
SSD1306
CCS811/HDC1080
BME680

And also more and more ESP32. So it should be fine that the ESP32 get more priority in my opinion. :)

[fmuntean]

question: what are the two methods: addPredefinedRules and addButtonRelayRule used for ?
they seems to be unused and take some firmware size due to the rule file defined there.

[TD-er]

They are used when you "factory default" to some Sonoff devices.
They generate rules and tasks for the push button + relay.
It is called from void ResetFactory()

I guess they can be put behind some #define

[ShardanX]

Plugins in use here, I'll split list by device FYI:
Weather Station unit:

• PCF8574
• Pulse Counter
• DS18B20
• Dummy Device
• TLS2561
• BMx280
• System Info
  Room Sensors
• DS18B20
• TLS2651 or BH1750
• SI7021/HTU21D
• System Info
  Weather Sensor (Old version)
• BMP180
• BH1750 / TLS2561
• SHT1x
  Dust Sensors
• PMSx003
• BMx280
• NeoPixel Basic
• Oled 1306
  Remote box
• PCF8574 / MCP23017
• NeoPicel Basic

[TD-er]

Just a suggestion, so please let me know what you all think of it.

Some sensors/devices have quite a specific purpose and are not likely to also have a number of other sensors connected.
For example displays and power meters.

Although it is unlikely you will connect an e-ink display and a Nextion for example, so at first it may seem not making sense to combine them in the same build, they do have in common that they are not likely to be combined with a lot of other sensors.
So I was wondering, would it make sense to make a build containing the "normal" set of plugins + all displays and one of the "normal" set + all energy meters?
This way you still have builds present to get people started with those and we no longer have "zombie" plugins that are not part of a nightly build.
Later when we have a build server operational, it doesn't matter anymore what build combinations we have.

So let me hear what you think about it.

[tonhuisman]

That should be good enough as a workaround for now.

[berbergh]

Ik think this would work for me, but what do you mean with 'normal' set?

[TD-er]

Ik think this would work for me, but what do you mean with 'normal' set?

Every plugin marked here as "normal"

[berbergh]

I see.
I guess it is virtually impossible to make the perfect prefab build. I can't.
So, I my guess is that your proposal is a good one.

[berbergh]

Just updated a Display and and an energy version. So for all's well.

Can I use ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8266_1M_OTA_Domoticz.bin tp update my 4 channel Sonoff?
I do not understand what this sentence means: Minimal OTA builds (and other builds with reduced build size) link to online CSS and JavaScript files

[TD-er]

Until this change for linking to external CSS/JS files, all these static files were included in the build.
Meaning the build size was larger due to files which your browser could also fetch from another site.

I linked to a CDN URL of GitHub, which hosts copies of our repository.
Meaning your browser will fetch them from the GitHub servers when needed.
However the "TTL" of those values (the time they can be safely cached by your browser) is set to 1 year.

So what does this al mean?
If you operate your units linking to an external hosted CSS or JS file, your browser must be able to reach the GitHub servers.
But this only has to be done once and after that the browser will cache those files for 1 year (or until you clear your cache).
This means you may need to have an internet connection on the device you use to access the ESPEasy node if those files are not in your browser cache.

There is a way to overcome this limitation, by saving the files using the same names onto your SPIFFS file system of the node.
If the files are present on the local file system, the ESP will serve them to you.

The files are located here: https://github.com/letscontrolit/ESPEasy/tree/mega/static

• espeasy_default.css
• favicon.ico
• update_sensor_values_device_page.js
• fetch_and_parse_log.js
• rules_save.js
• github_clipboard.js
• reboot.js
• toasting.js

[clumsy-stefan]

@TD-er this is a great improvement!! My build are ~4K smaller, the 1M build even twice as much!

One suggestion: the URI could be in a static #define or probabky even as config option so it would be possible to easily change it to a local server (eg. home-automation server) which would work even without having access to the internet and not serving it from the node itself (and therefore have a central place to make changes)...

Thanks a lot for your great work and this update!

[TD-er]

I will indeed add it to a define so you can set it yourself in Custom.h for example.
I only have to find out how to update the URL to fetch it based on the last tag to make sure a node always uses the files intended for the specific build.

[berbergh]

@TD-er Thank you for this explanation, but I still need to bee sure.
Can I use ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8285_1M_OTA_Domoticz.bin to update my 4 channel Sonoff?
If it fails, I have to unmount the Sonoff from a difficult place to reset.

[tonhuisman]

I have done an OTA update test using an ESP-01S unit, in the usual 2-step process, but not with that specific build tough. My test was done using a beta release. It should work, as the size is nearly the same as the minimal_core_274_sdk3_ESP8266_1M_OTA_Domoticz file I used for my tests. The usual differences between ESP8266 and ESP8285 should not get in your way here, just select the correct bin file.
I'm assuming that the 2step_uploader is capable of running correctly on an ESP8285, but I have no ESP8285's so I can't confirm that. If you have done this with older builds, it should still be possible though.

[TD-er]

The 2-step OTA bin has not been changed for years, and never heard of anyone having trouble with it on ESP8285.
There is not that much of a difference between ESP8285 and ESP8266.
The main difference is the availability of GPIO pins (ESP8285 has more unused) and the internal flash.
If a build is made to use the flash with at most 2 pins (DIO mode for example) then I am almost certain any ESP8266 build will run on ESP8285 (given it does not use > 1 MB flash).

Only other thing that may be an issue is when you update from a very old version.
The settings should be compatible, but I cannot guarantee it from very old builds. So better also make a backup of your settings before updating.

[berbergh]

Just what I was affraid of, happened.

My 4 chanel Sonoff gave this in the web interface:
Update error: ERROR[4]: Not Enough Space

After I first uploaded:
ESPEasy_2step_UploaderMega_1024.bin
and then:
ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8285_1M_OTA_Domoticz.bin

But after a reboot, it seems to have updated correctly.

No, there are some issues. I try to find out what they are, but some of the settings are gone.

[berbergh]

Sonoff is no longer acting on messages like http://DEVICE-IP/control?cmd=GPIO,0,1 I send to it from Domoticz

[TD-er]

Hmm I just tested it on my breadboard and it does reply with a JSON reply showing the new state.
Can you try that on your node?

[berbergh]

per haps this is not the right place to continue with this specific problem.
But I had the same results with JSON, so I dug in a bit deeper. Now I directly control the relay ports of the Sonoff, It seems to be okay again. Beforehand I used to send commands to the switch ports, that in turn toggled the relays accordingly via a rule, but that does not work as it used to anymore.
Most important is that I can reach the Sonoff and manipulate the relays.
That gives me time to find out why things do what they do.

[TD-er]

Maybe you should also have a look at Domoticz MQTT helper

[berbergh]

Thank you, I just did and followed the directions. However, if I do not put http://SONOFF-IP/control?cmd=GPIO,12,1 in the action on line of the switch in Domoticz, nothing is switched to on.

[tonhuisman]

nothing is switched to on.

Hm, that could be true if you don't have a MQTT server in your network, that Domoticz is subscribed to. Did you install a new MQTT server to enable that? In that case a reboot may be required to get it all up and running.

[berbergh]

It was even worse than that. I forgot to enable the MQTT controller

[berbergh]

In one of the ESPeasy modules, I use dust sensor PMSx003, which is still being tested.
It is in neither Display nor Energy. What can I use instead to update?

[TD-er]

I can move it to "normal", as it is for sure stable for years.
Let's hope it will not break other builds then for becoming too large.

[berbergh]

I am using the sensor for some 2 years now.
I see it is also still here ESP_Easy_mega_20201227_test_ESP8266_4M1M_VCC.
That could work for me, I guess.