Espeasy: ESP32 build 20200426 Update error: Could Not Activate The Firmware

[TD-er]

See also this topic on the forum

Edit:
Changed the topic title, as the error is different when trying to use the non-factory image (as I should have used)
Still it doesn't work.

[TD-er]

This error is sent from these lines of code: https://github.com/espressif/arduino-esp32/blob/c8215315aef8a11529743dd403f2e894350ad547/libraries/Update/src/Updater.cpp#L236-L238

Here the description of the return types of that check function: https://github.com/espressif/arduino-esp32/blob/1977370e6fc069e93ffd8818798fbfda27ae7d99/tools/sdk/include/app_update/esp_ota_ops.h#L136-L150

[uzi18]

@TD-er exact error:

E (323432) esp_image: Checksum failed. Calculated 0xb4 read 0xdc
Could Not Activate The Firmware

error is generated in firmware patched with crc2.py - we can disable it for esp32 temporary
it looks like esp32 image have internal crc and it need to be also updated after patching

[TD-er]

Then there is no need for generating CRC of the binary.
The filename we also patch can also be generated at build time.

[fmuntean]

@TD-er : I don't understand your last comment.
I need to be able to replace the firmware file name inside so it gets displayed correctly in sysinfo page.
If I upload it using the VSCode platform IO it shows firmware.bin where all my files follow a certain naming convention to be able to keep track of them.
Please explain what we can do to be able to upload the firmware using the /update page and how do we CRC the binary for ESP32

[uzi18]

fix for this was merged yesterday, please try to pull and rebuild and now you can try to upload with /update subpage

[uzi18]

@fmuntean please try: ESP_Easy_mega_20200505_test_ESP32.zip

[fmuntean]

I was using the code which I pulled today. and get this signature error when trying to upload firmware using the /update page.
Maybe a process of how can I change the binary filename inside the bin file would be great. (I am talking changing the SET_BUILD_BINARY_FILENAME after the firmware is built. like for the esp8266 firmware.)

[uzi18]

@TD-er we should leave firmware.bin in build dir because --target upload will not work properly

[fmuntean]

I don't want to use the VS Code each time I want to update the ESP32 firmware.
I keep a folder with all my bins builds and the system records the firmware filename automatically in a DB upon boot.
I want to be able to do the same for ESP32 firmware.
My preferred way of uploading the firmware is using the /update page.
I batch build all my custom firmwares at once so can't all have the same filename when archiving them in the dist folder.

[TD-er]

Wel can parse an environment variable like SET_BUILD_BINARY_FILENAME and use that as file name in the build process. (also to include it in the system info page)

I will check if I also see the issue with OTA upgrade of ESP32 units.

@uzi18
What do you mean with this:

@TD-er we should leave firmware.bin in build dir because --target upload will not work properly

[uzi18]

@TD-er ok it works as expected with SET_BUILD_BINARY_FILENAME setting

[TD-er]

I just tested the OTA update (via web) on a fresh build for ESP32 and it updated without any issues.

Build:⋄ | 20107 - Mega32
-- | --
System Libraries:⋄ | ESP32 SDK v3.2.3-14-gd3e562907
Git Build:⋄ |  
Plugin Count:⋄ | 15 [Normal]
Build Origin: | Self built
Build Time:⋄ | May 6 2020 15:56:55
Binary Filename:⋄ | ESP_Easy_mega_20200506_custom_ESP32_4M316k
Build Platform:⋄ | Windows-10-10.0.18362-SP0
Git HEAD:⋄ | bugfix/RN2483_send_without_wifi

[fmuntean]

@uzi18 how do you want me to upload this test firmware?
The tool that is used for esp8266 is not working and neither esptool.py seems to work.
Currently the only way I can upload to esp32 is using the VSCode and a full rebuild.
do you have any instructions on how to upload the bin files?

[fmuntean]

I just tested the OTA update (via web) on a fresh build for ESP32 and it updated without any issues.

Build:⋄ 20107 - Mega32
System Libraries:⋄ ESP32 SDK v3.2.3-14-gd3e562907
Git Build:⋄  
Plugin Count:⋄ 15 [Normal]
Build Origin: Self built
Build Time:⋄ May 6 2020 15:56:55
Binary Filename:⋄ ESP_Easy_mega_20200506_custom_ESP32_4M316k
Build Platform:⋄ Windows-10-10.0.18362-SP0
Git HEAD:⋄ bugfix/RN2483_send_without_wifi

@TD-er : how did you changed the default "Binary Filename" after the build?

[fmuntean]

This is my ESP32 PIO build environment:
[env:MFD_ESP32_4M316k]
extends = esp32_common
platform = ${esp32_common.platform}
build_flags = ${esp32_common.build_flags}
-D PLUGIN_BUILD_CUSTOM
-D MFD_PLUGIN_SET_PROD
-D USE_CUSTOM_H
-D DONT_USE_SERVO
-D ESP32
board = esp32dev
board_build.flash_mode = qio
extra_scripts = pre:pre_custom_esp32.py
post:post_build_ESP32.py

[TD-er]

@TD-er : how did you changed the default "Binary Filename" after the build?

I didn't.
I changed the whole build process in the last few days and now we can just define the name at build.

Not tested it, but you should be able to set it via something like this in your Custom.h file:

#define SET_BUILD_BINARY_FILENAME "this is my fancy build name"

Setting it in the platformio.ini config is a bit hard, as I had to jump through a lot of hoops to get Python to generate enough levels of double quotes to keep them in the defines in the C++ code.
But defining it in your Custom.h file should work just fine.

[fmuntean]

I build multiple environments at the same time. My format includes the environment name plus the date. How can we inject this using the new process?
Any chance we can update the pre_esp32.py and some post_esp32.py to do this and also to copy the resulting bin file into another destination ? (I personally use the 'dist' folder.)
I can try your approach for now as I only have one testing firmware for now.

[TD-er]

If those always have the same name, you can wrap those in a single .h file that you include from your own Custom.h file (or edit your Custom.h, whatever is easier)

Then you create some structure like this in that .h file:

#if defined(BUILD_FMUNTEAN_1)
#define SET_BUILD_BINARY_FILENAME "FMuntean build 1"
#elif defined(BUILD_FMUNTEAN_2)
#define SET_BUILD_BINARY_FILENAME "FMuntean build 2"
#elif defined(BUILD_FMUNTEAN_3)
#define SET_BUILD_BINARY_FILENAME "FMuntean build 3"
#else
#define SET_BUILD_BINARY_FILENAME "FMuntean does get the idea"
#endif

[uzi18]

@fmuntean maybe if you show your scripts we can guide you how to improve it
have a look at before_deploy and tools/buildESPEasy.sh shell scripts

[fmuntean]

I am using windows so .sh will not work for me.
I have my own environment files and using custom.h file.
Starting today I get the following error trying to build for ESP32
ModuleNotFoundError: No module named 'pygit2':
from pygit2 import Repository
However if I start python inside the VScode then run the same command is working fine and I double checked the pygit2 module exists. I can even run the command used in the python script to get the branch name with no issue.
@uzi18: Please let me know what scripts do you want to see. (is there a way to send them to you privately?)

[fmuntean]

SET_BUILD_BINARY_FILENAME 

The way that you propose is not sustainable. It is better to handle this in the pre: python script somehow.
Unfortunately I am still learning this part and I see that the whole building process changed lately.
Do you have an easy way to debug these pre and post python scripts in VS code ?

[uzi18]

you need to read more about Scons and python scripts hooking,
because python is interpreter you can simulate and debug some parts or whole scripts in python shell.

can't you just use gist or pasteit services? secure versions are also available

maybe VScode are able to execute bash scripts?

[uzi18]

about debugging try this one: https://github.com/pdbpp/pdbpp

[fmuntean]

@uzi18: Thanks. I just figure out that platformio under vscode was using another python installation and not the global one. Once I installed pygit2 module under that special folder (c:\users\fmunt.platformio\penv\scripts\python.exe) I am now able to build again.
It also seems that I can run bash files under vscode in Windows using git bash. I am learning right now the new build scripts.
The problem with debugging those python scripts is the missing platformio env objects.
I installed pdbpp and now I have to learn how to use it.
For now I can manually set the #define SET_BUILD_BINARY_FILENAME under my environment build flags and can see later on how this can be done in python in a pre:hook

[TD-er]

See: https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/generate-compiletime-defines.py
Specifically this part:

https://github.com/letscontrolit/ESPEasy/blob/dd4ac70c1128fdad702cb63bd337fca0b2599a56/tools/pio/generate-compiletime-defines.py#L11-L14

I can make it so that it can check for a Custom.py file and use that one to generate the file name.
For now, you can adjust the last line of that function to suit your build needs.

[fmuntean]

@TD-er : Thanks. I found that and was able to use that file. Just tested the resulted firmware and updated successfully using the browser.
I like your ideea to use another python file to override functions as it allows us to modify behaviour without messing the original file.

[TD-er]

Just to let you know.
The whole change of the build flow/tools was with the intent to make automatic building a lot more easy in the (near) future.
With bash scripting you can do a lot, but at some level it becomes quite painful to make it be capable of doing all I would like to do and remain flexible.
PIO + Python makes it a whole lot more convenient.

[fmuntean]

If i return the following MFD_ESP32_4M316k_2020_05_06.bin the system adds an extra .bin so the real binary filename:
esp32_create_factory_bin([".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin.bin"], [".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin"])

[TD-er]

I added a sticky issue for all running into the Python 3.8 upgrade: https://github.com/letscontrolit/ESPEasy/issues/3058

[fmuntean]

Just to let you know.
The whole change of the build flow/tools was with the intent to make automatic building a lot more easy in the (near) future.
With bash scripting you can do a lot, but at some level it becomes quite painful to make it be capable of doing all I would like to do and remain flexible.
PIO + Python makes it a whole lot more convenient.

I like this idea of running PIO+python and the new build scripts.
I just started with ESP32 as the firmware for ESP8266 just got stable so I can run my boards for a while with no issues and reboots.

[TD-er]

If i return the following MFD_ESP32_4M316k_2020_05_06.bin the system adds an extra .bin so the real binary filename:
esp32_create_factory_bin([".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin.bin"], [".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin"])

That's correct.
As you can see in the name generation function, it doesn't have the .bin extension.
For ESP32 we do generate 2 builds:

• -factory.bin - for initial flash via serial
• .bin - for OTA updates

ESP82xx:

• .bin - uncompressed
• .bin.gz - compressed

And for all there are also the .elf file needed for stack decoding.

[fmuntean]

If i return the following MFD_ESP32_4M316k_2020_05_06.bin the system adds an extra .bin so the real binary filename:
esp32_create_factory_bin([".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin.bin"], [".pio\build\MFD_ESP32_4M316k\MFD_ESP32_4M316k_2020_05_06.bin"])

That's correct.
As you can see in the name generation function, it doesn't have the .bin extension.
For ESP32 we do generate 2 builds:

• -factory.bin - for initial flash via serial
• .bin - for OTA updates

ESP82xx:

• .bin - uncompressed
• .bin.gz - compressed

And for all there are also the .elf file needed for stack decoding.

I see: 'factory' is not very clear. maybe -serial.bin will be more explicit.
I will then have somehow to change my DB recording scripts to manually add the .bin extension.

[TD-er]

The .bin will be added in the build process.
It is all automated now, you only have to give it the descriptive name and the rest will happen.
I guess you should try yourself.

[fmuntean]

I am all set now. I did not opened the ticket but I think it can be closed now based on my testing.