Tasmota: Upgrade to Tasmota from Shelly mongoose trough SW ( not cable)

[seritos]

Have you looked for this feature in other issues and in the docs?
Yes
Is your feature request related to a problem? Please describe.
_A clear and concise description of what the problem is._
Be able to install Tasmota directly from the device
Describe the solution you'd like
_A clear and concise description of what you want to happen._

Describe alternatives you've considered
_A clear and concise description of any alternative solutions or features you've considered._

Additional context
_Add any other context or screenshots about the feature request here._

(Please, remember to close the issue when the problem has been addressed)

[ascillato2]

Hi,

The idea is nice, but shelly firmware don't provide any way of local file upgrading by OTA. Sorry.

[ascillato2]

Hi, sorry. You didn't provide much information about that.

After a web search I found https://github.com/mongoose-os-apps/shelly-homekit/blob/master/README.md and they found how to upload by OTA a new firmware.

So, seems there is no need to modify anything in Tasmota. Just add the information to the docs.

From your browser seems you just need to put:

http://shellyIP/ota?url=http://thehackbox.org/tasmota/tasmota.bin

We need to test that and if it works we can add it to the docs. It is very useful to change the firmware like this.

Thanks for sharing.

[seritos]

Oh!! great news. I didn't found it, but would try to test it soon.

Thanx!

[seritos]

@ascillato2 , I have created a case under the mongoose FW for the Shelly. The developer might support a developer somehow to create an intermediate FW that would allow to install 3rd party FW.
Is not as simple as download the .bin file as appeared

https://github.com/mongoose-os-apps/shelly-homekit/issues/69#issuecomment-628925608

[rojer]

hi. i am the author of the shelly-homekit firmware (and of mongoose-os). let me give a bit of context.
stock firmware uses this flash layout.
stock fw will accept ota to any mongoose os app with that layout as long as the app name matches.
name and flash size are key here - https://github.com/mongoose-os-apps/shelly-homekit/blob/master/mos.yml#L60

mongoose os application can be running from app0 or app1 slot (depending on the number of OTA updates performed to date, it starts from app0 on initial flash and switches on every update).
i see that tasmota firmware size is about 600K, this is well within limits. i'll take a guess that it expects to be flashed from address 0, this means that the mos application that performs flashing needs to be running from app1 slot. this can be done by simply repeating OTA update if it happened to be in app0 slot initially.
so, once the intermediate mongoose-os app is up and running in app1 slot, all it needs to do is download and flash the tasmota firmware and reboot.

[arendst]

Thx for the info.

Tasmota indeed needs to be loaded from address 0.

[seritos]

Thanx so much, @rojer

I personally see a huge potential on this intermediate FW.

There is a massive amount of people who has a lack of knowledge in programming and/or soldering that cannot make the step of installing a 3rd party FW because of this.

I believe that it would be the first comercial product that would easily allow to install any FW you wish from stock.
It’s a win-win case!!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rojer]

i also think it's a good idea. i am happy to assist anyone who want to give it a go.

[yaourdt]

i came across this issue a few days ago and decided to give it a try. at this point i have an intermediate mongoose fw running in app1 partition that will download tasmota and assemble it in app0. after assembly is done, tasmota will be moved to the beginning of esp8266 flash and the device reboots. this way, network problems or faulty downloads won't brick the device, as we can retry before overwriting the bootloader.

the firmware is currently working on a nodemcu dev board, there are however two things that need to be done before i can release it on github:
1) detect if we are booting from app0, in this case copy the fw code to app1, change boot config and reboot. at the moment, booting from app0 will overwrite the running code and brick the device
2) app0 flash address for caching tasmota is hard coded for the 2MB layout at the moment. we need a better way to determine where app0 is, as this changes for the different layouts

as far as i understand, both tasks can be performed via the closed-source ota common lib in mongoose os. could you help me with interfacing the libs api, @rojer ?

[rojer]

@yaourdt excellent, you've made great progress!
you can get boot configuration by using get_rboot_config() from esp_rboot.h.
the struct is declared here and contains all the bits you need to figure out current boot status as well as to change status for the next boot.
you just need to swap previous_rom and current_rom.
copy rom_sizes[0] bytes from roms[0] to [1], fs_sizes[0] bytes from fs_addresses[0] to [1].
set current_rom to 1 and previous_rom to 0.
set fw_updated and is_first_boot to 1 as well, for good measure.

[yaourdt]

thank you for your fast reply, @rojer !
i'm setting config via platforms/esp8266/rboot/rboot/appcode/rboot-api.h -> rboot_set_config() and a first reboot works as expected, booting from app1:

[Jun 20 20:48:43.700] main.c:208 ******** current_rom 1
[Jun 20 20:48:43.707] main.c:209 ******** previous_rom 0
[Jun 20 20:48:43.708] main.c:210 ******** fw_updated 1
[Jun 20 20:48:43.714] main.c:211 ******** is_first_boot 1

after reset, however:

[Jun 20 20:48:53.833] main.c:208 ******** current_rom 0
[Jun 20 20:48:53.838] main.c:209 ******** previous_rom 0
[Jun 20 20:48:53.842] main.c:210 ******** fw_updated 0
[Jun 20 20:48:53.847] main.c:211 ******** is_first_boot 1

do i still need to commit the firmware after first successful boot from app1?

apart from this minor issue everything else seems to be working, and i'll post a link here as soon as i get around to stiching it all togehter some time next week.

[rojer]

hm, it should be committed automatically. can you post full debug.level=3 log of the first boot?

[rojer]

or share the app with me, i will test it

[yaourdt]

this is the full log:

nodemcu_v3.log

i just invited you to the repo with a minimum working example

[rojer]

the problem is that you did not include OTA library, that's why firmware was not being committed after successful boot.
i sent you a pull request to fix, also added settings that should make the firmware compatible with shelly 1.

[yaourdt]

i made some progress, and you can now clone and test a first working draft: yaourdt/mgos-to-tasmota

warning: this code is not yet as reliable as it should be for pure ota use. if you cannot recover your device over a wired connection, do not use this software yet!

having said that: i'd appreciate, if someone could test this on a Shelly1 and confirm it works for them as well. before starting the upgrade, please ensure you have the Shelly1 connected to a wifi network.

@rojer - there is one last i currently don't understand why it is happening: after downloading the last http chunk it can take up to four minutes before MG_EV_HTTP_REPLY is called (the call only happens after the server closes the tls connection). my quick fix is a timeout of 30 seconds after the last block write to force the connection to close. i tried different servers, issue stays the same. do you have any idea why this may happen?

[rojer]

@yaourdt this is a bug in mongoose, fixed in https://github.com/cesanta/mongoose/commit/95fcb261ebb44a9664c336824f0968d56faaa50b
in will be in mongoose 2.18 release (coming soon), meanwhile you can switch to latest (=master) branch by specifying

libs_version:        latest
modules_version:     latest
mongoose_os_version: latest

in the manifest

[yaourdt]

thanks, @rojer that did the trick.

i added a build pipeline and did some further cleanup of the code, so the firmware can now be tested using

http://shellyip/ota?url=https://github.com/yaourdt/mgos-to-tasmota/releases/latest/download/fw.zip

as OTA update url. feedback is very welcome, for problems please open an issue at yaourdt/mgos-to-tasmota.

warning: this code is not yet as reliable as it should be for pure ota use. if you cannot recover your device over a wired connection, do not use this software yet!

[digiblur]

nice job! @yaourdt I tried on a Shelly1 but can't seem to get it to work. I'll head over to issues thread there. Very promising!

[yaourdt]

thanks to @digiblur and @rojer i managed to fix the remaining issues and it is now working for Shelly1 switches. i also added some other shellys, but i don't have any hardware besides a Shelly1 and my dev-board to confirm it is actually working, so feedback would be very welcome!

if the other shellys work as well, i think we can document and close this issue. i'll try to write a firmware for going from tasmota to mongoose as well, as soon as i get around to it, and open a new issue once it is safe to use and can be documented.

[TriderG75]

thanks to @digiblur and @rojer i managed to fix the remaining issues and it is now working for Shelly1 switches. i also added some other shellys, but i don't have any hardware besides a Shelly1 and my dev-board to confirm it is actually working, so feedback would be very welcome!

if the other shellys work as well, i think we can document and close this issue. i'll try to write a firmware for going from tasmota to mongoose as well, as soon as i get around to it, and open a new issue once it is safe to use and can be documented.

I've tried with a Shelly 1 too. It works perfectly. Thanks very much.

[rojer]

@yaourdt good job so far, but we need to make the process safer. since we start writing from address 0, first thing we do is overwrite bootloader and its config and enter danger zone: if connection is terminated and we reboot (say, lose power), we'll attempt to boot a partially flashed tasmota firmware which is almost guaranteed to cause a bricked devices.
when writing, you need to set aside bootloader (0...4096) and its config (BOOT_CONFIG_ADDR ... BOOT_CONFIG_ADDR) and only write them out if the rest was downloaded and written successfully.
you can use free space at 0x100000 or just keep them in memory.

[rojer]

also, i see that large part of the app is dealing with keeping track of sector boundaries and erasing as needed.
you can use esp_flash_write, it will erase as necessary and make sure writes are properly aligned:

#Include "esp_flash_writer.h"

static struct esp_flash_write_ctx s_wctx;

...
case MG_EV_CONNECT::
  esp_init_flash_write_ctx(0x2000, (0x100000 - 0x2000));
...
case MG_EV_HTTP_CHUNK:
   esp_flash_write(&s_wctx, hm->body);

[yaourdt]

@rojer i didn't know about the esp_flash_writer, that will simplify things indeed, thank you! i'll upgrade that as soon as i get around to it this week.

with respect to the safety: while i agree in principle, i don't think the current implementation is very dangerous, as we first cache in app0 partition. so if there is power loss, interrupted network connection, etc during download, we can still reboot to app1 and retry as often as we like. this is why we do not download to 0x0 directly.
only after all is cached in app0, we move data blockwise to address 0x0. and yes, in this very moment, we might end up with a partly moved tasmota if we lose power, however, the process of moving blocks from cache to 0x0 takes less than 10 seconds and is unlikely to fail.

nonetheless, yours is the more robust approach, so as soon as i'll get around to it this week, i'll try to lower the risk further (we can as far as i understand not avoid it entirely, as we'll have to erase and write the first block eventually) by caching first block and boot config in memory and overwrite them last, as you suggest.

I'll also try to implement a tasmota-to-mongoose variant, so people can go back

[rojer]

we first cache in app0 partition. so if there is power loss, interrupted network connection, etc during download, we can still reboot to app1 and retry as often as we like. this is why we do not download to 0x0 directly.

oh, ok, somehow i missed that. so it's not quite as dangerous, but still - we can make it safer _and_ more efficient by writing most of the image to the final destination and overwriting bootloader and the config last. you are right that there's still a little bit of risk involved and i don't think there's a way to avoid that completely.

[digiblur]

@yaourdt Let me know when you have Shelly1PM, Shelly2, Shelly2.5, and ShellyEM. I have some Shelly Floods but I won't be flashing those as I use them with MQTT as is and they sleep most of the time anyways.

[yaourdt]

@digiblur Shelly1PM, Shelly2, and Shelly2.5 are already available, but those versions are untested, as i don't have the hardware. you can give them a try if you like

[digiblur]

Sure. I'll let you know! Thanks

[seritos]

@yaourdt: First of all, thanks a lot for your great job. This opens a new world for many people !!

....Would it be possible to compile this software for use it also with HAA ( https://github.com/RavenSystem/haa/releases/latest/download/fullhaaboot.bin) ?

About HAA: Add native HomeKit support, OTA updates and some custom settings to any device with an ESP8266 or ESP8285 microcontroller. This means that bridges like HomeBridge are not necessary.

... and last but not least, your method has been published in several groups and the feedback is excellent!

Thanks!!

[yaourdt]

@seritos I'm happy if people use and like this tool! If you or @digiblur can confirm the other versions for Shelly2,... etc are working, I'd like to remove the untested warning with the next release (which will improve the details discussed with rojer further up).
As for your request with HAA: If you have a Shelly1 with stock firmware you can try this link:

http://shellyip/ota?url=http://dl.dasker.eu/haa-dev/fw.zip

Just replace shellyip with the IP address of your Shelly1. I had a look at the HAA installation instructions and it should be working, but it is untested. If it works, I'll add it to the next release

[yaourdt]

@rojer - I'm currently working on an ota firmware to convert back to mongoose os. I plan on building an intermediate firmware that will use ota-http-server to apply the shelly stock firmware as an update. In order for this to work, the intermediate mongoose firmware image should be less than 512K in size. I had a look at mongoose-os/platforms/esp8266/Makefile.build and tried different build_vars, but there seems to be no way to shrink app0 rom_size in a manner that rboot, rboot cfg, and app0 all fit within the first 512K. Is there a way to make this work? Or do I need to switch to the arduino framework?

[rojer]

@yaourdt the following manifest produces firmware of about 440K

version: 1.0

libs_version: latest
modules_version: latest
mongoose_os_version: latest

sources:
  - src

filesystem:
  - fs

libs:
  - origin: https://github.com/mongoose-os-libs/wifi
  - origin: https://github.com/mongoose-os-libs/mbedtls
    variant: esp8266-noatca
  - origin: https://github.com/mongoose-os-libs/mongoose
    variant: esp8266-nossl

build_vars:
  FS_SIZE: 65536

manifest_version: 2017-09-29

edit: changed version to latest

[yaourdt]

@rojer: very nice, thank you very much!

[digiblur]

@seritos I'm happy if people use and like this tool! If you or @digiblur can confirm the other versions for Shelly2,... etc are working, I'd like to remove the untested warning with the next release (which will improve the details discussed with rojer further up).
As for your request with HAA: If you have a Shelly1 with stock firmware you can try this link:

http://shellyip/ota?url=http://dl.dasker.eu/haa-dev/fw.zip

Just replace shellyip with the IP address of your Shelly1. I had a look at the HAA installation instructions and it should be working, but it is untested. If it works, I'll add it to the next release

I did test a Shelly1 and Shelly 2.5 on a live stream a week ago or so. Worked great! I need to find some firmware for the Shelly 2 and Shelly1 PM to go back as I flashed it a good while back. I think I have a plug too for testing.

[yaourdt]

I did test a Shelly1 and Shelly 2.5 on a live stream a week ago or so. Worked great!

Nice!

For the stock firmware, there is a list with downloads from the official API: https://github.com/mongoose-os-apps/shelly-homekit#reverting-to-stock-firmware (I'll be using those for the tasmota->mongoose version of this firmware)

[seritos]

@seritos I'm happy if people use and like this tool! If you or @digiblur can confirm the other versions for Shelly2,... etc are working, I'd like to remove the untested warning with the next release (which will improve the details discussed with rojer further up).
As for your request with HAA: If you have a Shelly1 with stock firmware you can try this link:

http://shellyip/ota?url=http://dl.dasker.eu/haa-dev/fw.zip

Just replace shellyip with the IP address of your Shelly1. I had a look at the HAA installation instructions and it should be working, but it is untested. If it works, I'll add it to the next release

Thanx much!

I have request some people in the forum to test it with a Shelly 1
I will have access tomorrow to some Shelly 2.5. Would this version work with it?

Thanx!!

Edit: Just test it in Shelly 1. Works flawlessly !!!

[yaourdt]

@seritos this awesome, then I'll just add it to the next release for all supported shellys. In the meantime, this is a build for the shelly2.5: http://shellyip/ota?url=http://dl.dasker.eu/haa-dev/fw25.zip

[yaourdt]

i released haa versions for all supported devices, please refer to yaourdt/mgos-to-tasmota for instructions on how to install them.

[seritos]

i released haa versions for all supported devices, please refer to yaourdt/mgos-to-tasmota for instructions on how to install them.

Hi @yaourdt!

I am trying to flash some new 2.5 with hAA and is not working. I have seen that if replace mg2tasmota by mg2haa, the link http://dl.dasker.eu/firmware/mg2haa-Shelly25.zip doesn't work, and therefore is not possible to update.

is the link broken?

Thanx!!

[yaourdt]

@seritos - it seems that the upload to the ftp server did not work for some builds, but the actions workflow still finished without error, so i'll have to investigate. i've uploaded the release manually, and it should work now. sorry, i should have checked the final download before posting, not just the release.

[seritos]

@seritos - it seems that the upload to the ftp server did not work for some builds, but the actions workflow still finished without error, so i'll have to investigate. i've uploaded the release manually, and it should work now. sorry, i should have checked the final download before posting, not just the release.

Thanx much for you quick reply. I will test it this evening.
FYI, I flashed them with Tasmota instead, and worked perfectly.

Edit: All perfectly well working!!

[digiblur]

@yaourdt thank you for all of your hard work on this project! I did a tutorial on it. Enjoy. https://youtu.be/_oRr8FZyyQ0

[yaourdt]

@digiblur this is really awesome, thank you! I'll add more devices tomorrow or on Monday and would like to link your video in the readme file, as you explain the process in way more detail. Is this okay for you?

PS: I also don't know how to pronounce my username 😁

[digiblur]

That would be great to include it!

And I fully agree with the waiting for someone to confirm the process works on a device before putting them in "production".

[yaourdt]

http://shellyip/ota?url=http://dl.dasker.eu/test/shellyDimmer1.zip
http://shellyip/ota?url=http://dl.dasker.eu/test/shellyDimmer2.zip
http://shellyip/ota?url=http://dl.dasker.eu/test/shellyRGBW2.zip
http://shellyip/ota?url=http://dl.dasker.eu/test/shellyEm.zip
http://shellyip/ota?url=http://dl.dasker.eu/test/shellyBulb.zip
http://shellyip/ota?url=http://dl.dasker.eu/test/shellyVintage.zip

I completely forgot to mention this here: There is some new devices I added. Feel free to test them and give me feedback before I add them to the next release.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem.