Tasmota: mqtt backup/restore command

[brnnrc]

Have you look for this feature in other issues and in the wiki?
It could be related to
https://github.com/arendst/Sonoff-Tasmota/issues/2394

Is your feature request related to a problem? Please describe.
I'd like to send a mqtt command to a device or device's group and receive the configuration backup by devices.
I'd like restore those configuration by mqtt command too.

Describe alternatives you've considered
I could download a single configuration by web interface but I have to know the device's IP (it could be dynamic) and I can't do it disabling web interface or all at once.

Thanks for your time.

[ascillato]

Hi,

By now, if you use a home automation system, you can set a script or automation to retrieve the actual _IP address_ of the device from the STATUS 5. Then you can save the config by wget [IP]/dl

That file will have as name: Config_[friendlyname]_5.14.0a.dmp.

I'm curious about how do you will save the config data on the MQTT Server Side?

[curzon01]

@ascillato - binary payload? MQTT is not restricted to text

Alternate way to get the IP. Send cmd

Backlog SetOption4 ON;IPAddress1;SetOption4 OFF

will result in specific topic

stat/..../IPADDRESS = {"IPAddress1":"x.x.x.x (y.y.y.y)"}

here you have the additional info if it is given by DHCP (x.x.x.x=0.0.0.0) or not (x.x.x.x=y.y.y.y)

[brnnrc]

@ascillato and @curzon01 thanks for your reply!
I'm using node-red to interface with devices by mqtt (using a mosquitto server). I know there are some ways to get the IP address and fire a wget command but I'd like turning off the tasmota web interface on my devices and addressing all at once (on backup command at least). Almost all web interface features seem to be mapped on mqtt commands but this one...

[arendst]

@brnnrc as the current backup/restore feature is using HTTP it's not so strait forward to change this to MQTT. Problem is buffer management.

I'll add the enhancement tag as I also would like to see a solution someday.

[ascillato2]

just for reference: https://github.com/arendst/Sonoff-Tasmota/issues/3181

[ascillato]

just for reference: #3531

[Luzifer]

just for reference: #3181

As that issue was closed in favour of this one just re-adding the request/intention from #3181:

The intention was not to have a binary dump of the Settings object but a possibility to use a plain text configuration (JSON, ENV-like file, TOML, ) which can be generated using a configuration management tool like Ansible. That way the configuration would not just sit on the device itself but could be provisioned automatically, checked for differences and updated if there would be differences.

The flow for example could be this:

• Ansible retrieves the current configuration of the device to a temporary file
• The file is altered from the variables stored inside the Ansible vars
• If the file changed the configuration is pushed to the device
• Temporary file gets deleted

Together with an inventory generated from DHCP - or in my case my Unifi controller - this could be used to auto-provision all Sonoff devices with a basic configuration and some special variables per device without having to do any manual interaction.

[curzon01]

As already mentioned (also from Theo) it occupies a lot of additional code memory to even create such play text readable info.

My suggestion - (preliminary: create a binary basic setting for each device type/group)

Restore a device setting

• post the binary config using http e.g. curl
• make the device specific changes using MQTT commands

Back a device setting

• wget/curl http//dl

[meingraham]

Would a built-in function that reads the stored binary and generates a JSON payload be a middle of the road compromise? In other words, not storing the configuration in text, only the code for the function that generates the textual configuration... and vice versa to restore.

[curzon01]

if you extract the C struct (doesn't matter if you output 'describe code' or the struct value itself) you need to handle each struct element - this cost the space
also if you adapt within a structure description array which "describes" all types of structure elements - cost also a lot of code

Anyway - one way would be writing a small external which extract the binary into readable text and reverse - I thought in past to wrote something but
a) had no time just now :-)
b) such solution have to synchronize to changes within Tasmota settings

[meingraham]

This would be a periodic task and probably used by only a limited number of TASMOTA users. So, it probably doesn't warrant the code space expense. An external to do the conversion of binary dmp to text and text to binary would fit the bill. In my case, I'd probably run the decoder only when I update the firmware in my device.

The reason I would want a "readable" configuration backup is to have an accurate documentation source that I can refer to. It would primarily be a reference for setting up any new devices, using it as a basis to edit and then restore onto a new device, or simply as an easily readable document which I (or anybody) can refer to in the overall documentation of the automation setup of my home.

I see how keeping the external function in sync with the corresponding TASMOTA data structures it needs to "understand" in order to perform the conversion is tricky. You'd have to have a version that corresponds to each of the TASMOTA version(s) in one's home. And then insuring the user uses the right decoder that corresponds to the particular TASMOTA version on the device. Question - is the TASMOTA firmware version in the same location in the binary dmp (e.g., at the beginning always)? That would serve as version "correspondence" check. I suppose the external could even have the ability to decode "all" TASMOTA versions and be kept updated to include all data structure version as the TASMOTA evolves. The latter could be quite a tall order!

I guess for now, I'll settle for storing the STATUS 0 output for each of my devices and use that to document my setup. That along with the saved dmp file for each device.

Regards.

Mike

P.S. I was thrilled to learn from this thread (I'm sure it's somewhere in the wiki) that I could enter http://<IP>/dl in my browser and that I could save the dmp as a file. BTW, when I performed a wget on my openHABian server, the wget <IP>/dl did not create a "named" file - the output was always to a file name dl. I had to manually rename the file to something like Config_[friendlyname]_5.14.0a.dmp. However, using a browser, it did prompt to save the file named "properly".

[curzon01]

@meingraham - still pulled with #4180 a version of decode-config.py under tools which may meet your requirements. You are now able to create readable configs for backup and upload (restore) it to your devices if needed.

@brnnrc:
This does not full meet your request but due to the circumstance to limited mqtt buffer length and others I don't think a mqtt backup will be feasible.

As you can use decode-config.py tool outside Tasmota web UI it may meet also your request to create bunch of backups in one task.
With disabled Web interface you can use a batch process which temporarly enables web admin interface via mqtt, request the backup and disables the web interface again via mqtt. This method can be enhanced to get also the IP from each device via mqtt before running decode-config.py.

[ascillato2]

@curzon01

Thanks a lot for your implementation. Super useful. :+1:

Closing issue as now the tool is now available for being used.

[meingraham]

@curzon01 Thanks!

Is this an issue with the @h replacement parameter or have I used it incorrectly? I ran this command:
./decode-config.py -d sonoff-masterbr_lamp-0043 --backup-file Config_@h_@f_@v

and the file name came out
Config_%s-%04d_Master_Bedroom_Lamp_Sonoff_6.1.1.2.json

[curzon01]

If you use %s-%04d as hostname the result is correct. It makes sense using @h only if you don't use placeholder in hostname too.

Don't expect the real hostname of the device, the source of @h is the configuration data, not the live dynamic network settings.

[meingraham]

Ah, yes, that's the setting in the configuration data. I thought it grabbed the hostname from the device by the same means that it retrieved the Friendly Name and the firmware version. No worries, I can encode the filename manually to add the hostname.

[curzon01]

All of the three infos are grabbed from the config data after the config data was pulled from Tasmota deviicw, also friendly name and version, none is grabbed from the device directly.
Otherwise it won't work if you use a file as source instead of the device.

[meingraham]

@curzon01

I don't understand the "architecture" well enough, obviously. Since I manually configured the Friendly Name through the web management GUI, I thought the tool was querying that from the device in the same way that it would query the firmware version and the hostname... somewhat in the manner that one can send a STATUS message and have the device respond with operational and configuration information stored on the device.

[terba]

I have more and more Tasmota devices, and none of them has the web server compiled in (space needed for MQTT TLS), so I can't backup the configurations. Some solution for MQTT-only setups would be really appreciated. Even an external tool could be used which queries all the settings over MQTT, if I would be able to restore it with one backlog command or with some begin-commit transaction handling like:

mosquitto_pub -t 'cmnd/sonoff/begin' -m ''
mosquitto_pub -t 'cmnd/sonoff/setoption1' -m '1'
mosquitto_pub -t 'cmnd/sonoff/setoption2' -m '1'
#...
mosquitto_pub -t 'cmnd/sonoff/commit' -m ''
# OR
mosquitto_pub -t 'cmnd/sonoff/begin' -m ''
mosquitto_pub -t 'cmnd/sonoff/backlog' -m 'setoption 1;syslog 1;...'
mosquitto_pub -t 'cmnd/sonoff/backlog' -m 'setoption 2;setoption3 1;...'
#...
mosquitto_pub -t 'cmnd/sonoff/commit' -m ''

[meingraham]

@terba

https://github.com/tasmota/decode-config

Backup
./decode-config.py -d --backup-file Config-@H-@f-@v --backup-type json

Restore
./decode-config.py -d --restore-file

[terba]

@meingraham Thanks for your comment. I have already seen this, but I think it doesn't support MQTT config retrieval and restore. My Tasmotas doesn't have a webserver, they are only reachable over MQTT.

[meingraham]

Ah, yes, I misread your post.

You are SOL.

[s-hadinger]

Since version 6.6.0 you can use both TLS and web server thanks to heavy TLS optimization. It could help.

[curzon01]

As Theo mentioned in the past Tasmota does not support config data backup by MQTT due to large needed buffer sending 4k at all. This will exceed the size possibilities of ESP so http is currently the only way to get the data from it.
The only way getting config data by MQTT would be change buffer management or spilt config it into smaller parts and send it as partitions but that's currently not in.

[meingraham]

@terba

As @s-hadinger you may want to consider putting the web server back into your firmware since TLS support has been optimized to fit while still having the web server enabled.

[terba]

My Tasmotas are fully separated on the network, they can not even see each other. Nothing can access them, and they have only access to my NTP, syslog, OTA and MQTT server. I wouldn't change that. And compiling webserver in just for backup is an overkill, I think. I will stick with my current error prone-method: I write the settings into a text file by hand :) Thanks for the inputs!
P.S.: If there will be some progress in MQTT backup support, drop a note here if I can help (with testing or anything).

[tobias-richter]

I came across this issue some days ago and I implemented an ansible role for sending commands to tasmota devices. Sadly this only helps people using tasmota with the webserver.
See: https://galaxy.ansible.com/tobias_richter/tasmota

Maybe this helps someone here :)

[issacg]

I stumbled on this thread when looking for a solution to #3785

I also use MQTT+TLS exclusively and due both to the historic issue of not being able to use TLS and WebServer (I know it's not so anymore, but needed a solution back then), and general security of wanting to keep webserver off, I developed an MQTT (with TLS support) solution to configure my devices. The code is available at https://github.com/issacg/autoconf

I hope this helps some of you!