Ems-esp: (v2) normalize command infrastructure for devices (mqtt & console)

[proddy]

The MQTT commands inherited from v1.9 and the new Console commands are not always in sync. The code needs some refactoring and simplification. This will also help reduce the memory hog when running mqtt, telnet, and web at the same time on an ESP8266.

Specification

1) sending commands to EMS-ESP is done by a single topic called cmd. The payload determines the device/sub-system and the action. An additional id field can be used to set the heating circuit. For example:

• topic=cmd payload={"target":"boiler", "cmd":"flowtemp", "data":55}
• topic=cmd payload={"target":"thermostat", "cmd":"mode", "data":"auto"}
• topic=cmd payload={"target":"thermostat", "cmd":"mode", "id":2, "data":"auto"}
• topic=cmd payload={"target":"system", "cmd":"fetch"}
• topic=cmd payload={"target":"system", "cmd":"gpio", "id":1, "data":"on"}
  2) The exception to the above is for Home Assistant's default climate component. Here the subscribed topic will be homeassistant/climate/ems-esp/hc<num>/<cmd_temp|cmd_mode>. At some point, we should have our own component installed via HACS.
  3) Console commands will be derived from the MQTT commands and accessible via set <cmd> <data> [hc] in the associated context. For example:

ems-esp:/$ boiler
ems-esp:/boiler$ set flowtemp 55
ems-esp:/boiler$ exit
ems-esp:/$ thermostat
ems-esp:/thermostat$ set temp 20 2
ems-esp:/thermostat$ set mode 2 auto

This will simplify the code significantly. When using a console command it will just invoke the MQTT function effectively simulating an incoming MQTT request. It will mean each command function will take now a char * and be responsible for deciphering the values from the payload string. For example the setcomfort() function will no longer an int values 1, 2 or 3 but "hot", "eco", "intelligent".

4) The WebUI will be used for configuring EMS-ESP. The console will have a few set commands for debugging mainly such as setting the wifi credentials or changing the UART tx mode

[proddy]

@MichaelDvP anything to add?

[MichaelDvP]

If all home automation systems can do that, it's ok.
Before you use "data", now "value", is there a special reason?
But i think normal json with : instead of = and how do you want to manage the different hc on thermostat?

{"target":"thermostat", "hc":2, "cmd":"mode", "value":"day"}

is than in telnet? As a context or a third parameter?

# thermostat
# hc 2
# mode day

The mqtt commands should have exactly the same name as the mqtt pubish. i.e. if published as mode the command should be mode not setmode.

How do you think about nested commands like {"thermostat":{"hc2":{"mode":"auto"}}} and
#thermostat hc2 mode auto

[proddy]

i was making changes while you were replying it seems - I think most of your comments have been addressed. I was thinking whether we should use 'mode' or 'setmode'. The reason I choose 'set' was 1) because it's an action and 2) maybe one day we'll need to add a get command to query a value via MQTT. But thinking about it more you're right. How about the command is 'mode' and in the console the command would be set mode <hc> <data>

[proddy]

worked a bit of this today. It changes a lot of files so needs careful testing.

[MichaelDvP]

I have thought about that an tested in small range only inside thermostat and would prefer if the command is
set <cmd> <data> [hc]
with hc optional as last parameter defaults to first active hc if not set. Also there is no confusion with thermostat global- and boiler-commands: set <cmd> <data>

(this was my test-command finally: thermostat_set_command.txt)

[proddy]

that's good since my implementation follows that same standard. I fixed a few bugs along the way too. Now you just need to register the command and it'll will propagate to the mqtt and console functions. Since it's quite a major change I'll create a feature branch and let you know when it's pushed.

[proddy]

@MichaelDvP I'm working on the thermostat bit now. I see you implemented parsing of nested commands like
{"hc2":{"temp":21}}.

Is this important to have? It adds a lot of complexity to the new code I'm refactoring. Wouldn't just
{"cmd":"temp", "hc":"1", data="21"} suffice? Or do you have a specific use case.

[MichaelDvP]

No, not important. I added this a while ago in 1.5 for me and when switching to 2.0 some of my scripts were using it, but easy to change.
For the actual splitting in boiler_cmd/boiler_data and thermostat_cmd/thermostat_data i also find it more intuitive to use the same structure for data and command. But if you go for a single cmd it's clear to use a different structure.
(I think you plan to deserialize in the cmd-function to check the target and the thermostat-on-command will receive cmd, data, hc seperated)
BTW: i forgot some thermostat commands to add to cmd/data.

[proddy]

Right now I took a slightly different approach. I eliminated the 'target'. Now each device when its recognized will subscribe to its own MQTT topic called "<device>_cmd" like thermostat_cmd and boiler_cmd. There is a special one for the system.

The payload must have a cmd and optional are data, id, or hc. So these are all valid

thermostat_cmd {"cmd":"temp", "hc":1, "data"="21"} // hc must be numeric
thermostat_cmd {"cmd":"temp", "data"="21"} // hc is optional
thermostat_cmd {"cmd":"temp", "data"="21", "id":1} // id here is the same as hc
boiler_cmd {"cmd":"comfort", "data"="eco"}
system_cmd {"cmd":"gpio", "id":1, "data"="on"} // sets D1 to HIGH

this does impose some restrictions imposed by the json library that I'm not happy about, like

• the data must be always be a string, so represented in quotes e.g. "data":"blah"
• the id and hc must always be a numeric value, e.g. "id":1
• it assumes there is only one data element and an optional id. It most cases this is fine but there may be situations where more variables are needed, like 'set temp holiday 22?

will this work?

[proddy]

also @MichaelDvP I'm not to happy using the set command as it conflicts with the internal settings of EMS-ESP.
e.g. set master <n> or set tx_mode <n> change the settings which are also available via the web.

where set comfort eco in the boiler context is used to request a change on an EMS device. I think a better word would be call or cmd or change.

?

[MichaelDvP]

cmd looks a bit odd in the help as cmd <cmd> <data>, change <cmd> <data> is much better, or use ems <cmd> <data> since it changes a ems value.

on the other hand, cmd is more in line with the mqtt command.

[proddy]

It really needs to be a verb, like set is. So options are
run <cmd>
exec <cmd>
call <cmd>
do <cmd>

I'll stick with call for now. It's only one line that needs to be changed anyway ;-)

[proddy]

I created a branch https://github.com/proddy/EMS-ESP/tree/v2_cmd with the latest changes. There's quite a bit. Still needs some more testing but let me know if you run into issues.

This branch has all the latest updates from v2.

I've been using the standalone build for testing. If you have gcc installed type make run and it'll run the simulation with a few tests being automatically executed.

[MichaelDvP]

I made a real word test first, but limited to my system.
some fixes:

• helpers:value2xxx should be called by reference i.e. value2float(const char * v, float & value)
• thermostat::set_temperature_value should have hc as int8_t
• use AUTO_HEATING_CIRCUIT instead of DEFAULT
• mixing gpio and Wemos-D-numbers leads to confusion.

some observations:

• free memory seems to be much lower now (~22% with telnet open, ~42% with telet closed)
• only subscriptions for thermostat and boilder, system_cmd is missing
• mqtt reconnects every 5 minutes (exactly 297 seconds)
• direct after the reconnect i get these messages from [mqtt]:

000+00:33:48.578 I 34: [mqtt] MQTT disconnected: TCP
000+00:33:53.593 I 35: [mqtt] MQTT connected
000+00:33:54.200 E 36: [mqtt] No MQTT handler found for topic thermostat_cmd and payload {"cmd":"summertemp","data":"14"}
000+00:33:55.903 I 37: [thermostat] Setting thermostat temperature to 14.0 for heating circuit 2, mode summer
000+00:33:55.903 E 38: [mqtt] No MQTT handler found for topic thermostat_cmd and payload {"cmd":"summertemp","data":"14"}
000+00:38:50.559 I 39: [mqtt] MQTT disconnected: TCP
000+00:38:55.575 I 40: [mqtt] MQTT connected

[proddy]

thanks for testing and the feedback.

[proddy]

only subscriptions for thermostat and boiler, system_cmd is missing

I called it system, Should it be system_cmd ?

[proddy]

mixing gpio and Wemos-D-numbers leads to confusion.

What do you suggest? Stick to the GPIO numbers or D numbers? It uses D* numbers now and maps to gpio values (using your code). Should the command be renamed to "pin" for example?

[proddy]

mqtt reconnects every 5 minutes (exactly 297 seconds)

I'm not seeing this. Been running for 15hrs will no MQTT errors. Make sure you don't have multiple EMS-ESPs running (e.g. ESP8266 and ESP32) using the same MQTT client ID.

I'll keep testing

[MichaelDvP]

The esp32 has a lot of spare gpio, most without wemos D-numbers, so it's better to use gpio, but readme/wiki should note the difference to D-numbers.

The reconnects are gone when i clear the value in my mqtt broker. Also there no more error messages. I guess the commands are registered after subscription of thermostat_cmd, but the broker connection is publish on subscribe. I'll try with QoS.

I don't have a system subscription, but i see this.
// add the system MQTT subscriptions, only if its a fresh start with no previous subscriptions

[proddy]

there was a bug in the MQTT re-subscribe so I fixed that. Also renamed the topic to "system_cmd" so it's consistent with the rest.

I'll change to use the gpio numbers. This also means your MQTT logic will change too as I believe the code you added took pin numbers D0-D3. Ok if I change that?

[MichaelDvP]

The D0-D3 was from #375. Since the command has changed, there is no problem. I think it's clear that `gpio' command also means gpio-numbers.

[proddy]

changed it anyway :-) pin <gpio> [data]. In MQTT the id is the gpio,

[proddy]

The latest build 2.0.0b12-2 in v2_cmd addresses some of the memory issues. What I did is

• experimented with using static memory (on stack) vs dynamic (heap) for the Rx, Tx, Mqtt queues. This did not make a huge difference and with the risk of stack overflowing, I decided to keep it in dynamic memory. When compiling with EMSESP_DEBUG flag it will show the free heap and defragmentation on the console.
• experimented with storing std::shared_ptr to objects in the vector rather than the objects themselves, but it had little effect since all the strings were stored in flash memory anyway.
• replaced std::deque with std::list. It uses less memory per object (around 5%). I also tried custom arrays and my own implementation of a poor-mans queue using C++ templates which was fun to program in 30 degrees heat, but didn't yield any gain so dumped it.
• replaced all std::binds with C++ lambda functions. Each bind takes up 16 bytes on the heap. Using lambda functions which is new in C++11 reduces this overhead by half.
• added std::moves where I could in the code to reduce object copying and creating dud memory blocks in the heap space
• the commands (console & mqtt) are built dynamically based on the device type and device flags. So an RC20 thermostat will get fewer commands than an RC300, and also a lot more free system memory.
• I reduced the Rx buffer from 20 to 10 commands. Since Rx telegrams are processed immediately and there is already a circular buffer in the UART code it will probably never go higher than 1 anyway. At some point it could be just a static array with 2 objects.
• with the few remaining std::vectors left I blocked out some heap memory using the .reserve() function. This helps with defragmentation.
• Michael's changes to using staticjosn for the boiler and solar also probably helped on the web side.

So compared to 2.0.0b11 from the v2 branch, the available heap on my ESP8266 has gone up from 64% to 75%. The web loads all the time now and shows around 2% fragmentation and 65% free memory from within the UI. With both telnet and Web active its around 55% free memory.

If you guys can test, let me know of any feedback and then I'll merge it back into v2 and remove this branch.

[bbqkees]

Testing now. Indeed the web interface loads nicely on the ESP8266! Lets see how this holds up over the weekend.

What I did notice since b11 is that I now have incomplete RX telegrams, which I have never seen before.

Do you plan on moving around specific settings in the web interface or do you plan to leave it as it is now?
I need to start writing new instructions for V2.

[MichaelDvP]

Very smooth, i'm testing now on 8266 with NTP enabled and 80MHz. Now issues so far. Web keeps responsive and reloads fast, also with parallel telnet session. After closing telnet heap fragmentation (as percentage) is a bit higher than before, but the largest block is bigger and always >9k, enough for good web response. If telnet is reopend fragmentation is low again.
Also after some telnet open/close/open.. all is stable.

[proddy]

thanks for testing. I have 1 incomplete Rx too (meaning a corrupt telegram). I don't remember seeing that before so will need to check why. I also noticed the console (not just telnet, also serial) doesn't free up all the heap when it closes the session. There's always a 1-2kb left behind. It's always been like that in v2 and I haven't been able to find the root cause yet. I eventually gave up searching as it's not such a huge problem I think as only power-users would use the telnet interface.

[MichaelDvP]

i get two subscriptions for boiler_cmd on the mqtt-broker, but none for system_cmd,

mqtt.0 | 2020-08-13 14:54:35.827 | info | (1502) Client [ems-esp] publishOnSubscribe
mqtt.0 | 2020-08-13 14:54:35.826 | info | (1502) Client [ems-esp] subscribes on "mqtt.0.ems.thermostat_cmd"
mqtt.0 | 2020-08-13 14:54:35.626 | info | (1502) Client [ems-esp] publishOnSubscribe
mqtt.0 | 2020-08-13 14:54:35.625 | info | (1502) Client [ems-esp] subscribes on "mqtt.0.ems.boiler_cmd"
mqtt.0 | 2020-08-13 14:54:34.827 | info | (1502) Client [ems-esp] publishOnSubscribe
mqtt.0 | 2020-08-13 14:54:34.826 | info | (1502) Client [ems-esp] subscribes on "mqtt.0.ems.boiler_cmd"
mqtt.0 | 2020-08-13 14:54:34.815 | info | (1502) Client [ems-esp] reconnected. Old secret 1597320264839_7724. New secret 1597323274814_4982

also show mqtt give:

MQTT subscriptions:
 topic: ems/boiler_cmd, [cmd]: comfort wwactivated wwtapactivated wwonetime wwcirculation flowtemp wwtemp burnmaxpower burnminpower boilhyston boilhystoff burnperiod pumpdelay
 topic: ems/thermostat_cmd, [cmd]: wwmode temp mode remotetemp datetime minexttemp clockoffset calinttemp display building language control pause party holiday nighttemp daytemp nofrosttemp ecotemp heattemp summertemp designtemp offsettemp holidaytemp

MQTT queue is empty

sending commands result in:

000+00:11:49.277 E 1: [mqtt] No MQTT handler found for topic ems/system_cmd and payload {"cmd":"pin","data":"on","id":"5"}
000+00:12:52.777 E 2: [mqtt] No MQTT handler found for topic ems/system_cmd and payload {"cmd":"send","data":"0B 90 37 20 00"}

[proddy]

strange. works for me. I'll do some further testing. Are you on ESP8266 or ESP32?

[MichaelDvP]

I'm on esp8266, NTP and 80MHz.

Another strange thing. show devices lists for every device all known type-ids:

These EMS devices are currently active:

Boiler: GB125/MC10 (DeviceID:0x08, ProductID:72, Version:03.00)
 This Boiler will respond to telegram type IDs: 0x10 0x11 0x18 0x19 0x34 0x1C 0x2A 0x33 0x14 0x35 0x15 0x16 0x1A 0xD1 0xE3 0xE4 0xE5 0xE9 0xA3 0x06 0x3E 0x3D 0x48 0x47 0x52 0x51 0x5C 0x5B 0xA5 0x37 0xAA 0xAB 0xAC
 Subscribed MQTT topics: boiler_cmd

Thermostat: RC35 (DeviceID:0x10, ProductID:86, Version:01.18) ** master device **
 This Thermostat will respond to telegram type IDs: 0x10 0x11 0x18 0x19 0x34 0x1C 0x2A 0x33 0x14 0x35 0x15 0x16 0x1A 0xD1 0xE3 0xE4 0xE5 0xE9 0xA3 0x06 0x3E 0x3D 0x48 0x47 0x52 0x51 0x5C 0x5B 0xA5 0x37 0xAA 0xAB 0xAC
 Subscribed MQTT topics: thermostat_cmd

Mixing Module: MM10 (DeviceID:0x21, ProductID:69, Version:02.01)
 This Mixing Module will respond to telegram type IDs: 0x10 0x11 0x18 0x19 0x34 0x1C 0x2A 0x33 0x14 0x35 0x15 0x16 0x1A 0xD1 0xE3 0xE4 0xE5 0xE9 0xA3 0x06 0x3E 0x3D 0x48 0x47 0x52 0x51 0x5C 0x5B 0xA5 0x37 0xAA 0xAB 0xAC

Controller: BC10/RFM20 (DeviceID:0x09, ProductID:68, Version:02.03)
 This Controller will respond to telegram type IDs: 0x10 0x11 0x18 0x19 0x34 0x1C 0x2A 0x33 0x14 0x35 0x15 0x16 0x1A 0xD1 0xE3 0xE4 0xE5 0xE9 0xA3 0x06 0x3E 0x3D 0x48 0x47 0x52 0x51 0x5C 0x5B 0xA5 0x37 0xAA 0xAB 0xAC

Maybe there is something wrong with my built (merging your v2_cmd). I'll fetch your v2_cmd to a new branch and compile again.

[proddy]

is your MQTT Qos 1 or 2? I didn't test that

[proddy]

This Thermostat will respond to telegram type IDs: 0x10 0x11 0x18 0x19 0x34 0x1C 0x2A 0x33 0x14 0x35 0x15 0x16 0x1A 0xD1 0xE3 0xE4 0xE5 0xE9 0xA3 0x06 0x3E 0x3D 0x48 0x47 0x52 0x51 0x5C 0x5B 0xA5 0x37 0xAA 0xAB 0xAC

I see this too. it's a bug.

[MichaelDvP]

is your MQTT Qos 1 or 2? I didn't test that

QoS 0 and CleanSession on

(I've changed mqtt.cpp line 86 to subscribe(1, topic, cb); and system_cmd worked again.)
Now compiling a fresh branch withou chnages.

[proddy]

i fixed the bug with show devices too, checking that it now

[MichaelDvP]

With clean build system_cmd is subscribed and working.

[MichaelDvP]

Yes, now mqtt subscriptions and type-ids are ok, heap is a little bit less.

I also noticed the console (not just telnet, also serial) doesn't free up all the heap when it closes the session. There's always a 1-2kb left behind.

It's only 0.1-0.2kb for me, and after ~5 times opening/closing the console memory keeps stable and does not drop further. After a while free memory goes up again. I don't think it's a problem.

[bbqkees]

B12 is still running fine on ESP8266.
Uptime of one day now, still a responsive and instantly loading web interface.

Its just those incomplete RX telegrams that I never had before.

[proddy]

thats good. same thing here, running fine for 24hrs with 3 corrupted Rx telegrams today, like:

ERROR [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 90 (CRC 90 != 10)

because its missing the last byte of the telegram. A correct telegram looks like

OK [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 FF DF (notice the extra byte)

Why this has changed I don't know since the UART code is the same as with the previous builds. Still looking into it

-- edit --
I've moved this conversation back to https://github.com/proddy/EMS-ESP/issues/398#issuecomment-674116979

[FredericMa]

I took the latest code for a run but it looks like some thermostat commands are missing:

ems-esp:/# show mqtt
MQTT is connected
MQTT Heartbeat is enabled
MQTT publish fails: 0

MQTT subscriptions:
 topic: ems-esp/boiler_cmd, [cmd]: comfort wwactivated wwtapactivated wwonetime wwcirculation flowtemp wwtemp burnmaxpower burnminpower boilhyston boilhystoff burnperiod pumpdelay
 topic: ems-esp/thermostat_cmd, [cmd]: wwmode temp mode
 topic: ems-esp/system_cmd, [cmd]: pin send

MQTT queue is empty

I have a Junkers FW200.

This command if failing on topic ems-esp/thermostat_cmd :
{"cmd":"nofrosttemp" ,"data":17, "id":1}
I see this error: MQTT error: no matching cmd: nofrosttemp

With previous firmware the following command was working on the same topic:
{"hc1":{"mode":"nofrost","nofrosttemp":"28"}}

Maybe I missed something?
Is it also still possible to chain command like before? Like settings the mode and temp at the same time like the example above?

Thanks for your efforts!

[proddy]

cool that you tried it out @FredericMa , thanks. The missing Thermostat commands is a bug, I just forgot to extend the commands to also include Junkers. It's an easy fix.

chaining commands in a single MQTT got removed in the new way we now handle commands. But it's a nice idea. It would mean supporting two different structures:

{"cmd":<cmd> ,"data":<data>, "id":<n>}

and

{<id>:{"<cmd>":"<data>",...}

Do you have a use case you can share so I understand how you're building these MQTT payloads? Is this something that Domoticz does for example?

@MichaelDvP what do you think about adding this back? It'll require some tricky coding to parse each json element and check whether it's a valid command.

[MichaelDvP]

@FredericMa

This command if failing on topic ems-esp/thermostat_cmd :
{"cmd":"nofrosttemp" ,"data":17, "id":1}

"data":17does not work anymore, you have to use "data":"17" (with quotation marks)

@proddy i don't think this is realy needed. For manual testing it's less typing, but for normal operation it's no problem to send two or more commands in a row.

[proddy]

"data":17does not work anymore, you have to use "data":"17" (with quotation marks)

yes, this sucks so I'm working on change that allows the data and id to be a string or a number.

[MichaelDvP]

Q&D this works:

                if (doc["data"] == nullptr) {
                    LOG_ERROR(F("MQTT error: invalid payload cmd format. message=%s"), message);
                    return;
                }
                const char * data = doc["data"];
                char data1[30];
                if (data == nullptr) {
                    if (float f = doc["data"]) {
                         Helpers::render_value(data1, f, 10);
                    }
                } else {
                    strcpy(data1, data);
                }

if (!call_command(mf.device_type_, command, data1, n)) {

[proddy]

that would work too. I used JsonVariant id = doc["id"]; if (id.is<char*>()) { etc...

[FredericMa]

Do you have a use case you can share so I understand how you're building these MQTT payloads? Is this something that Domoticz does for example?

Actually I use it for the heat mode. I'm using Home Assistant but implemented some custom code since circuit is controlling the radiators circuit (on the second floor) but on that circuit I have valves that are controlled via KNX. So the KNX valve controller has a bit that indicates heat demand. This is captured by HA and that enables the heating circuit via ems-esp. I want the circuit 1 water temp to be around 65°C so I set the circuit 1 setpoint to current room temp + 5°C. The thermostat actually uses the current room temperature of circuit 2 (floor heating on ground floor) for circuit 1. By using current room temp + 5°C the heater will always target approx 65°C. The it is up to the valves to control the target temperature in each room.

Like Michael says, it is not necessary to have the ability to chain multiple commands. Two separate commands works fine as well.

[FredericMa]

Something else I noticed:
I can still set the mode to "heat". I guess because of the mapping over here: https://github.com/proddy/EMS-ESP/blob/67877d07c51b5e456e86bc55ec334151e3c827b1/src/devices/thermostat.cpp#L1497-L1512
But I'm unable to use the heattemp command because only nofrosttemp, nighttemp and daytemp are registered.

Also the reported mode_type is heat, eco or nofrost according to this code I guess:
https://github.com/proddy/EMS-ESP/blob/67877d07c51b5e456e86bc55ec334151e3c827b1/src/devices/thermostat.cpp#L627-L636

So if you set nofrosttemp this is for nofrost mode_type that is also reported as mode_type nofrost which is OK.
But the following can cause some confusion since if you set nighttemp and night mode_type, you actually set the temp and mode_type for the mode_type that will be reported as eco.
The same for daytemp and mode_type day which modify the mode_type reported as heat.

So maybe this needs some alignment?

[proddy]

First we need to get the right commands for your Junkers thermostat. Could you look at the code in Thermostat::add_commands() and let me know which commands should be supported?

Then for the mapping of modes, are you saying when you change the temperature for a mode (like daytemp or nighttemp) it should also automatically switch the mode? Or is it that the reported mode is just in correct?

[FredericMa]

First we need to get the right commands for your Junkers thermostat. Could you look at the code in Thermostat::add_commands() and let me know which commands should be supported?

According to the thermostat the supported modes are Automatic, Comfort, Economy, and Frost
So that would map to auto, comfort (or heat), eco and nofrost.

There also seems to be a Vacation mode in the thermostat but I never used that before. The description for that mode is:
You can use this function if you want to set a constant operating mode for several days (e.g. Frost ) without changing the heating program. When the vacation program is active, the heating circuits and domestic hot water systems are operated according to the operating mode set in the vacation program (frost protection is automatically provided).

Then for the mapping of modes, are you saying when you change the temperature for a mode (like daytemp or nighttemp) it should also automatically switch the mode? Or is it that the reported mode is just in correct?

Indeed, I wanted to point out that reported mode is incorrect, or I would rather say inconsistent compared to the sent command.

[MichaelDvP]

If i understand correctly the actual situation is:

• Your thermostat is a Junkers FW?
• The thermostat shows modes: Automatic | Comfort | Economy | Frost
• Ems-esp shows on telnet:
  mode: Manual | Auto | Holiday
  modetype: heat | eco | nofrost
• mqtt publish for mode and modetype is the same as terminal
• mqtt commands for mode is heat | eco | nofrost | auto, also working day | night | nofrost | auto
• Ems-esp shows on terminal and mqtt only daytemp, nighttemp Edit: actually not shown, telegram SetJunkers is not read
• Ems-esp commands on mqtt and call are daytemp, nighttemp, nofrosttemp

You like to have the temperature publish and commands to be heattemp, ecotemp, nofrosttempand add the missing nofrosttemp to the publish, right?

[FredericMa]

Your thermostat is a Junkers FW?

Yes, FW200 to be pricise, so I cannot confirm that this report is valid for other Junkers thermostats.

The thermostat shows modes: Automatic | Comfort | Economy | Frost
Ems-esp shows on telnet:
mode: Manual | Auto | Holiday
modetype: heat | eco | nofrost

Yes, that is correct.

mqtt publish for mode and modetype is the same as terminal

Also correct

mqtt commands for mode is heat | eco | nofrost | auto, also working day | night | nofrost | auto

Yes

Ems-esp shows on terminal and mqtt only daytemp, nighttemp Edit: actually not shown, telegram SetJunkers is not read

Maybe I don't understand the question correctly but the setpoints are shown correctly for all modes. The shown mqtt subscriptions for thermostat are these:
topic: ems-esp/thermostat_cmd, [cmd]: wwmode temp mode nofrosttemp nighttemp daytemp

Ems-esp commands on mqtt and call are daytemp, nighttemp, nofrosttemp
Yes
You like to have the temperature publish and commands to be heattemp, ecotemp, nofrosttemp and add the missing nofrosttemp to the publish, right?

nofrosttemp is actually working since @proddy added the command. It is heattemp and ecotemp that are not working but daytempand ´nighttemp` do work. See line from terminal above.

[MichaelDvP]

Maybe I don't understand the question correctly but the setpoints are shown correctly for all modes. The shown mqtt subscriptions for thermostat are these:
topic: ems-esp/thermostat_cmd, [cmd]: wwmode temp mode nofrosttemp nighttemp daytemp

I mean the publish, not the subscription.

• if you type show in terminal you get a lis of values, but i think the setpoints are missing for Junkers.
• in mqtt publish thermostat_data the setpoints are also missing

[FredericMa]

Yes, indeed, I don't see those. So never missed them until now that I know 😄
This is the output of the show command:

Boiler: HT3 (DeviceID:0x08, ProductID:95, Version:23.04)
  Hot tap water: off
  Central heating: off
  Warm Water activated: on
  Warm Water charging type: 3-way valve
  Warm Water circulation pump available: off
  Warm Water circulation pump freq: 1x3min
  Warm Water circulation active: off
  Warm Water comfort setting: Hot
  Warm Water disinfection temperature: 75°C
  Warm Water selected temperature: 50°C
  Warm Water set temperature: 50°C
  Warm Water current temperature (intern): 64.7°C
  Warm Water current temperature (extern): 64.7°C
  Warm Water current tap water flow: 0.0l/min
  Warm Water # starts: 802
  Warm Water active time: 12 days 11 hours 32 minutes
  Warm Water charging: off
  Warm Water disinfecting: off
  Selected flow temperature: 0°C
  Current flow temperature: 29.5°C
  Gas: off
  Boiler pump: off
  Fan: off
  Ignition: off
  Burner selected max power: 100%
  Burner current power: 0%
  System service code:  (0)
  Heating temperature setting on the boiler: 66°C
  Boiler circuit pump modulation max power: 100%
  Boiler circuit pump modulation min power: 10%
  Boiler circuit pump delay time: 3min
  Boiler temp hysteresis on: -5°C
  Boiler temp hysteresis off: 0°C
  Boiler burner min period: 3min
  Boiler burner min power: 0%
  Boiler burner max power: 100%
  Outside temperature: 25.7°C
  Pump modulation: 0%
  Burner # starts: 104927
  Total burner operating time: 364 days 11 hours 56 minutes
  Total heat operating time: 352 days 0 hours 23 minutes

Thermostat: Junkers FW200 (DeviceID:0x10 ProductID:106, Version:12.14)
  Clock: 10:40:13 17/08/2020
  Heating Circuit 1:
    Current room temperature: 23.7°C
    Setpoint room temperature: 15.0°C
    Mode: manual
    Mode Type: nofrost
  Heating Circuit 2:
    Current room temperature: 23.7°C
    Setpoint room temperature: 16.0°C
    Mode: manual
    Mode Type: nofrost

Mixing Module: Junkers IPM (DeviceID:0x21 ProductID:102, Version:20.08)
  Heating Circuit: 2
    Current flow temperature: 28.2°C
    Setpoint flow temperature: 0°C
    Current pump modulation: 0%
    Current valve status: 0

Mixing Module: Junkers IPM (DeviceID:0x20 ProductID:102, Version:20.08)
  Heating Circuit: 1
    Current pump modulation: 0%

Solar Module: Junkers ISM1 (DeviceID:0x30 ProductID:101, Version:23.04)
  Collector temperature (TS1): 38.2°C
  Bottom temperature (TS2): 62.5°C
  Solar Pump (PS1) active: off
  Pump working time: 202 days 9 hours 0 minutes
  Tank Heated: off
  Collector shutdown: off
  Energy last hour: 0.0Wh

And this is the output of the thermostat_data publish:
{"hc1":{"seltemp":15,"currtemp":23.7,"mode":"manual","modetype":"nofrost"},"hc2":{"seltemp":16,"currtemp":23.7,"mode":"manual","modetype":"nofrost"}}

[proddy]

I removed the Rx queue on the latest v2_cmd. It never got more than one record in so no point using a buffer and expensive list. Makes the code tidier and I would expect slightly quicker. Apart from that no other side effects as far as I can tell.

I'm still getting the odd Rx corrupted telegrams. In my tests

• with tx_mode 1 after 12hrs I got 39 Rx failures
• with tx_mode 0 and the Tx line completely disconnected (so no Tx), after 7hrs I got 3 Rx failures

conclusion? no idea. I think perhaps we've built a quantum computer where the observer is changing the outcome. I'm now running another test at 80Hz to see if that helps.

-- edit --

• running at 80Hz made no difference
• back on tx_mode 1, after 9hrs 16 Rx failures again
• rolled back to earlier UART, made no difference
• conclusion: it's not the UART or roomcontrol changes, but something else in the cmd infrastructure changes from this issue. Now I have to reverse engineer bit by bit to find out why.

Aug 18 00:58:55 ems-esp - 000+04:57:02.114 E 11: [telegram] Rx: 17 0B A8 00 01 00 FF FF 8F (CRC 8F != 75)
Aug 18 01:21:55 ems-esp - 000+05:20:02.241 E 12: [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 03 97 (CRC 97 != DD)
Aug 18 01:41:14 ems-esp - 000+05:39:21.362 E 13: [telegram] Rx: 17 08 1F 89 (CRC 89 != 53)
Aug 18 01:42:55 ems-esp - 000+05:41:02.366 E 14: [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 FF 89 (CRC 89 != DF)
Aug 18 02:50:55 ems-esp - 000+06:49:02.351 E 15: [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 FF 89 (CRC 89 != DF)
Aug 18 03:12:56 ems-esp - 000+07:11:02.754 E 16: [telegram] Rx: 17 0B A8 00 01 00 08 00 2A 00 00 00 00 00 00 00 00 01 06 00 00 80 00 00 80 00 80 00 80 00 00 B6 (CRC B6 != ED)
Aug 18 03:38:55 ems-esp - 000+07:37:02.253 E 17: [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 FF 89 (CRC 89 != DF)
Aug 18 03:54:37 ems-esp - 000+07:52:43.987 E 18: [telegram] Rx: FE 08 00 1C 00 00 2F 24 AC 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 38 (CRC 38 != 05)
Aug 18 04:11:55 ems-esp - 000+08:10:02.375 E 19: [telegram] Rx: 17 0B A8 00 01 00 FF F6 01 07 89 (CRC 89 != EC)
Aug 18 05:23:55 ems-esp - 000+09:22:02.050 E 20: [telegram] Rx: 17 0B 91 00 80 CA 89 (CRC 89 != 5D)

they look like valid telegrams, just cut short because the BREAK is falsely detected.

[FredericMa]

I tried the latest code. It works like a charm!
It shows the temps in the console and mqtt data.
Thanks @MichaelDvP !

[MichaelDvP]

@proddy i've pushed a uart branch based on latest v2_cmd to make sure that the timer is not triggert by something else in the framework and sends a unwanted break.

[proddy]

@MichaelDvP just tried that now. Got the first Rx fail after 3 minutes (which is usual btw) so that change didn't seem to make a difference. I'll need to go back to a v2_cmd build where everything was working fine, and then do a diff.

[MichaelDvP]

My last rx errors were 15.8. as reported in #398, before there were also a few errors (not logged). I made a platformio update and platformio upgrade and i think that have fixed the errors, since there was no other change in the software.
@FredericMa do you have rx/tx errors (i suppose tx_mode 3)?

[FredericMa]

@FredericMa do you have rx/tx errors (i suppose tx_mode 3)?

This is the status of ems:

EMS Bus is connected, but Tx is not stable.

EMS Bus info:
  Tx mode: 3
  Bus protocol: HT3
  #telegrams received: 8458
  #read requests sent: 9256
  #write requests sent: 7
  #incomplete telegrams: 0 (0%)
  #tx fails (after 3 retries): 1164

Tx Queue is empty

I guess I need to look at incomplete telegrams for failed reads? So those are 0 at 19h uptime. The failed tx are coming from Norberts hardware on RPi which is also still connected to the bus.

[MichaelDvP]

I guess I need to look at incomplete telegrams for failed reads? So those are 0 at 19h uptime.

Yes, that look good.

The failed tx are coming from Norberts hardware on RPi which is also still connected to the bus.

I remember, the rpi uses modem device-id, but do not reply to version request from ems-esp. So you get a tx-fail every minute.

[proddy]

I remember, the rpi uses modem device-id, but do not reply to version request from ems-esp. So you get a tx-fail every minute.
we can make a simple change to ignore post-Tx checks when sending to one of the known ems-bus-ids (0xA, 0xB, 0xD, 0xF, 0x12)

still its nice to see that you have no incomplete Rx's.

[FredericMa]

Eventually I'll disconnect the RPi from the bus after I finish moving some remaining automations from OpenHAB to Home Assistant. So implementing a fix is not really necessary from my side.

[proddy]

My last rx errors were 15.8. as reported in #398, before there were also a few errors (not logged). I made a platformio update and platformio upgrade and i think that have fixed the errors, since there was no other change in the software.

I tested 8 different builds going back as early as b9 on the old v2 branch from July 28 and each test gave Rx incompletes every 1 hr or so. Then I tried rolling back the espressif core to 2.6.1 and 2.6.0 but that didn't help either. I can't figure out why we're seeing this now. I'll open a new Issue for this.

In any case it has nothing to do with the v2_cmd changes so I'll merge this back into v2.

[proddy]

merged into the v2 branch.