Ems-esp: (v2) ESP8266 & ESP32 UART optimizations

[proddy]

We can merge later. My system is also EMS 1.0 and all modes working. Maybe it is uncritical because all components are on the boiler and the ems-bus lines are less than 1m (no roomcontroler, only weather controlled), No emc problems, no reflections, etc.
Can you check with less delay, set emsTxWait = 5 * EMSUART_BIT_TIME * 11; for 11 bittimes.
Also you can check with break ended by timer in the isr uncommend the last lines but set the timer to timer1_write(5 * EMSUART_TX_BRK_WAIT.

_Originally posted by @MichaelDvP in https://github.com/proddy/EMS-ESP/issues/397#issuecomment-643620901_

[proddy]

@MichaelDvP With the latest version 2.0.0a18 with your UART changes I can only get tx_mode 1 working. tx_mode 2 gives me:

[telegram] Sending read Tx [#12], telegram: 0B 97 02 00 20 44
[telegram] [DEBUG] New Tx [#25] telegram, length 1
[emsesp] [DEBUG] Last Tx operation failed. Retry #1. Sent: 0B 97 02 00 20, received: 09
[telegram] Sending read Tx [#25], telegram: 0B 97 02 00 20 44
[telegram] [DEBUG] New Tx [#26] telegram, length 1
[emsesp] [DEBUG] Last Tx operation failed. Retry #2. Sent: 0B 97 02 00 20, received: 09
[telegram] Sending read Tx [#26], telegram: 0B 97 02 00 20 44
[emsesp] [DEBUG] Last Tx operation failed. Retry #0. Sent: 0B 97 02 00 20, received: 09
[emsesp] Last Tx operation failed after 3 retries. Ignoring request.

EMS Bus info:
  Bus protocol: Buderus
  #telegrams received: 41
  #read requests sent: 0
  #write requests sent: 0
  #incomplete telegrams: 0 (0%)
  #tx fails (after 3 retries): 9

I noticed there is also no echo. With tx_mode 1 I would get the echo after the send, like

[telegram] Sending read Tx [#64], telegram: 0B 97 06 00 20 54
[emsesp] [DEBUG] Echo: 0B 97 06 00 20 54
[emsesp] Last Tx read successful

[proddy]

Michael, are you also using the same platformio.ini file with the ESP8266 running at 160Mhz instead of 80Mhz (board_build.f_cpu = 160000000L). Was thinking this could perhaps interfere with the timing?

[MichaelDvP]

Yes i'm running 160 MHz, but i tested the timer before with dummy counts,
Have you tried mode 2 or 3? 3 gives no echo, since it clears the fifos before sending break. The timer controlled mode 2 should echo.
With mode 3 i see a strange thing, the device answers, but ems-esp says no response. But it's only in mode 3.
ems:/ems# 000+00:10:01.547 D 653: [telegram] Sending read Tx [#101], telegram: 0B 88 33 00 20 78
ems:/ems# 000+00:10:01.670 D 654: [telegram] New Rx [#166] telegram, length 11
ems:/ems# 000+00:10:01.670 T 655: [emsesp] Boiler(0x08) -> Me(0x0B), UBAParameterWW(0x33), data: 08 FF 30 FB FF 28 FF 07 46 00 00
ems:/ems# 000+00:10:01.670 D 656: [emsdevice] Processing UBAParameterWW...
ems:/ems# 000+00:10:02.087 D 657: [telegram] Sending read Tx [#110], telegram: 0B 88 33 00 20 78
ems:/ems# 000+00:10:02.170 D 658: [telegram] New Rx [#167] telegram, length 11
ems:/ems# 000+00:10:02.170 T 659: [emsesp] Boiler(0x08) -> Me(0x0B), UBAParameterWW(0x33), data: 08 FF 30 FB FF 28 FF 07 46 00 00
ems:/ems# 000+00:10:02.170 D 660: [emsdevice] Processing UBAParameterWW...
ems:/ems# 000+00:10:02.182 D 661: [mqtt] Publishing topic stat/ems/boiler_data (#102, attempt #1, pid 1)
ems:/ems# 000+00:10:02.566 D 662: [telegram] Sending read Tx [#111], telegram: 0B 88 33 00 20 78
ems:/ems# 000+00:10:02.575 E 663: [emsesp] Last Tx operation failed after 3 retries. Ignoring request.
ems:/ems# 000+00:10:02.667 D 664: [telegram] New Rx [#168] telegram, length 11
ems:/ems# 000+00:10:02.667 T 665: [emsesp] Boiler(0x08) -> Me(0x0B), UBAParameterWW(0x33), data: 08 FF 30 FB FF 28 FF 07 46 00 00

[proddy]

only tx_mode 1 is working. 2,3 4 give failures. not even a single Tx makes it through. I think I just need to bring out the scope and see what is actually happening on the line.

For tx_mode 3 this was a special modification I worked with @philrich on last year. See https://github.com/proddy/EMS-ESP/issues/103#issuecomment-493368592. The timings are slightly different for Junkers/HT3.

[MichaelDvP]

In this comment he wrote:

For me this patch works and i get close to none Corrupted Telegrams when i set EMS_TX_WAIT_GAP to 7 Bits (728us). (3, 4, 5, 6 Bits also works, when setting this to 10 or more Bits i get many Corrupted Telegrams)

So we need a gap of 3-9 bittimes for HT3, 10 for EMS+, and a low value for EMS1.0?
I do this to the uart, revert the mode 2, it's never good to change a working mode. Now there are:

• mode 1..3 as before, exept i do not clear buffers on break to get the echo in rx.
• mode 4 as before, only hardwarecontroled
• mode 5, same as mode 4, but 1,5 stopbits, I see in #103 that the time between master-poll and master-break is 1,5 bittimes.
• mode 11..30: timer controlled modes with number of bittimes, ems+ should work with mode 20, ht3 with mode 17
• mode 6..10 same as modes 12, 14, 16, 18, 20, but timer controlled clearing of break-flag.

• for esp32 i have also implemented timermodes, 4, 6..30 works the same way, 1..3 is also timermode and 5 is the same as 4. I have to look how to change the stopbits.

All modes can be changed without reboot. You know, i don't like the modes 1-3, where the processor making millions of nop-loops to wait for tx done, There are other usefull things to do.

I've also merged your latest commit. Should i make a pr, or wait for feedback from hans?

In my system all modes work. Mode 1 has very few rx-crc-errors only if roomcontrol is activated and a roomcontrol poll-ack is right before a incoming telegram (it's always the UBAMonitorFast to all) . Then i get a 19 as additional first value in this telegram.

[proddy]

I like the approach. Plenty of modes to test with, and we can ask some of the Junkers users to also test. I'd say push the PR with the changes you made for Hans as you also fixed a lot of silly mistakes I made so thanks again for that. I agree the original rx/tx code is not efficient and blocking other important cycles and your approach is cleaner and makes use of the hardware interrupts.

[MichaelDvP]

As long as you never need a timer, 8266 have only 2 timers and one is used by wifi, the other here. Esp32 has more resources.

[proddy]

did a quick test, still only tx_mode 1 works on my system. I'll try all the other combinations later tonight.

[MichaelDvP]

something strange, seems the timer is now used, blocked, or reconfigured by logger for debug and trace. In both log modes tx fails in all timer-controlled modes, with log info the timer works, also if watch is on. As soon i switch log to info or lower tx works again and show emsbus counts no more errors, also i can see the reply to me in watch on/raw.

This log shows only the relevant lines.
20200614_163200.log

I checked the logs from before merging your latest changes, all modes works in debug/trace modes.

[proddy]

Hmm, can’t think of what that could be. And Log Level INFO with ‘watch on’ works you say? I’ll check the code when I’m back home.

[proddy]

did some more testing with ranges 4, 5, 6-10, 11-30 and only tx_mode 1 and tx_mode 5 work in my environment. See screenshots:

I can't see how the debug would affect the timings. It's best not to include any logger objects in the uart code is it may be blocking.

[MichaelDvP]

Oh, good, mode 5 working.

My issues with mode 1 come from the poll. Poll uses tx_brk also for mode 1, but here is no delay for mode 1. Found also a issue with HT3.

I deleted logging from emsuart, but that doesn't help. It is the LOG_DEBUG(F("Sending.. near the transmit() call, if i delete these loggings, all works. Seems the logger uses the timer for output. ESP32 has more timers and no conflict.
I'll push + pr the changes after testing. I've also adapted esp32 uart completly, but not tested yet.

[proddy]

tx_mode 5 uses the same code as tx_mode 1 (the legacy stuff) but with 1.5 bitstops?

I can't see why the logs would affect the timings, the LOG_DEBUG just appends the message to a message queue. I think these messages are important (for debugging!) so does it work if you move them after the transmit() ?

[MichaelDvP]

No, mode 5 uses the same code as mode 4, but with 1,5 stopbits: Pushing all in the fifo and the hardware sends it out and appends the break. The transmit-function doesn't have to wait and returns instantly. I think it is the best mode for ems1.0.
Putting the sending.. after transmit-call doesn't help (done that), because the timer-mode sending in the background aren't finished. There are other debug-messages if tx fails and on receive Last Tx read successful which doesn't conflict, since the tx is finished then. So there is a debug feedback in any way.

[proddy]

ah, I see what I did wrong now. I modified some of the esp8266 uart code, let me revert and test again. I don't expect tx_mode 5 will work sadly.

[proddy]

ok, tx_mode 5 doesn't work. Only tx_mode 1 on my setup.

[MichaelDvP]

I have tested a bit more and think i understannd better. I tried to measure the time from start sending to complete echo received, but with confusion results. I realize, that you havve modified the get_uptime() and the time is only updated in set_uptime once per loop-cycle, so all in one loop gives the same time (tx-modes 1-3 executes in 0 ms ;-)). Is there a special reason for this modification?
I used ::millis() instead and can see the time.
The esp uart seems to have a lag of 1 bytelength when receiving a byte, maybe there is a shiftregister for rx and after stopbit it is shifted to fifo in the next cycle, tx_mode 1 needs 16 ms for 6 bytes+break, same as tx_mode 2 or tx_mode 21. tx_mode 4 executes in 9 ms.
so it's 1dummy+6+break for mode 4, 6dummy+6+break for mode 1. Your ems1.0 seems to be more ems+ than mine. Try with tx_mode 21 or 22 if it works for you.

Therefore i changed the modes once again, leaving 1..4, but set from 5 on timer as number of delay between bytes: mode 5 is 1bytetime+5bittimes until next byte, If there is no delay needed, mode 4 will do, HT3 will work with 5,6,7, ems+ 20, 21, 22. My system resposes to all modes, also 50 works.
In mode 1 i added back the timeout, if there is no loopback from the bus, it will hang forever.

With esp32 i can't get the modes with delayMicroseconds working, but 4..50 without any collision with logger.

[proddy]

The reason for implementing get_uptime() is because I thought millis() was quite expensive and used many times in the code. I decided to control the timestamp once in the loop() cycle.

https://github.com/esp8266/Arduino/blob/1bfb29395f71da2caa5d14cbc6bdf8cf9c092d7a/cores/esp8266/core_esp8266_wiring.cpp#L166

I'll test your changes, thanks

[proddy]

quickly tested. I couldn't get any of the modes to work. tx_mode 1 also now shows errors so I need to check what was changed

```
[telegram] Sending read Tx [#29], telegram: 0B 97 06 00 20 54
[emsesp] [DEBUG] Echo after 11 ms: 0B 99 F0 C0
[telegram] [DEBUG] New Tx [#62] telegram, length 1
[emsesp] [DEBUG] Last Tx operation failed. Retry #1. Sent: 0B 97 06 00 20, received: 89
[telegram] [DEBUG] New Rx [#80] telegram, message length 25
[emsdevice] Processing UBAMonitorFast...
[telegram] [DEBUG] New Rx [#81] telegram, message length 21
[emsdevice] Processing MC10Status...
[telegram] [DEBUG] New Rx [#82] telegram, message length 19
[emsdevice] Processing UBAMonitorWW...
[telegram] [DEBUG] New Rx [#83] telegram, message length 13
[mqtt] Publishing topic homeassistant/climate/ems-esp/state (#45, attempt #1, pid 1)
[telegram] Sending read Tx [#62], telegram: 0B 97 06 00 20 54
[emsesp] [DEBUG] Echo after 33 ms: 0B 9F F0 C8 D4 FC
[telegram] [DEBUG] New Tx [#63] telegram, length 1
[emsesp] [DEBUG] Last Tx operation failed. Retry #2. Sent: 0B 97 06 00 20, received: 89
[telegram] Sending read Tx [#63], telegram: 0B 97 06 00 20 54
[emsesp] [DEBUG] Echo after 12 ms: 0B 99 F0 C0
[emsesp] [DEBUG] Last Tx operation failed. Retry #0. Sent: 0B 97 06 00 20, received: 89
[emsesp] Last Tx operation failed after 3 retries. Ignoring request.

EMS Bus info:
Tx mode: 1
Bus protocol: Buderus
#telegrams received: 257
#read requests sent: 39
#write requests sent: 0
#corrupted telegrams: 0 (0%)
#tx fails (after 3 retries): 23
```

Also I think it's time we have other people test to rule out something strange in my environment.

[MichaelDvP]

Change in tx mode 1 since a19 is the old timeout (22 bittimes) back and change the poll-ack to the same logic, before poll uses tx_brk, but tx_brk has not wait defined for mode 1, causing some crc errors for me.
Strange that the echo is corrupted in that way, thats not a timing-issue, the bits aren't shifted, they are puzzled. Bad contacts? Cold solderings?

[MichaelDvP]

Here is a log with nearly all modes on my system, you can see the different timings. The roomcontroller simulation is on, the 0xAF from 0x19 is from that function, so ems-esp answers all polls to 0x0B and 0x19.
20200616_165225_EMS.log
I'm surprised that the master does not interrupt the very long modes.

[proddy]

nice can clean. You know, perhaps its my circuit. I have a few older and newer ones which bbqkees gives me to try. I'll go back to an earlier board and try the jack interface with a shorter cable and see if it makes any difference.

[proddy]

@bbqkees we have a similar setup. When you find time could you grab the latest v2 and test tx_mode 1, 4 and 5? Instructions are in https://github.com/proddy/EMS-ESP/tree/v2#uploading-the-firmware

[bbqkees]

I'm already running v2 with tx_mode 1 on a Premium II Gateway with a Lolin ESP8266.

This is from about 12 hours or so:
`EMS Bus info:
Tx mode: 1
Bus protocol: Buderus

telegrams received: 24713

read requests sent: 4522

write requests sent: 0

corrupted telegrams: 0 (0%)

tx fails (after 3 retries): 0`

Will do some logging when I'm at home.

[proddy]

I'm already running v2 with tx_mode 1 on a Premium II Gateway with a Lolin ESP8266.

is this using the latest a21 version with Michael's new tx modes?

[bbqkees]

No you're too fast for me :-).
Its on a19 from yesterday morning. Will update it tonight.

[bbqkees]

on a21 all modes give TX errors.
log-mode-1-4-5-bbqkees-17062020.log

[MichaelDvP]

Oh, i'll look what is the change in mode 1, i thought nothing was changed, can please also try mode 10, or 12 with longer timer delays?

[bbqkees]

I'm not seeing any real difference between modes.

EMS Bus info:
Tx mode: 12
Bus protocol: Buderus
#telegrams received: 202
#read requests sent: 4
#write requests sent: 0
#corrupted telegrams: 0 (0%)
#tx fails (after 3 retries): 38

Rx Queue is empty

Tx Queue (7 telegrams):
[104] READ Me(0x0B) -> Thermostat(0x10), RC30Set(0xA7), data: 20 (offset 0)
[115] READ Me(0x0B) -> Boiler(0x08), UBAMonitorSlow(0x19), data: 20 (offset 0)
[116] READ Me(0x0B) -> Boiler(0x08), UBAParameterWW(0x33), data: 20 (offset 0)
[117] READ Me(0x0B) -> Boiler(0x08), UBAParameters(0x16), data: 20 (offset 0)
[118] READ Me(0x0B) -> Thermostat(0x10), RCTime(0x06), data: 20 (offset 0)
[119] READ Me(0x0B) -> Thermostat(0x10), RC30Monitor(0x41), data: 20 (offset 0)
[120] READ Me(0x0B) -> Thermostat(0x10), RC30Set(0xA7), data: 20 (offset 0)

EMS Bus info:
Tx mode: 10
Bus protocol: Buderus
#telegrams received: 384
#read requests sent: 16
#write requests sent: 0
#corrupted telegrams: 0 (0%)
#tx fails (after 3 retries): 58

Rx Queue is empty

Tx Queue (17 telegrams):
[240] RAW Me(0x0B) -> Boiler(0x08), UBAParameterWW(0x33), data: 20 (offset 0)
[194] READ Me(0x0B) -> Boiler(0x08), UBAParameters(0x16), data: 20 (offset 0)
[195] READ Me(0x0B) -> Thermostat(0x10), RCTime(0x06), data: 20 (offset 0)
[196] READ Me(0x0B) -> Thermostat(0x10), RC30Monitor(0x41), data: 20 (offset 0)
[197] READ Me(0x0B) -> Thermostat(0x10), RC30Set(0xA7), data: 20 (offset 0)
[208] READ Me(0x0B) -> Boiler(0x08), UBAMonitorSlow(0x19), data: 20 (offset 0)
[209] READ Me(0x0B) -> Boiler(0x08), UBAParameterWW(0x33), data: 20 (offset 0)
[210] READ Me(0x0B) -> Boiler(0x08), UBAParameters(0x16), data: 20 (offset 0)
[211] READ Me(0x0B) -> Thermostat(0x10), RCTime(0x06), data: 20 (offset 0)
[212] READ Me(0x0B) -> Thermostat(0x10), RC30Monitor(0x41), data: 20 (offset 0)
[213] READ Me(0x0B) -> Thermostat(0x10), RC30Set(0xA7), data: 20 (offset 0)
[234] READ Me(0x0B) -> Boiler(0x08), UBAMonitorSlow(0x19), data: 20 (offset 0)
[235] READ Me(0x0B) -> Boiler(0x08), UBAParameterWW(0x33), data: 20 (offset 0)
[236] READ Me(0x0B) -> Boiler(0x08), UBAParameters(0x16), data: 20 (offset 0)
[237] READ Me(0x0B) -> Thermostat(0x10), RCTime(0x06), data: 20 (offset 0)
[238] READ Me(0x0B) -> Thermostat(0x10), RC30Monitor(0x41), data: 20 (offset 0)
[239] READ Me(0x0B) -> Thermostat(0x10), RC30Set(0xA7), data: 20 (offset 0)

[proddy]

@MichaelDvP could it be the circuit? are you using one of bbqkees' gateway boards or a home grown circuit board?

[MichaelDvP]

I'm using BBQKees Premium 2 Gateway with the original Wemos (purcased december 2019), but it's not buspowered, im using the service jack. I have also a second noname wemos with encapsulated modul, and a wemos D1 mini 32. All working with the gateway. System is Buderus GB125 with BC10, MC10, RC35 and MM10, all pure EMS1.0 from 2011.

The difference from a19 to a21 in mode 1 is

• the timeout, with a value as in earlier versions
  The timeout is now uint8_t, maybe the preprocessor calculated wrong: (22 * EMSUART_TX_BIT_TIME / EMSUART_TX_BUSY_WAIT) should be 176, but if (22 * EMSUART_TX_BIT_TIME) is skipt to 8 bit before the division we alway get a timeout.
  @proddy I can't test, since all modes working on my system, can you test with (22*8) for EMSUART_TX_TIMEOUT.
• the poll handling:
  I use the same logic for poll, in a19 it was a simple wait:
  USF(EMSUART_UART) = data;
delayMicroseconds(EMSUART_TX_BRK_WAIT);
tx_brk(); // send <BRK>
  but i'm astonished that a wait doesn't work for sending in your system, but should work for a poll.

[bbqkees]

@MichaelDvP Can you try bus powered?
Just put the buck on the upper row then its bus powered even when connected to the service jack.

I and Proddy have the exact same boiler.

[MichaelDvP]

I was a bit to fast, Seems the big capacitor has a lot of load also after disconnecting and burned something on the board or wemos when changing the buck.
Have to check first.

[bbqkees]

Could also be that you need to reinsert the jack cable. I have that sometimes when testing new Gateways when you change the buck position the Wemos does not boot. Only after unplugging it from the boiler and plugging it back it in boots.
If you fried the Gateway let me know I'll send you a new one.

[proddy]

I just tested with the jack and it behaves the same as bus-powered, so rules that out.

Changing EMSUART_TX_TIMEOUT to (32*8) works. Anything lower than 256 gives the error. I don't think you need the timeoutcnt in there at all?

[proddy]

I changed EMSUART_TX_TIMEOUT so v2 works for us folks still using the legacy tx_mode 1, while we work out why tx_mode 4 and higher with hardware timings don't work on my setup

[MichaelDvP]

@bbqkees Thank you for the offer, but it seems only the 3.3V regulator on the wemos died and is now shortend in/out, the processor is now on 5V heating up very fast. With the other Wemos the gateway works. I've ordered a new wemos-clone to have one in spare. I think i'll wait for the delivery before testing again to switch to buspower.
I've disconnected the jack before changing the buck position, but when plugging the buck to the upper pos. there was a very bright flash from the Wemos-LED due to the charge of the capacitor.

@proddy 256 is a bit special for a uint8_t, but works since it counts --timeoutcnt. Better to change to uint16_t. The timeout is not needed if connected to ems with continous receiving data. But when there is no rx, the sending waits forever (watchdog). Not good if you want to test something (wifi, mqtt) on a standalone wemos. But for this cases also a very large timeout will give the chance to telnet and change the tx_mode.
If you really need this time for sending, then timer-mode tx_mode 22 will give the same timing without waiting for finish.

[proddy]

@proddy 256 is a bit special for a uint8_t, but works since it counts --timeoutcnt. Better to change to uint16_t.

Yes, I had already changed that.

If you really need this time for sending, then timer-mode tx_mode 22 will give the same timing without waiting for finish.

tx_mode 22 works, which is frigging awesome because it uses the h/w timers and we can finally remove all the hogging legacy code with the delays(). Nice! Why not just make this is default?

For EMS+ and HT3, do you expect differences in the timings too? We can start asking others to run some tests.

[proddy]

@MichaelDvP I'm sure Kees and myself can ship you a few Wemos' and Gateway boards to try out via DHL. I'll cover the costs. If you're interested I'll email you.

[MichaelDvP]

Good that timer mode is working. We should ask people with ems, ems+ and ht3 to check their timings. For instance start with 22, go down until it gives errors, go up till error. Than we have a working range for each system, maybe one value fits all, or make different defaults for the systems with enough distance to critical timings.
If i get problems with myy gateway, i'll come back to your offer, for now one gateway is enough. For the wemos: i think dhl within europe is more expensive than ordering a hand full of wemos in china. I've allready ordered.

[bbqkees]

@MichaelDvP its better to use only original Lolin Wemos D1 Mini's and not those clones.
Most clones have an under-powered voltage regulator. All Gateways have an additional capacitor on the 3V3 rail as a mini buffer but still.
Also some clones like from Do-It.AM have manufacturing issues and behave erratically.

[proddy]

@MichaelDvP tested also with ESP32 and tx_mode 22. Get about a 75% success rate.

[MichaelDvP]

@proddy: Have you tried to increase/decrease mode?
@bbqkees: good to know. I've used some 4-5 clones before, nodemcu and wemos, only one making some problems. I always thought i've killed it by ESD, i grabed it from thresh to see if i can desolder his regulator, but it's a Do-It and LDO is marked 4A2D. A short google shows that this is the underpowered LDO.

[bbqkees]

You can get the original D1 Mini's from Lolin on AliExpress
The underpowered LDO on the clones does not have to be problematic. Its just if your application is more demanding or you are powering external devices from the 3V3 rail of the Wemos then random problems start to arise.

[MichaelDvP]

@bbqkees thanks for the link, next wemos will be ordered there.
I've taken the LDO from the dead Do-It and put it on the Lolin, works. Seems the 8266 survives a few minutes on 5V.

[MichaelDvP]

I have tested a bitmore with the timings and checked with a oszi. I see that the Buderus devices do not have delays between the bytes, but always they start sending 3-4 ms after the poll. Some long messages have delay inbetween, but i can't see a system, i think it depends on internal calculation power of the devices.

I have added modes with delay before sending. tx_mode 5 with a simple fixed delayMicroseconds(3000), modes over 50 have only a starttimer with 100µs per unit, eg. 55 is 0,5 ms delay, 80 has 3 ms.
Modes 5..50 have delay between the bytes, but now also before sending first byte.

For me all modes working, so we need other testers with EMS1.0, EMS+ and HT3. I suggest first test with tx_mode 90 (4ms)

[proddy]

tested a31 (and b1 on v2_web) with the latest uart code on an ESP32. With tx_mode 5, 22 and 90 I still get a few misses, around 10-20% but it has improved a lot.

With tx_mode 22 for example:

[MichaelDvP]

The esp32 timer does not work as described, the timing of all modes >5 is nearly identical with mode4, i.e. the timer fires without waiting. On esp8266 the timer works, better to test these modes with 8266. What about mode5? It's only a delay before sending, but my oszi says that the buderus-ems-bus clients do exactly this when sending. You can try to increase this delay a bit to 5 ms.
I'm waiting for delivery of a analyser to check the timings.

[proddy]

I'm back to testing on the ESP8266 with v2_web and trying out the different tx_modes. What used to work before, with no errors was tx_mode 22. Now I'm getting 50% miss. Same with 1, 4, 5 and 90. Is this expected? I haven't compared your changes in the UART code but something is definitely different. Or it's something in my new code. dunno.

[MichaelDvP]

1 to 4 has no change, 5 was before a test to use the mode-1 logic, but in the rx-interrupt. But this does not work for you, so i changed this mode to a logic as mode-4, but delayMicrosecond(3000) before sending. Timer-modes 6-50 are changed in that way: before the first byte was send immediatly, now it starts with a delay. Modes >50 are new and same as 4/5 but the startdelay is controlled by timer.

[proddy]

I'm still getting tx errors on both ESP32 and ESP8266. Tried multiple different configs (its easier now in the web UI as you can instantly see the results after changing the mode). In v2 and the previous builds tx_mode 22 worked for me on the ESP8266 and I expect many other people too (including bbqkees). Is there an equivalent for this in your latest uart code on v2_web?

[MichaelDvP]

What about the modes 1, 2, 3, they are unchanged and should work like in 1.9.5 and v2a.
Here is the change back for timermode 22 to start with sending. emsuart_esp8266.zip
I don't want to make a make PR now, because i have some trouble with 8266, web isn't reachableny more if connected to ems.
With esp32 web is working, but the dallas sensors sometimes show not all sensors (sometimes 1, sometimes 3, sometimes all 7). This does not happen with 8266.

[proddy]

@MichaelDvP the only mode that worked for me was 22 on an ESP8266, with the previous builds. Now all modes give errors. Did you get the logic analyzer?

[MichaelDvP]

Yes it arrived 2 days ago and i made some measurements yesterday. I think i have misunderstood the bus. I allways thought it is a half-duplex communication like RS485 and we see on rx what we send. But that's wrong. As far as i understand now (@bbqkees: please correct me if i'm wrong), the master is sending with 5V voltage modulation, but the clients sending as 20mA current loop. Only the master is capable to receive the current loop and repeats every byte in voltage modulation, so we can receive it. We have a full-duplex communication and can send and receive the same time. In tx-mode 4 this looks like:

The master starts echoing after he received the frst byte. Since the sending break is ending on receive back the break, it is very long, but not the masters echo.

The boiler as master is sending without any gap and there is no poll before sending:

Clients like thermostat sending with a byte gap, the master repeats every byte and waits for the next.

The gaps are a bit different, RC35 has 1,2ms, MM10 only 1,1ms. I will test the other tx-modes later.

My conclusion so far: Timing is not very critical, thats why all modes working for me. But current-loop can be influenced by the supply voltage (which modulates while master is sending) and therefore it's better to make a gap after each byte and wait hte msters sending. Also bus-powering can be critical, the esp draws 80-180 mA (180 while wifi-sending) and the master should see a 20mA modulation for communicaton on this signal, maybe the blocking sending modes are better in this case. Also in bus-powered config it can be usefull to reduce wifi-tx-power.

For the uart-code that means: mode 5 and 50+ are useless and same as 4, a reply-gap is not needed. Modes 1-3 should work as before, in mode 1 we can set the tx_brk to a fixed length.

[bbqkees]

Yes indeed the bus is full-duplex. The Master uses the voltage domain and the clients (I like 'Minions' more btw) use the current domain. In the past I did measurements with a scope and you could see both sides sending at the same time. The current when sending only lets the voltage signal drop a few hundred millivolts which the clients don't notice.

In the end we should just 'do as the locals do' so we need to behave on the bus close to what the thermostats do.

I did lots of tests with changing the TX resistors to get a specific current. From what I can remember the TX current of the client needs to be slightly higher than its average base current. All the simple thermostats use a TX resistor of 220 to 250 Ohms while the more power hungry Gateway needs 170 Ohms. Because only one client is allowed to send at at any time this power draw is no problem.
The Gateways have a big reservoir capacitor so the very intermittent and changing power draw of the ESP will be smoothed out.

[MichaelDvP]

New measurement, i've used this uart with fixed break-length also in timer modes.
emsuart_esp8266.zip

Mode1 (send after readback):

Mode 2 (EMS+ delay, mostly identical timing to mode 1):

Mode3: here we have overlap tx/rx

Mode 12 (timer sending, mostly identical to mode 1 and 2):

Mode 22 (longer):

Mode 50 (very long):

[proddy]

What about the modes 1, 2, 3, they are unchanged and should work like in 1.9.5 and v2a.
Here is the change back for timermode 22 to start with sending. emsuart_esp8266.zip
I don't want to make a make PR now, because i have some trouble with 8266, web isn't reachableny more if connected to ems.
With esp32 web is working, but the dallas sensors sometimes show not all sensors (sometimes 1, sometimes 3, sometimes all 7). This does not happen with 8266.

@MichaelDvP just tried with the old emsuart_esp8266.zip you sent with tx_mode 22 (which did work before) and only getting errors now. Do you need some help fine tuning the uart code because if it doesn't work for me and Kees it probably wont' work for others too with EMS1.0. As soon as this is working we can move v2_web to dev.

[MichaelDvP]

@proddy, as bbqkees wrote

In the end we should just 'do as the locals do' so we need to behave on the bus close to what the thermostats do.

That's what i try, but mode 1, 2 and 12 give the same timing as my thermostat with different algorithms. I can not see any reason why it is not working with your system, but without any fail on my EMS1.0 system.
Have you tried to set the esp8266 back to 80MHz, than it draws less current and performance is also good (i have tried that). Maybe the tx-resistor is not optimal for the higher clock and current.

[proddy]

@MichaelDvP it works now. The problem was that I was testing with the Wemos powered by USB. I went back to v2 where I know it worked and had the same Tx errors, and then remembered your comment:

Also bus-powering can be critical, the esp draws 80-180 mA (180 while wifi-sending) and the master should see a 20mA modulation for communication on this signal, maybe the blocking sending modes are better in this case. Also in bus-powered config, it can be useful to reduce wifi-tx-power.

If it's bus-powered both tx modes 1 and 22 work fine with 100% success. On both 80Hz and 160Hz clocked EMS8266's.

Now I'll try lowering the Wifi Tx power from 20.5 to 20dbm like I did in version 1.9 and see if that helps.

[MichaelDvP]

Good to know it is working now, for academic reason: can you test to lower tx-mode from 22 stepwise and see at which mode you get errors?

My thoughts were the other way, lowering the current consumption and current peaks help if the esp is powered by the ems-bus. USB should always have enough power, but some cheap chargers does not have stabilized voltage and large ripple can cause such errors (for phone charging the ripple does not matter). Try with another usb-power-device or with a powerbank.

[proddy]

I get to tx_mode 18. Anything lower than that gives errors (after 3 Tx retries). I'm powering the Wemos via my Desktop PC USB.

EMS Bus info:
  Tx mode: 18
  Bus protocol: Buderus
  #telegrams received: 511
  #read requests sent: 32
  #write requests sent: 0
  #corrupted telegrams: 0 (0%)
  #tx fails (after 3 retries): 0

[proddy]

After running the ESP8266 for 12hrs on tx_mode 18, bus-powered I did get a few errors (<2%), so trying 19 now

(Tx) Successfully sent telegrams = 2769
(Tx) Failed = 20

[bbqkees]

I'm trying out 18 as well in bus powered mode.

[proddy]

I'm trying out 18 as well in bus powered mode.

With tx_mode 20 I had 9 fails in 3 hours. Trying 21....

[bbqkees]

For me 18 does not work well in bus powered mode.

Tx success: 2497
Tx errors: 5093

Now testing 19

[proddy]

I'm on 25. Everything lower gave me errors after a few hours.

[joanwa]

my tx_mode testing of combination of bbqkees ems-esp on usb power and HT3 boiler, EMS2 modules MM100, MS200 and Gateway Bosch MB LAN 2:

• working: 2, 3
• not working: 0,1, 4 - 30

[MichaelDvP]

Now i have also tested a lot on esp32 with it's timer and break differences and reached the same timings.
I'll add it to the PR with the esp8266 uart.
@all: please test after it's merged, the actual v2-uarts have much different timing.

Mode 1:

Mode 2:

Mode 3:

Mode 4:

Mode 12:

Mode 22:

Mode 40:

[lmdecroos]

I'm on b7 and only mode 40 works for me.
I have 7 errors in 24 hours
When testing the other modes, I get an error within a few minutes.
using ESP8266 EMS gateway from BBQKees

[proddy]

@lmdecroos thanks for testing. As I recall you have a RC35, RC20, MM10 and a Buderus GB162?

[lmdecroos]

Yes this is correct

On 23 Jul 2020, at 22:20, Proddy [e-mail] wrote:

@lmdecroos https://github.com/lmdecroos thanks for testing. As I recall you have a RC35, RC20, MM10 and a Buderus GB162?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/proddy/EMS-ESP/issues/398#issuecomment-663212400, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDW443RKBYYMCYCLCZIVYTR5CLQDANCNFSM4N47XUTA.

[bbqkees]

I'm now on b7 and bus powered.

Mode 19: 4532 writes, just 5 errors.

[MichaelDvP]

Very different results, all from EMS1.0 devices, from HT3 and EMS + we don't have any test now?

I tested again with the analyser and looked on the normal communication Master-RC35 and Master-MM10, and also ems-esp-RC35, ems-esp-MM10. There is always one large delay of around 100ms in the reply.
RC35: On Master poll the RC35 replys with it's address <10>, than delays for 110 ms and than sends the rest of the telegram with gaps of 1,2ms. On a request the RC35 makes first a delay of 120 ms, and than sends the whole telegram beginning with <10> and gaps of 1.2 ms.
For MM10 it's the same, but the delay on poll is 97ms and the gap 1.1ms, When replying to a request the delay is only 24ms before the telegram starts.
This looks like:
RC35 poll reply: :

RC35-delay:

RC35-telegram(rest):

and the reply to a request from MM10 looks this:

I am now testing this timing, using modes 100.. with 100ms delay after frist byte and rest of timing like the timermodes 5..99.
This is mode 112 telegram <0B>delay<90><37><00><20>, delay is 100 ms and gaps between bytes are 12 bittimes aka 1.2ms.

And this is the code i used.
uart.zip

@proddy: should i make a PR or do you want to test first and merge this file?
You mentioned that mode 1 produces also errors on your system, try to increase the mode 1 timeout to the v1.9 value or something greater 100ms.

On my GB125 all modes working with zero tx-errors, i can't know what causes the errors on other systems.

[proddy]

What we can do is create a special github feature branch just for the UART tests where you can push all your changes directly to it. That way we have a constant base to test against. Kees and myself still need to run similar tests on our systems to compare. I tried yesterday with two different logic analyzers but couldn't get any signals so either I wired it up incorrectly or there's something wrong my windows10 driver. I was using Serial/UART and connecting one probe to the Rx/D7 female connector on the gateway board (without a Wemos inserted obviously).

I've been running the ESP32 with your latest changes for 21hrs now with tx_mode of 5. Have 2170 Tx Sent and 27 Tx failed. Using USB power. With the ESP8266 it was a lot worse.

Back with EMS-ESP 1.3 I calculated the poll time and used this in the Tx timings. Perhaps this could also be an approach?

[proddy]

@MichaelDvP

You mentioned that mode 1 produces also errors on your system, try to increase the mode 1 timeout to the v1.9 value or something greater 100ms.

I tried that a while back, making the timeout really large (not 32*8) and has some Tx issues. I'll try again.

[MichaelDvP]

What we can do is create a special github feature branch just for the UART tests where you can push all your changes directly to it.

Than we have to find people to test this branch,

logic analyzers but couldn't get any signals
(without a Wemos inserted obviously).

I think the gateway needs the 3.3V from the Wemos to work.

[proddy]

What we can do is create a special github feature branch just for the UART tests where you can push all your changes directly to it.

Than we have to find people to test this branch

done. Push direct to the v2_uart branch

[bbqkees]

I think the gateway needs the 3.3V from the Wemos to work.

Indeed the board itself needs to be powered as well. You can use 3V3 from f.i. a seperate Wemos.

[proddy]

@MichaelDvP I'm running some tests now, first with tx_mode 0 so I can see how my EMS1.0 Nefit Trendline communicates to the RC20.

When I try tx_mode 22, which I know it works it shows some very interesting observations. Like the poll from EMS-ESP is not working, it just sends the frame-break. I was thinking since you know what to explicitly look for, why don't I send you the capture dump? I'm using the Saleae software which you can download free from https://ideas.saleae.com/f/changelog/233/). I have 200 EUR professional logic analyzer and some fancy software to go with it, but I prefer using this 3 EUR aliexpress logic analyzer.

[proddy]

@MichaelDvP using latest code you posted, ESP8266, bus-powered, tx_mode 1, no EMS devices attached, just the master boiler.

25% Tx failure:

poll and poll repsonse looks correct:

send Tx (Mode1 with send after readback). I did a refresh command from the console. Notice the 0xFF at the end which may be because the brk is too long?

[MichaelDvP]

Thanks, that's very interesting. The poll looks the same as for me, the master echo is immediately after the end of sending a byte.
For the sending there is a delay after sending out a byte and the master echo of around 1ms, confirming that timer-modes <22 will not work properly. But for the break the echo is immediately.

Notice the 0xFF at the end which may be because the brk is too long?

Yes, that's causing tx-errors. Normally a uart should trigger the startbit by falling edge, not level. But in this case we should reduce the break to 10 bit, on my system this works too. I've pushed the uart for test, based on v2b7. For now only tested for esp8266.

[proddy]

testing v2_uart on ESP8266, bus powered and tx_mode 1. No errors so far.

[bbqkees]

I'm still on V2 b7 with ESP8266, bus powered.
Mode 19 is quite good. 5392 success, only 32 errors.

On thing I did notice is that the display of the boiler turns on sometimes and the boiler restarts.
But the ESP never reboots. So it seems it locks up the bus occasionally.

[proddy]

I have 0 tx errors (5123 tx sends) on v2.0.0b8 with tx_mode 1. 14hrs with no outages or hic-ups. Web sometimes is a bit slow but it always shows and doesn't crash. b8 is using Michael's adjusted break timing as it was in v1.9.

The boiler restart is scary. Perhaps we can try and detect this on the bus and report it back in the UI.

[joanwa]

for my EMS2 system, the least amount of errors with b8 are now apparently with tx_mode 22 (tried 0-5, 18, 22). On b7, tx_mode 2 & 3 didn't produce any errors but on b8 now quite regular.

[proddy]

@MichaelDvP as mentioned above. I think for tx_mode 1 we stick to the new EMSUART_TX_WAIT_BRK of 10 bits as in 2.0.0.b8 now and for tx_mode 2 go back to the previous 11 bits. Is that worth trying? I can't see any other changes between b7 and b8 that would give Johannes errors on his EMS2 system.

@joanwa are you comfortable changing a file, building and upload the firmware yourself?

[proddy]

On thing I did notice is that the display of the boiler turns on sometimes and the boiler restarts.
But the ESP never reboots. So it seems it locks up the bus occasionally.

@bbqkees when the boiler starts you'll see a UBAMaintenanceData message appear in the console (telegram type 0x15). You could watch for this telegram in a console so see how often it happens (watch raw 15) or use syslog and check the logs the next day.

[joanwa]

@MichaelDvP as mentioned above. I think for tx_mode 1 we stick to the new EMSUART_TX_WAIT_BRK of 10 bits as in 2.0.0.b8 now and for tx_mode 2 go back to the previous 11 bits. Is that worth trying? I can't see any other changes between b7 and b8 that would give Johannes errors on his EMS2 system.

@joanwa are you comfortable changing a file, building and upload the firmware yourself?

yes, already forked and committed some cosmetics in my fork. Just seeing that you build a newer b8 binary. Shall I gives this a go? Or what are you interested in that I can build myself?

[proddy]

Or what are you interested in that I can build myself?

change EMSUART_TX_WAIT_BRK back to using 11 bits in emsuart_esp8266.h

#define EMSUART_TX_WAIT_BRK (EMSUART_TX_BIT_TIME * 11)

and test with tx_mode 2

[MichaelDvP]

@MichaelDvP as mentioned above. I think for tx_mode 1 we stick to the new EMSUART_TX_WAIT_BRK of 10 bits as in 2.0.0.b8 now and for tx_mode 2 go back to the previous 11 bits. Is that worth trying?

Yes, i updated the uart branch. For we know:

• mode 1 works better with break less than 11 bit
• Junkers needs more thn 10 bit break
• EMS+ unknown, but from v1.9 11 bit should work.

For the timer modes i set the break to a bit more than 10 bit, maybe this works on both worlds.
I've also tested esp32 and esp8266 now with analyzer and checked the timing.

[joanwa]

Or what are you interested in that I can build myself?

change EMSUART_TX_WAIT_BRK back to using 11 bits in emsuart_esp8266.h

#define EMSUART_TX_WAIT_BRK (EMSUART_TX_BIT_TIME * 11)

and test with tx_mode 2

I switched to EMSUART_TX_BIT_TIME * 11 on the commit that is now the b9 build.
tx_mode 2 still produces Tx read errors. And on a side note, not sure if somehow related: I think since using your second b8 build, USB power didn't do it anymore, the esp basically starts up, blinks once or a few times and than the LED just goes dark and stays like this, becomes unreachable. Via bus power it works.
The issue is only by using USB power I had 0 Tx read errors. Since I now have to use bus power, I can't exclude this one being responsible for the Tx read errors...

Update: tx_mode 1 appears to produce no Tx read errors. tx_mode 1 never worked for me before:

┌──────────────────────────────────────────┐
│ EMS-ESP version 2.0.0b9_joanwa               │
│ https://github.com/proddy/EMS-ESP        │
│                                          │
│ type help to show available commands     │
└──────────────────────────────────────────┘

ems-esp:/$ show ems
EMS Bus is connected.

EMS Bus info:
  Tx mode: 1
  Bus protocol: HT3
  #telegrams received: 364
  #read requests sent: 73
  #write requests sent: 0
  #corrupted telegrams: 0 (0%)
  #tx fails (after 3 retries): 0

Rx Queue is empty

Tx Queue is empty

[proddy]

ok that looks promising, although not sure why tx_mode 1 works better than tx_mode 2 on your HT3 system. I'll look into the USB/LED issue tonight.

@MichaelDvP tx_mode 1 works fine with both the ESP8266 and ESP32 on my system EMS1.0 system. This is bus-powered. When I plug in a USB from the desktop PC (which is a stable 5v) I start to see errors. My focus is on bus-powered to be compliant with BBQKees's independent circuits. I think now we need some Buderus EMS2.0 and more Junkers/HT3 guinea pigs to try this out.

[MichaelDvP]

@proddy if you power from usb, do you have the dcdc-converter from the gateway removed? If powered from both sides it could make strange efffects.

[proddy]

@proddy if you power from usb, do you have the dcdc-converter from the gateway removed? If powered from both sides it could make strange efffects.

yes, the buck is removed in the ESP32. Ideally I would need to do a logic analyzer scan of bus-powered vs USB (via a powerbank) and compare the charts. And also do a test with the mqtt disabled and then later the complete wifi to see if the wifi is somehow affecting it. And lastly maybe we should look into isolating some of the commands to a single-core on the ESP32 (using SemaphoreHandle_t).

[bbqkees]

I'm now on v2.0.0b9 (not the uart one) and mode 1 works fine with zero errors.
As a test I uploaded the bin file via the web interface of b7 and that went smoothly.

[proddy]

I've been running ESP8266, bus-powered on tx_mode 2 with 0 errors for the last 24hrs. Weird.

[proddy]

On ESP8266, TxMode 1, USB powered I'm getting a 100% success rate with Rx and Tx. But as soon as I change it use bus-powered I get the odd corrupted/incomplete Rx. e.g.

[telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 90 (CRC 90 != 10) ERROR

because its missing the last byte of the telegram. A correct telegram looks like:

[telegram] Rx: 17 0B A8 00 01 00 FF F6 01 06 00 01 0D 01 00 FF FF 01 02 02 02 00 00 05 1F 05 1F 02 0E 00 FF DF (notice the extra byte) OK

@MichaelDvP this is since the last UART updates from https://github.com/proddy/EMS-ESP/commit/d6c5321a5f73e3c28624c1f877af7be3ff44fd06

@bbqkees this is also what you've been experiencing testing the latest v2_cmd branch.

I'll make a capture from USB vs bus-powered with the logic analyzer to see what the differences are.

[MichaelDvP]

Are you sure it's the uart-update? There is no change in rx and i used it before without crc-error, but now i also get some (3 in 36 h). I'm logging now every crc error to syslog to see if there is a system (same src/dest/type).
The uart change was only different break-length for tx-modes modes due to this

[proddy]

I'm seeing this also with USB power (only 5 incompletes in 20hrs) and it only started to happen in the last days when I pushed the big change with commands (v2_cmd), so I agree I don't think it is related to these UART changes. I'm tracing too to syslog to find a pattern

[proddy]

I have a suspicion its due to the NTP server which I re-enabled for the ESP8266 build. When removed I'm not getting Rx errors but it's only been running for the last 2 hrs so need to leave it a while longer to confirm. It may be that the lwip and sntp service is conflicting with the interrupts we use in the UART.

[MichaelDvP]

I had 2 errors yesterday with very strange telegrams, thermostat don't send these messages.

Aug 15 13:34:54 ems - 000+00:00:17.350 E 3: [telegram] Rx: 10 08 19 00 01 08 01 04 01 05 FF FF FF 00 02 0F 48 09 8A E7 00 00 00 09 02 83 02 04 49 7D 00 1D #033[0;31m(CRC 1D != 1F)#033[0m
Aug 15 15:00:54 ems - 000+01:26:18.051 E 449: [telegram] Rx: 10 08 18 1B 1A 08 00 34 00 30 02 72 7D 00 21 00 00 03 00 01 19 45 00 18 B8 D7 #033[0;31m(CRC D7 != A2)#033[0m

Than i updated platformio and framework and get no errors since that (NTP activated, tx_mode 1, virtual roomcontrol-thermostat activated), maybe it was something in the framework.

[proddy]

@MichaelDvP just tried the latest v2 (which was v2_cmd) on an ESP32 and it crashes. Error says "Debug exception reason: Stack canary watchpoint triggered (emsuart_recvTas)" pointing to somewhere in the UART code. Are you seeing the same?

[MichaelDvP]

Tried the latest v2 code. When connected to the bus the esp32 restarts after around half a minute.

[proddy]

same here. Something in the Tx code I think because when I use tx_mode 0 it works. My first thought was because of the dual-core calling the callback functions too fast at the same time now that there is no Rx queue. So may need need to stick in some mutex calls (or use std::atomic). Still investigating.

[MichaelDvP]

I moved process_telegram back to the main thread, but without queue, and esp32 worked without crashes. Since process_telegram also calls a lot of other stuff like mqtt, i think it's better to keep recvTask fast and clean.

void RxService::loop() {
    if (telegram_rx_) {
        (void)EMSESP::process_telegram(rx_telegram); // further process the telegram
        increment_telegram_count();                  // increase count
        telegram_rx_ = false;
    }
}

[proddy]

thanks Michael. I'll try this too and push a fix

[proddy]

ok, back to a working v2 on ESP32 (thanks Michael!) and seeing some errors with bus-powered vs USB-powered (tx mode 1). Running for ~30 minutes below are the stats:

via usb: all is good

via bus-powered: not so good. Both Rx and Tx fails.