Rtl_433: New firmware for EmonTX breaks the protocol

[hkapanen]

EmonTxV3, the OpenEnergyMonitor transmitter, is now shipping with a new firmware that changes the protocol:

_Note: This firmware uses data whitening to improve the reliability of the data transmission. A standard pattern of 1s and 0s is overlayed on the underlying data, this prevents sync issues that result from too many zero values in a packet. When the packet is received it is decoded by emonhub. The 'whitening = 1' is required to tell emonhub to decode the packet correctly._

   #ifdef RF_WHITENING
       byte WHITENING = 0x55;
       for (byte i = 0, *p = (byte *)&tmp; i < sizeof tmp; i++, p++)
           *p ^= (byte)WHITENING;
   #endif

Some samples of the new firmware transmissions can be found in this Google groups thread.

Can you please add support for this as well.

[klohner]

Here's a first stab at an analysis. Was able to pull the data from these samples using this flex spec.

$ rtl_433 -R 0 -s 1024k -X n=emonTx,m=FSK_PCM,s=20,l=20,r=1000 -r g002_433.92M_1024k.cu8

However, g001 and g006 don't seem to contain the signal, and the messages returned by rtl_433 for the other cu8 files seem to contain a few incorrect bits in spots. I was able to filter the signal and pull clean messages with urh.

Here's a bitbench with a decoding that I modeled after RF12 packet format and design and RF12 broadcasts and ACKs documents.

Computing and comparing the CRC-16 on some of the older messages and these with reveng, it looks like the same polynomial.

$ reveng -w 16 -s 2dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000399f 2dd2081a8404010035043d00ce5db80bb80bb80bb80bb80bb80b3d04000012a6 2dd2081a9304000029043a00615cb80bb80bb80bb80bb80bb80ba2b0000073cb
width=16  poly=0x8005  init=0x9e7f  refin=true  refout=true  xorout=0x0000  check=0xe1bc  residue=0x0000  name=(none)

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000
399f

$ reveng -w 16 -s 2dd20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22 2dd20f2845555555e83455555555555555555555555555555555555555555555555565206520652055555555d8d6 2dd20f2844555555dc34555555555555555555555555555555555555555555555555652065206520555555557243 2dd20f28475555553e34555555555555555555555555555555555555555555555555652065206520555555551e5b 2dd20f2846555555bc34555555555555555555555555555555555555555555555555652065206520555555552406 
width=16  poly=0x8005  init=0x9e7f  refin=true  refout=true  xorout=0x0000  check=0xe1bc  residue=0x0000  name=(none)

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

The old style messages decode with rtl_433 -y if we append the normal expected 0x555555 prefix and '0xaa' suffix.

$ rtl_433 -R 45 -y {288}5555552dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000399faa
model     : emonTx-Energy node      : 08           ct1       : 7             ct2       : 0             ct3       : 0             ct4       : 0             batt_Vrms : 236.65
pulse     : 1

However, these new whitened messages come with a slightly different prefix of 0x555555aa with the expected 0xaa suffix. It may be that the 0x555555aa is some kind of flag to say that the message is whitened. The whitening seems to be repeating 0x55 XOR'd just to the payload.

I tried to trick rtl_433 to decode the whitened g002 sample by de-whitening the payload, recomputing the CRC-16, and adding the old-style 0x555555prefix and 0xaasuffix, but this was not successful. The current driver doesn't provide - vv messages, but it seems like it's picky about the payload being exactly 26 (0x1a) bytes to decode fields, and not the 40 (0x28) byte payload specified in the header and found in these new messages.

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

WHITENED PAYLOAD: 5a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
WHITENING:        55555555555555555555555555555555555555555555555555555555555555555555555555555555
XOR'd PAYLOAD:    0f000000ae6100000000000000000000000000000000000000000000000030753075307500000000

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f280f000000ae6100000000000000000000000000000000000000000000000030753075307500000000
5d9c

rtl_433 -vv -R 45 -y {400}5555552dd20f280f000000ae61000000000000000000000000000000000000000000000000307530753075000000005d9caa
[no result]

There seems to be a good bit of work needed in order to get the driver to recognize and decode these new whitened signals. I don't have the bandwidth right now to do this.

[klohner]

Figured out a bit more.

First, the CRC-16 does not include the 0x2d sync byte in its computation. Removing this before feeding to reveng changes the CRC-16 initial value to 0xffff, which with polynomial 0x8005 (or reversed as 0xa001 for how _crc16_update() knows it) gives us a recognized CRC format of "crc-16/modbus".

$ reveng -w 16 -s d20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22 d20f2845555555e83455555555555555555555555555555555555555555555555565206520652055555555d8d6 d20f2844555555dc34555555555555555555555555555555555555555555555555652065206520555555557243 d20f28475555553e34555555555555555555555555555555555555555555555555652065206520555555551e5b d20f2846555555bc34555555555555555555555555555555555555555555555555652065206520555555552406
width=16  poly=0x8005  init=0xffff  refin=true  refout=true  xorout=0x0000  check=0x4b37  residue=0x0000  name="CRC-16/MODBUS"

$ reveng -m crc-16/modbus -c d20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

I took a look at the older EmonTx3 firmware and the newer EmonTxV3CM firmware and see that besides the whitening, the message structure has changed. The first byte of the HEAD section is the Node ID. Depending on this ID, the format of the PAYLOAD could be one of several different formats.

There's info on most all of the older types, including the "EmonTx v3" 0x08 format which rtl_433 currently supports, in the EmonHub Configuration document. This newest "EmonTxV3" format (which includes the data whitening feature) is shown in comments near the top of its firmware source code.

Using all this info, I was able to take the de-whitened payload from the g002 sample (which has a NodeID of 15 (0x0f), reformat it like the older EmonTx v3 NodeID 0x08 format, compute a CRC, and feed it to rtl_433 for parsing.

# Orignal g002 message:
# {408}555555aa2dd20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22aa
# KEY: ~~~~~~~~SSGGNNLLPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPCCCC~~
#   ~ = prefix and suffix of signal, typically 0x55 or 0xaa to help get bitrate
#   S = SYNC (0x2d is what the HOPE RF12B and RF69CW chips like)
#   G = Group (0xd2 (int 210) seems to be hardcoded into the EmonTx3 source)
#   N = Node ID (helps identify the EmonTX payload format)
#   L = Length of Payload in bytes (up to 66 bytes as defined by jeelabs R12)
#   P = Payload (format of this is typically determined by the NodeID, may be "whitened" by XOR'd with 0x55)
#   C = CRC-16 (poly=0x8005  init=0xffff) "CRC-16/MODBUS" for Group through Payload bytes

Node ID:           0f

Payload Format for Node ID 15:
  nodename = EmonTxV3CM_15
  [[[rx]]]
    names = MSG, Vrms, P1, P2, P3, P4, E1, E2, E3, E4, T1, T2, T3, pulse
    datacodes = L,h,h,h,h,h,L,L,L,L,h,h,h,L
    scales = 1,0.01,1,1,1,1,1,1,1,1,0.01,0.01,0.01,1
    units = n,V,W,W,W,W,Wh,Wh,Wh,Wh,C,C,C,p

PAYLOAD:               5a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
XOR'd PAYLOAD:         0f000000ae6100000000000000000000000000000000000000000000000030753075307500000000
KEY:                   MSG-----VrmsP1--P2--P3--P4--E1------E2------E3------E4------T1--T2--T3--PULSE---

Payload Format for Node ID 8:
  nodename = emonTx_3
  firmware =V2_3_emonTxV3_4_DiscreteSampling
  hardware = emonTx_(NodeID_DIP_Switch1:OFF)
  [[[rx]]]
     names = power1, power2, power3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
     datacodes = h,h,h,h,h,h,h,h,h,h,h,L
     scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
     units =W,W,W,W,V,C,C,C,C,C,C,p

KEY:                   P1--P2--P3--P4--VrmsT1--T2--T3--T4--T5--T6--PULSE---
Reformatted PAYLOAD:   0000000000000000ae6130753075307530753075307500000000
with GRP + HEAD: d2081a0000000000000000ae6130753075307530753075307500000000

$ reveng -m crc-16/modbus -c d2081a0000000000000000ae6130753075307530753075307500000000
9775

$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61307530753075307530753075000000009775aa
model     : emonTx-Energy node      : 08           ct1       : 0             ct2       : 0             ct3       : 0             ct4       : 0
batt_Vrms : 250.06       pulse     : 0             temp1_C   : 3000.0        temp2_C   : 3000.0        temp3_C   : 3000.0        temp4_C   : 3000.0
temp5_C   : 3000.0       temp6_C   : 3000.0        Integrity : CRC

The spurious temperature readings highlight something interesting. The firmware source code implements some special temperature values to denote a few special cases. In emonLibCM.h we find:

#define UNUSED_TEMPERATURE 30000     // this value (300C) is sent if no sensor has ever been detected
#define OUTOFRANGE_TEMPERATURE 30200 // this value (302C) is sent if the sensor reports < -55C or > +125C
#define BAD_TEMPERATURE 30400        // this value (304C) is sent if no sensor is present or the checksum is bad (corrupted data)
                                     // NOTE: The sensor might report 85C if the temperature is retrieved but the sensor has not been commanded
                                     //  to measure the temperature.

The driver does know about the 300C special case (which was used in the older 0x08 Node ID), but it's not picked up because the new 0x0f Node ID protocol defaults temperature values to be 0,01 * value, instead of 0,1 * value as seen in the 0x08 Node ID protocol. So, for sake of completeness to force the new data into the old format:

300C * 100 = 30,000 (0x7530)
300C *  10 =  3,000 (0x0bb8)

So change: d2081a0000000000000000ae6130753075307530753075307500000000
To:        d2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b00000000

$ reveng -m crc-16/modbus -c d2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b00000000
5d6e

$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b000000005d6eaa
model     : emonTx-Energy node      : 08           ct1       : 0             ct2       : 0             ct3       : 0             ct4       : 0
batt_Vrms : 250.06       pulse     : 0

And finally, there doesn't seem to be any flag to say that the payload has been whitened. Although whitening is the default and should probably be assumed if Node ID is 15 or 16. However, perhaps an extremely high power reading (like above 16,383 watts) would be a good indicator that the entire payload should have been de-whitened first.

[hkapanen]

Bump -- could anyone help on this?