Core: ModBus Read multiple registers (count:) only displays first register when using data_type: custom

[wills106]

The problem

To reduce the workload on my Solar Inverter I am trying to read multiple registers in a single transaction.

I can read 7 holding registers (0 - 6) that returns the value as a string of 14 numbers / letters that results in the serial number. ie A1A111A1111111 (Technical manual states these registers are Uint16)

If I try and read 3 holding registers (133 - 135) that represent the RTC seconds, minutes and hours using data_type: unit I get the error Unable to detect data type for Solax Group Test 3 sensor, try a custom type (Technical manual states these registers are Uint16)
If I use the following data_type: custom structure: ">3H" I only get the first register displayed as the sensor result.
I am expecting a similar result to reading the registers separately. 12 38 12 Which would be sec min hrs

The same happens when I read multiple input registers. (Technical manual states these registers are Uint16)
Input register 3 and 4 represent the PV Voltage.
Register 3 (256.2v)
Actual value returned 2562
Register 4 (260.1v)
Actual value returned 2601

But again I only ever see the first registers value when reading both together.
I am expecting to see for example 2526 2601 But I would only see 2526

Environment

arch | x86_64
-- | --
chassis |  
dev | false
docker | true
docker_version | 18.09.8
hassio | true
host_os |  
installation_type | Home Assistant Supervised
os_name | Linux
os_version | 4.4.59+
python_version | 3.8.5
supervisor | 247
timezone | Europe/London
version | 0.116.4
virtualenv | false

• Home Assistant Core release with the issue: 0.116.4
• Last working Home Assistant Core release (if known):
• Operating environment (OS/Container/Supervised/Core): Supervised v247
• Integration causing this issue: ModBus
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/sensor.modbus/

Problem-relevant configuration.yaml

Working example where the returned registers just result in a string (Serial Number)

modbus:
  name: SolaX
  type: tcp
  host: !secret inverter_ip
  port: 502

sensor:
- platform: modbus
  scan_interval: 2
  registers:
    - name: SolaX Serial Number
      hub: SolaX
      register: 0
      count: 7
      data_type: string

RTC Example:

sensor:
- platform: modbus
  scan_interval: 2
  registers:
    - name: Solax Group Test 3
      hub: SolaX
      register: 133
      count: 3
      data_type: custom
      structure: ">3H"

PV Voltage Example:

sensor:
- platform: modbus
  scan_interval: 2
  registers:
    - name: SolaX Group Test
      hub: SolaX
      register: 3
      register_type: input
      count: 2
      data_type: custom
      structure: ">2H"

Traceback/Error logs

RTC Log for reading 3 registers

2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Current transaction state - TRANSACTION_COMPLETE
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Running transaction 39
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] SEND: 0x0 0x27 0x0 0x0 0x0 0x6 0x0 0x3 0x0 0x85 0x0 0x3
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.client.sync] New Transaction state 'SENDING'
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] RECV: 0x0 0x27 0x0 0x0 0x0 0x9 0x0 0x3 0x6 0x0 0x28 0x0 0x18 0x0 0xc
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.framer.socket_framer] Processing: 0x0 0x27 0x0 0x0 0x0 0x9 0x0 0x3 0x6 0x0 0x28 0x0 0x18 0x0 0xc
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.factory] Factory Response[ReadHoldingRegistersResponse: 3]
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Adding transaction 39
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Getting transaction 39
2020-10-16 12:23:03 DEBUG (SyncWorker_17) [pymodbus.transaction] Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
2020-10-16 12:23:03 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=sensor.solax_group_test_3, old_state=<state sensor.solax_group_test_3=38; friendly_name=Solax Group Test 3 @ 2020-10-16T12:23:01.226637+01:00>, new_state=<state sensor.solax_group_test_3=40; friendly_name=Solax Group Test 3 @ 2020-10-16T12:23:03.225624+01:00>>

PV Volatage Example

2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Current transaction state - TRANSACTION_COMPLETE
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Running transaction 45
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] SEND: 0x0 0x2d 0x0 0x0 0x0 0x6 0x0 0x4 0x0 0x3 0x0 0x2
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.client.sync] New Transaction state 'SENDING'
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] RECV: 0x0 0x2d 0x0 0x0 0x0 0x7 0x0 0x4 0x4 0xa 0x27 0xa 0x4e
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.framer.socket_framer] Processing: 0x0 0x2d 0x0 0x0 0x0 0x7 0x0 0x4 0x4 0xa 0x27 0xa 0x4e
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.factory] Factory Response[ReadInputRegistersResponse: 4]
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Adding transaction 45
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Getting transaction 45
2020-10-16 13:12:13 DEBUG (SyncWorker_2) [pymodbus.transaction] Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'
2020-10-16 13:12:13 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=sensor.solax_group_test, old_state=<state sensor.solax_group_test=2572; friendly_name=SolaX Group Test @ 2020-10-16T13:12:11.033668+01:00>, new_state=<state sensor.solax_group_test=2599; friendly_name=SolaX Group Test @ 2020-10-16T13:12:13.034278+01:00>>

Additional information

https://community.home-assistant.io/t/modbus-how-to-read-multiple-registers-how-to-use-data-type-and-structure/234655

https://docs.python.org/3.2/library/struct.html

[probot-home-assistant[bot]]

modbus documentation
modbus source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @adamchengtkc, @janiversen, @vzahradnik, mind taking a look at this issue as its been labeled with an integration (modbus) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[wills106]

I have noticed there is a PR to add data_count: to the climate section of ModBus
https://github.com/home-assistant/core/pull/32439

Does this work in the same way as the count: I am using, or is it a different approach?
If it's a different approach, could this be causing the issue I am experiencing where I can only see the value of the first register?

[vzahradnik]

Does this work in the same way as the count: ?

Yes, I ported the code to Climate. Except the data_count attribute, there's no difference.

[wills106]

The same happens when I read multiple input registers. (Technical manual states these registers are Uint16)
Input register 3 and 4 represent the PV Voltage.
Register 3 (256.2v)
Actual value returned 2562
Register 4 (260.1v)
Actual value returned 2601

But again I only ever see the first registers value when reading both together.
I am expecting to see for example 2526 2601 But I would only see 2526

I have tried adding:
reverse_order: true

As the documentation states:

reverse_order boolean (optional, default: false)
Reverse the order of registers when count >1.

But all that results in is you seeing 2601 instead of 2526, so it still only display the one register.

[vzahradnik]

I'd like to see at the issue this Weekend. If I find something, I will let you know.

[vzahradnik]

I know what's the problem. I'm not sure if it's a bug, but I will sum it up here.

How it works

• We internally use a table, which finds a corresponding struct based on requested data type and data count. It looks like this:

DEFAULT_STRUCT_FORMAT = {
    DATA_TYPE_INT: {1: "h", 2: "i", 4: "q"},
    DATA_TYPE_UINT: {1: "H", 2: "I", 4: "Q"},
    DATA_TYPE_FLOAT: {1: "e", 2: "f", 4: "d"},
}

• For example, if you want to read a value from two registers as a float, you'll get the structure f
• If we don't find a match, you'll get the message to use custom data type
• In your case, you want to read 3 registers, which is not in a dictionary
• As a reference, here's the conversion table:
  https://docs.python.org/3/library/struct.html

Your example:

...
count: 3
data_type: custom
structure: ">3H"

In your case, we correctly read 3 registers. Later, we use the struct definition >3H to parse the value.
Here's the relevant code:

byte_string = b"".join([x.to_bytes(2, byteorder="big") for x in registers])
if self._data_type != DATA_TYPE_STRING:
    val = struct.unpack(self._structure, byte_string)[0]

What the code does?

• First, we read the bytes into a byte array (I think this can be improved/avoided, but it's not an issue).
• The problem is in the struct.unpack(...) method. Our code expects that the unpack method returns a single value. Note the [0]. It ignores the rest
• Normally, this is not a problem. If there's a datatype in Python, which is large enough to hold the data, and there exist a struct format to return a single value. This value is then converted into a string and displayed in the UI
• But your struct actually returns 3 values, not just one
  

Solution?

Ideally, you should define your struct format so that it parses the bytes into a single value. Perhaps we could extend the code to check for the size of the array after unpack(...), but I'm not sure, how we should represent such data. We need to combine those 3 values into a single value, which can be stored internally.

@janiversen any suggestions?

[tipi85]

The capability to read multiple registers at once and to assign them to different sensors would be a very nice and efficient solution. It would suits more to the Read Multiple Register from the Modbus protocol.

For example, I am driving with HA a De Dietrich boiler via a RTU Serial Modbus connections. Today HA is pooling 30 registers one by one. It is not optimized and creates limits in pooling capability. With a buffer read of 128 registers ( I think it is the maximum defined in the standard ), HA could make only 2 reads, populating an array. Even if the limit would be 10 to 16 registers it would be a great help.

[wills106]

For me personally it would be ideal if the value was returned comma separated.
The reason being is that not all registers return a 4 digit number.
If I read input registers 0-9 I get the following:
2438
20
435
1253
1250
4
4
4994
33
1
So Ideally when I use count: 10
My sensor would return 2438,20,435,1253,1250,4,4,4994,33,1

Then I can just split at the commas into separate sensors using a template.

I thought about splitting them after x numbers, but the likes of 20, 435, 4 & 4 in the above don't always stay 2, 3 & 1 digit long. They become 3, 4 or 2 long depending on the current / power generated.

So if I used the following:

    solax_split_1:
      friendly_name: "SolaX Split Sensor 1"
      value_template: '{{ states.sensor.solax_group_test[0:4] }}'
    solax_split_2:
      friendly_name: "SolaX Split Sensor 2"
      value_template: '{{ states.sensor.solax_group_test[4:6] }}'
    solax_split_3:
      friendly_name: "SolaX Split Sensor 3"
      value_template: '{{ states.sensor.solax_group_test[6:9] }}'

I would sometimes be reading values from a different register altogether, depending on how lengths change over the day.

[vzahradnik]

I think we could implement this. If the unpack() returns more than one number, we'll separate values by comma.

Today I'll prepare a pull request.

[wills106]

If you want me to try out the code change as a Custom Component of ModBus before you submit the PR just let me know.

[vzahradnik]

That would be great! I will mention you in the PR, then you can grab the code.

[vzahradnik]

@wills106 it's implemented in PR #42354. Feel free to grab the code. If you find any issues with the code, please report them in the PR. Thanks!