Zigbee2mqtt: Split up user and automatic configuration

[deisi]

I'm trying to improve the configuration of the z2m addon for hassio https://github.com/danielwelch/hassio-zigbee2mqtt/pull/115

However, because there is mixing between user input (add something to configuration.yaml) and automatic input up on pairing (also adds something to configuration.yaml), this is more complicated then it should be.
Would it be possible, to split up configuration.yaml into seperate files. I think of something like:

• configuration.yaml
  All user configuration goes here
• devices.yaml
  Gets updated during runtime if new devices are added.
• groups.yaml same as devices.

One could also combine groups.yaml and devices.yaml into a common file, but I have no good name suggestion maybe something like network.yaml

The reason for this is, that the current system forces the hassio addon to somehow merge two sources of configuration.

• 
  
  1. The user configures the addon within the hassio frontend > writes to some options.json and gets passed into configuration.yaml
  
  
• 
  
  1. The network writes something to configuration.yaml and now there is an outch, because we have to telling hassio that the addon changed its own configuration file by itself.
  
  

By splitting of these rather dynamic parts of the configuration into separate files. Things could be simplified, because we could then just say that these parts of z2m must be configured from within the configuration files itself and z2m takes care of the user input within these files (as it does already)

[Koenkk]

What if we could do something like this:

mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://localhost:1883'
  user: my_user

devices: devices.yaml

OR

devices:
  '0x00158d0001d82999':
    friendly_name: 'my_occupancy_sensor'
    retain: true

# same for groups.

In zigbee2mqtt we can easily check if devices is a string or an object.

[deisi]

This is fine as well.

[Koenkk]

Done, documentation: https://github.com/Koenkk/zigbee2mqtt.io/blob/dev/configuration/configuration.md

[andreasbrett]

I dig that also from another point of view. User's comments in configuration.yaml are no longer removed once z2m writes to that file. I love to document my config files (just like in the documentation for configuration.yaml).

[deisi]

Its time to close this then :)

[andreasbrett]

configuration.yaml got overwritten again although I specified separate devices and groups files :(

I'm on z2m commit #61116ec (dev)

[Koenkk]

@andreasbrett how to reproduce? Can you share your configuration.yaml?

[andreasbrett]

@Koenkk I can't reproduce. z2m just got stuck again yesterday when I paired a Aqara sensor to my network. I then restarted z2m and looking at my config file it got overwritten again (all my comments and formats like line breaks are gone). Here's my current config without the formatting.

homeassistant: false
permit_join: false
mqtt:
base_topic: zigbee
server: 'mqtt://localhost'
user: hidden
password: hidden
serial:
port: /dev/CC2531
disable_led: true
advanced:
pan_id: hidden
channel: 11
cache_state: true
network_key:
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 12
baudrate: 115200
rtscts: true
soft_reset_timeout: 0
elapsed: false
availability_timeout: 60
report: true
last_seen: ISO_8601_local
log_level: info
log_directory: data/log/%TIMESTAMP%
devices: devices.yaml
groups: groups.yaml

[Koenkk]

I see,I dont expect this feature to prevent this from happening. Changing e.g. the last_seen via MQTT will also trigger a write to configuration.yaml.

[andreasbrett]

I see,I dont expect this feature to prevent this from happening. Changing e.g. the last_seen via MQTT will also trigger a write to configuration.yaml.

That's too bad. I checked the latest dev branch and found only the last_seen setting that would be changed through the utils/settings.set method.

https://github.com/Koenkk/zigbee2mqtt/blob/889e972176e7bfa4d139d1d2fb2592fbe3ef2ac8/lib/extension/bridgeConfig.js#L87

All other settings that can be changed via MQTT will NOT be written to configuration.yaml. What if we put those dynamic (but persistent) settings like last_seen in a new section dynamic in configuration.yaml. That section would behave just like devices and groups (= can be outsourced to a different file).

Might be that I'm totally wrong as I did not change the last_seen setting through MQTT but I don't see in the code why my configuration.yaml got re-written.

[andreasbrett]

Another option... Swap out js-yaml for yawn-yaml which preserves comments.

https://github.com/mohsen1/yawn-yaml

[Koenkk]

Zigbee2mqtt originally used that library but this gave some problems (I dont remember what).