Zigbee2mqtt: [RFC] Revising MQTT bridge structure

[Koenkk]

Problem

The current zigbee2mqtt/bridge/# topics are a mess. They are not well structured and don't provide a good reply/response mechanism. This makes it hard for other applications to integrate with Zigbee2mqtt. Therefore I propose the revise the structure while keeping the current working (so no breaking change).

Proposal

I didn't work out the full new structure yet, but this is the general idea I have.
@ anyone, feel free to review this, but I especially would like to request @carldebilly to review this (since I think it can be for great use of Zigbee2mqttAssistant).

The initial implementation has been completed and can be tested in the latest dev branch, to enable the new api set the following in configuration.yaml.

experimental:
  new_api: true

In case you want to disable the legacy api add the following (not mandatory, legacy and new api can exist simultaneously)

advanced:
  legacy_api: false

The documentation can be found here.

Note that the api can still change as it's not officially released yet. The plan is to finalise it on 1 September 2020, any feedback resulting in a breaking change should be submitted before this date.

TODO

• [x] Change ID -> id in payloads (requested by @sjorge https://github.com/Koenkk/zigbee2mqtt/issues/3281#issuecomment-658040308) (breaking)
• [x] Add healthcheck (requested by @jsassner https://github.com/Koenkk/zigbee2mqtt/issues/3281#issuecomment-658032089)
• [x] Allow to specify clusters to bind/unbind to binding commands (requested by @sjorge)
• [x] Post logging to MQTT (requested by @CodeFinder2 https://github.com/Koenkk/zigbee2mqtt/issues/3281#issuecomment-633264118)
• [x] Enable/disable Home Assistant extension (requested by @bodiroga https://github.com/Koenkk/zigbee2mqtt/issues/3281#issuecomment-633248891)
• [x] Publish update information (e.g. {"update": {"progress": 20, "state": "available/updating/idle"}) to device state topics. (requested by @SeLLeRoNe https://github.com/Koenkk/zigbee2mqtt/issues/3634#issuecomment-659262736)

[sjorge]

Can we also get a configuration.yaml option to disable the old behavior?

[gdave321]

IMHO To be of any use the "everything" to zigbee2mqtt/bridge should include for the devices the type of the device (this is also missing in the current config/devices/get). If type like lamp/color_temp/color/motionsensor/sensor/switch would be available, it can be used to auto create devices in application. Now this info is only available via homeassistant_discovery_topic but there are plenty applications around not really HA compatible.

[sjorge]

That would be nice, then maybe HA extension can maybe move to a seperate reposed ourside of the core.

But there would probably need to be a better way to show what a device supports, sort of like feature flags.

Type: bulb, features: on/off, colorxy, brightness,... not sure how feasible that is though.

~ sjorge

On 5 Apr 2020, at 13:38, gdave321 [e-mail] wrote:


IMHO To be of any use the "everything" to zigbee2mqtt/bridge should include for the devices the type of the device (this is also missing in the current config/devices/get). If type like lamp/color_temp/color/motionsensor/sensor/switch would be available, it can be used to auto create devices in application. Now this info is only available via homeassistant_discovery_topic but there are plenty applications around not really HA compatible.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[nldroid]

Just my 2ct: if you publish 'everything' for zigbee2mqtt/bridge, the message could be huuuge. I've seen people on the issue list with 200 devices. Wouldn't it make sense to have a zigbee2mqtt/bridge/devices topic for that?

[Koenkk]

@gdave321 if some new discovery will be implemented, I think it should be an existing one, don't want to create yet another discovery mechanism.

@nldroid it can indeed grow large, but what would be the problem of this? It's not really meant for humans.

[sjorge]

On my earlier point, it doesn’t make sense for extensions to expose this over mqtt, the should be able to get this info some otherwise so disregard that remark.

[jsassner]

What i've been missing for a while is a Health Check topic, and that would fit really well in the Request/Response structure.
I have systems that want to check every x minutes that z2m is up and running (else it will warn me), and posting to zigbee2mqtt/bridge/request/healthcheck getting a response would be really good.

[nldroid]

@nldroid it can indeed grow large, but what would be the problem of this? It's not really meant for humans.

Yes but i still use MQTT explorer sometimes to check things,

Imho there should be a good balance in message size and number of topics. They didn't invent the hierarchy for nothing.

For software it's not a big deal to do two requests to get information about the bridge and the devices.

[Koenkk]

@nldroid indeed a valid point, so I think we should have zigbee2mqtt/bridge and zigbee2mqtt/bridge/entities (groups + devices). Just for my understanding, what information do you typically want to get from zigbee2mqtt/bridge?

[sjorge]

@nldroid @Koenkk we already have a zigbee2mqtt/bridge/group/ end point, maybe add a list to that?

Then we can perhaps move some of the bind / device stuff to zigbee2mqtt/bridge/device/ ?

[carldebilly]

Sorry I was late to respond... I'm over busy those days... a full time job at home + a full time daddy is asking a lot.

Love the request id... I was about to suggest it when I read it :-). Love the idea of this refactoring.

But my concern is about the migration from the old to the new topics. Z2MA is not the only tool right now... Do you think we should have a hybrid mode for the migration time?

And... what is the timeline for that? I want to be ready on time for the release. Maybe @magpern can help too :-) (he's very helpful!).

[sjorge]

But my concern is about the migration from the old to the new topics. Z2MA is not the only tool right now... Do you think we should have a hybrid mode for the migration time?

From the commits I'm seeing on dev, yes there is a legacy mode which is on by default

[Koenkk]

@sjorge it will by zigbee2mqtt/bridge/entities so no need to worry about that. Bind unbind stuff will move to something like zigbee2mqtt/bridge/request/bind.

@carldebilly there won't be a breaking change so the old API will still work. Timeline: not really soon, first want to have a proper discussion + prepare zigbee2mqtt internally for this.

[magpern]

@carldebilly

Maybe @magpern can help too :-) (he's very helpful!).

You make me blush!

Yes, count me in. I have tonnes of day job disturbing my spare time right now, but it will calm down in 30 days or so.

[nldroid]

Just for my understanding, what information do you typically want to get from zigbee2mqtt/bridge?

For me, the most important thing is the version. Best would be with latest commit number so you'll never have any doubt on what version people are.

[CodeFinder2]

I would also suggest to split "everything" of zigbee2mqtt/bridge, maybe to zigbee2mqtt/bridge (version, commit hash, permit_join, etc), zigbee2mqtt/bridge/devices (all device related information) and zigbee2mqtt/bridge/groups (for group related information). This structure reduces the network load, simplifies dealing with tools like MQTT Explorer, and (hopefully) increases the performance of JSON parsing.

Another thing: what about sending all logging output to some (separate) MQTT topic? IIRC, this is another desired feature of Z2MA.

(I was thinking to develop a Qt based desktop application as a UI for Z2M.)

[Koenkk]

@CodeFinder2 good idea will include this.

[sjorge]

perhaps rename <prefix>/bridge to <prefix>/coordinator ? That would also allow legacy and the new stuff to both be active.

[Koenkk]

@sjorge not sure, it's not only about the coordinator but just the whole stuff itself. Currently we didn't get into problems regarding conflicts with the old api.

[CodeFinder2]

@Koenkk great, thanks! Maybe also split the various "attributes" into a dedicated JSON message (and not only provide a single string as printed in the log), like (at least)

{
    "type": "info",
    "time": "2020-04-22 22:48:10",
    "message": "... message as printed in Z2M's log ..."
}

as this allows for a more fine-grained visualization (of the log). I don't know (yet) if it makes sense to even split (the content of) message into more attributes (possibly depending on the concrete message).

(In a UI frontend, I am also thinking about live-filtering the log based on some input pattern, e.g., to show only log entries for a specific device. For instance, this might be helpful in case of a pairing process where we are only interessted in observing pairing-device related messages. Also, showing all errors in the recent past would then be very easy.)

[IVI053]

If you go for <prefix>/bridge/devices then I would prefer <prefix>/bridge/config (or configuration, settings, what ever word you prefer) for the other information. I find it is kind of a bad practise to publish to a topic and also use it in deeper path. I know it is done a lot but it makes subscription using wildcards a bit more inconvenient. It also fits better for things like <prefix>/bridge/event IMHO

[Koenkk]

@IVI053 I didn't update the OP, but that's indeed what I've done. You can find a sneak peak here: https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/information/mqtt_topics_and_message_structure_new.md

[sjorge]

@IVI053 I didn't update the OP, but that's indeed what I've done. You can find a sneak peak here: https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/information/mqtt_topics_and_message_structure_new.md

Some minor feedback on the docs

• zigbee2mqtt/bridge/devices
  
  
  
  • add an unsupported device to the example (makes it easier for app devs to handle those payloads)
  
  
• zigbee2mqtt/bridge/groups
  
  
  
  • does it also update on rename of a group (wait do we even support that?)
  
  
• zigbee2mqtt/bridge/event
  
  
  
  • maybe expand the event json examples to be consistent with the rest of the document (it's also easier to read)
  
  

[bodiroga]

Hi @Koenkk!

What about adding the option to enable/disable extensions (Home Assistant support, for example) through MQTT? The information could be added to the zigbee2mqtt/bridge/info topic, as "homeassistant": false or create a new topic like:

zigbee2mqtt/bridge/extensions

The value could be changed using the request/respond mechanism:

zigbee2mqtt/bridge/request/homeassistant, with the following allowed payloads: { "value": true} and { "value": false}. That being said, the new value should be persisted in the configuration.yaml file, unlike the permitJoin parameter.

Is this something that has been considered?

Thanks for your amazing work!

[sjorge]

For a 2.0 I’d love to see just a list of extensions to load. That would also cut down on a bit of config file clutter. And maybe a namespace for per-extension configuration under extensions->NAME->prop or something.

~ sjorge

On 24 May 2020, at 17:36, Aitor Iturrioz [e-mail] wrote:


Hi @Koenkk!

What about adding the option to enable/disable extensions (Home Assistant support, for example) through MQTT? The information could be added to the zigbee2mqtt/bridge/info topic, as "homeassistant": false or create a new topic like:

zigbee2mqtt/bridge/extensions

The value could be changed using the request/respond mechanism:

zigbee2mqtt/bridge/request/homeassistant, with the following allowed payloads: { "value": true} and { "value": false}. That being said, the new value should be persisted in the configuration.yaml file, unlike the permitJoin parameter.

Is this something that has been considered?

Thanks for your amazing work!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Koenkk]

Yes thats possible, however I will first focus on porting the existing functionality.

[CodeFinder2]

@IVI053 I didn't update the OP, but that's indeed what I've done. You can find a sneak peak here: https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/information/mqtt_topics_and_message_structure_new.md

@Koenkk Will you also add the "log via MQTT" stuff I was suggesting here (asking because it's not listed in the linked doc or is it just missing yet)?

[Koenkk]

Yes logging will also be available, its far from completed yet, slowly adding stuff

[timoline]

Hello,
......

Same with last_seen, maybe an idea to make it default (ISO_8601 is very common in influxdb / grafana)
allthough last_seen is very clear what it does, but imho timestamp is a better choice in a mqtt structure. / time series database.

my 2 cents

[Koenkk]

The initial implementation has been completed, see OP for more details. I will scan through this thread somewhere this week and add a TODO list to the OP.

[jsassner]

What i've been missing for a while is a Health Check topic, and that would fit really well in the Request/Response structure.
I have systems that want to check every x minutes that z2m is up and running (else it will warn me), and posting to zigbee2mqtt/bridge/request/healthcheck getting a response would be really good.

Please if you would add this one to the TODO

[sjorge]

@Koenkk do we also accept id in the payload? everything seems to be lower snake case (yay!) except mention of ID are capitalized, having that lowercase (or both) would be more constant.

[timoline]

AFAIK mqtt topics should always be lowercase, have no spaces or strange/weird chars

[Koenkk]

Added todo list

[sjorge]

In case you want to disable the legacy api add the following (not mandatory, legacy and new api can exist simultaneously)

Should the listed code snippet not have legacy_api: false instead of true?

Enable/disable Home Assistant extension
Will this be generic or just home assistant? Like can we enable/disable reporting, availability checks, ... via mqtt, that would be nice.

[Koenkk]

Should the listed code snippet not have legacy_api: false instead of true?
Fixed

Will this be generic or just home assistant? Like can we enable/disable reporting, availability checks, ... via mqtt, that would be nice.

This will not be generic as extensions may also require a teardown so just stop the extension itself may not be sufficient for all.

[sjorge]

This will not be generic as extensions may also require a teardown so just stop the extension itself may not be sufficient for all.

Can we to be future proof for other extensions still have the MQTT topics be generic? Something like
zigbee2mqtt/bridge/extension/(enable/disable) and for now just support homeassistant?

I hope to pick back up the extension framework once things settle down for me a bit, COVID + in progess move has left with with little free time currently but hope to have more again to soon.

[Koenkk]

@sjorge I don't think users should be bothered with the concept extension as it's something internal (while I think it's fine to do so for the user written extensions). For Home Assistant e.g. zigbee2mqtt/bridge/request/config/homeassistant will be used.

[Koenkk]

@jsassner health check has been implemented, docs: https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/mqtt_topics_and_message_structure.md#general

Currently it will always return healthy: true (unless zigbee2mqtt is not running of course), have to think about cases when healthy would be false.

[jsassner]

@jsassner health check has been implemented, docs: https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/mqtt_topics_and_message_structure.md#general

Currently it will always return healthy: true (unless zigbee2mqtt is not running of course), have to think about cases when healthy would be false.

Thank you!! ❤️

[lionel590]

Hi I'm testing using the new API. 'zigbee2mqtt/bridge/devices reports in its paylaod what the device(s) do support which is great.
However:

-(1) It reports it as a string for exemple Philips motion sensor reports > "supports":"occupancy, temperature, illuminance"
While we can parse the string I think it would make more sense to report it as an array > "supports":["occupancy", "temperature", "illuminance"]

-(2) the format is inconsistant here is what reports a Xiaomi motion sensor > "supports":"occupancy and illuminance"

-(3) we should include all options supported. In the case of Xiaomi for exemple we should include "battery" and "voltage". This is not abvious as some sensors don't report battery / voltage

I can of course keep my own table but this defeats the purpose of adding this information to /devices topic

[Koenkk]

@lionel590 the support attribute was meant for the docs. I think a new attribute should be added to https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/devices.js (replacing supports) called exposes, which can e.g. be exposes: ["light_brightness"], exposes: ["action"], exposes: ["switch"], exposes: ["switch_l2"] (for devices with endpoint)

[lionel590]

Ok Thx will update teh devices I have here

[lionel590]

Hi Koenkk,

Today devices are publishing their state to 'zigbee2mqtt/[FRIENDLY_NAME]' which mean that from an app point of view I have 2 options to subscribe to the published messages:

(1) - I need to subscribe to 'zigbee2mqtt/#' but then I get polluted because I get also all the requests, not ideal

(2) - subscribe to 'zigbee2mqtt/[FRIENDLY_NAME]' for each device which is way too heavy since you need to add all the housekeeping mechanisms

So I would suggest to move 'zigbee2mqtt/[FRIENDLY_NAME]' to 'zigbee2mqtt/bridge/device/[FRIENDLY_NAME]' It would also make it more consistent since we have already i.e zigbee2mqtt/bridge/devices

What do you think ? at the minimum make "device" topic configurable via the config file ?

[Koenkk]

Hi Koenkk,

Today devices are publishing their state to 'zigbee2mqtt/[FRIENDLY_NAME]' which mean that from an app point of view I have 2 options to subscribe to the published messages:

(1) - I need to subscribe to 'zigbee2mqtt/#' but then I get polluted because I get also all the requests, not ideal

(2) - subscribe to 'zigbee2mqtt/[FRIENDLY_NAME]' for each device which is way too heavy since you need to add all the housekeeping mechanisms

So I would suggest to move 'zigbee2mqtt/[FRIENDLY_NAME]' to 'zigbee2mqtt/bridge/device/[FRIENDLY_NAME]' It would also make it more consistent since we have already i.e zigbee2mqtt/bridge/devices

What do you think ? at the minimum make "device" topic configurable via the config file ?

Subscribing to zigbee2mqtt/+ should do it.

[lionel590]

Hi Koenkk,
Today devices are publishing their state to 'zigbee2mqtt/[FRIENDLY_NAME]' which mean that from an app point of view I have 2 options to subscribe to the published messages:
(1) - I need to subscribe to 'zigbee2mqtt/#' but then I get polluted because I get also all the requests, not ideal
(2) - subscribe to 'zigbee2mqtt/[FRIENDLY_NAME]' for each device which is way too heavy since you need to add all the housekeeping mechanisms
So I would suggest to move 'zigbee2mqtt/[FRIENDLY_NAME]' to 'zigbee2mqtt/bridge/device/[FRIENDLY_NAME]' It would also make it more consistent since we have already i.e zigbee2mqtt/bridge/devices
What do you think ? at the minimum make "device" topic configurable via the config file ?

Subscribing to zigbee2mqtt/+ should do it.

Hi Koenkk,, yes correct, indeed that will do the trick ;)

[SeLLeRoNe]

I've enabled the new_api but I still see the old format, do I need to wait for an update to be available?

[Koenkk]

@SeLLeRoNe checking for an update is also enough.

[SeLLeRoNe]

Perfect! I thought z2m was checking on startup :)

[sjorge]

Is there a way to manually trigger a publish of zigbee/bridge/[devices|groups] ?

[Koenkk]

@sjorge currently not, this message is send as retained, so the application should always have the latest available

[sjorge]

Notice something else:

Zigbee2MQTT:debug 2020-08-20 15:30:53: Device 'office/desk_light/bulb' announced itself
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/lights/panel2', type 'readResponse', cluster 'genBasic', data '{"zclVersion":1}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Successfully pinged 'office/lights/panel2'
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/lights/panel1', type 'readResponse', cluster 'genBasic', data '{"zclVersion":1}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Successfully pinged 'office/lights/panel1'
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/desk_light/bulb', type 'attributeReport', cluster 'lightingColorCtrl', data '{"colorMode":1,"colorTemperature":374,"currentHue":21,"currentSaturation":197,"currentX":30199,"currentY":26096}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/desk_light/bulb', type 'readResponse', cluster 'genOnOff', data '{"onOff":1}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/desk_light/bulb', type 'readResponse', cluster 'genLevelCtrl', data '{"currentLevel":254}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/desk_light/bulb', type 'readResponse', cluster 'lightingColorCtrl', data '{"colorTemperature":374,"currentX":30199,"currentY":26096}' from endpoint 1 with groupID 0
Zigbee2MQTT:debug 2020-08-20 15:30:54: Received Zigbee message from 'office/desk_light/bulb', type 'attributeReport', cluster 'genOnOff', data '{"onOff":1}' from endpoint 1 with groupID 0
Zigbee2MQTT:info  2020-08-20 15:30:55: MQTT publish: topic 'zigbee2mqtt/office/desk_light', payload '{"brightness":254,"color":{"hue":30,"saturation":78,"x":0.4608,"y":0.3982},"color_temp":374,"state":"ON"}'

There was a device announce for 'office/desk_light/bulb' but there doesn't seem to be a zigbee2mqtt/bridge/event for this? It would be nice if there was.

[Koenkk]

@sjorge it's indeed not published as I can't imagine a use case for this (yet).

[sjorge]

Well I wanted to toggle my S1’s state to ON when they get power again so my lamps also get power. They default to off when power is restored.

So instead of working with the awkward extension I wrote, I though I’d do it over mqtt which, is more flexible.

You could probably also use it to do a poll of a devices state after a short delay too if it doesn’t support reporting for example.

~ sjorge

On 20 Aug 2020, at 20:29, Koen Kanters [e-mail] wrote:


@sjorge it's indeed not published as I can't imagine a use case for this (yet).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Koenkk]

@sjorge added!

[sjorge]

@Koenkk thanks! works like a charm, and much clear than messing with an extension.

A simple node-red function (well in a subflow for the config), I now have one of those for each switch and it works like a charm. (I did add a 5 sec delay, to give the device time to settel.

MQTT_PREFIX=zigbee2mqtt
MQTT_DEVICE=office/lights/switch
MQTT_DEVICE_PROPERTY=state
MQTT_DEVICE_VALUE=on

// decode json payload
if (typeof msg.payload == "string") {
    try {
        msg.payload = JSON.parse(msg.payload);
    } catch (e) {
        // payload was not json
    }
}

// route messages
if ((msg.topic !== undefined) && (msg.topic == env.get("MQTT_PREFIX") + "/bridge/event")) {
    if ((msg.payload.type === "device_announce") && (msg.payload.data.friendly_name === env.get("MQTT_DEVICE"))) {
        return {
            "topic": env.get("MQTT_PREFIX") + "/" + env.get("MQTT_DEVICE") + "/set/" + env.get("MQTT_DEVICE_PROPERTY"),
            "payload": env.get("MQTT_DEVICE_VALUE"),
        }
    }
}

return null;

edit: would be nice that device offline/online messages from availability also generated an event on the bus.

[sjorge]

I might have hit a bug with group removal

Zigbee2MQTT:info  2020-08-20 19:47:24: MQTT publish: topic 'zigbee2mqtt/bridge/groups', payload '[{"friendly_name":99,"id":99,"members":[]},{"friendly_name":"entrance/light","id":10101,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffe1476dc"}]},{"friendly_nam
e":"kitchen/table_lights","id":10102,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffe393e9e"},{"endpoint":1,"ieee_address":"0x680ae2fffe0c91ba"},{"endpoint":1,"ieee_address":"0x680ae2fffe34ab1d"},{"endpoint":1,"ieee_address":"0xccccccfffe2a7b3e"}]},{"friendl
y_name":"kitchen/workarea_lights","id":10103,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffe37240d"},{"endpoint":1,"ieee_address":"0xccccccfffef4c3b6"},{"endpoint":1,"ieee_address":"0x680ae2fffe103e33"}]},{"friendly_name":"livingroom/lights","id":10104,"mem
bers":[{"endpoint":1,"ieee_address":"0x680ae2fffe2f3e23"},{"endpoint":1,"ieee_address":"0x680ae2fffe11ed6f"},{"endpoint":1,"ieee_address":"0xccccccfffed4558a"},{"endpoint":1,"ieee_address":"0x680ae2fffe11fcfb"}]},{"friendly_name":"livingroom/floor_light","id":101
05,"members":[{"endpoint":1,"ieee_address":"0x00158d000361b295"}]},{"friendly_name":"hallway/light","id":10106,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffe11d9f0"}]},{"friendly_name":"pantry/lights","id":10107,"members":[{"endpoint":1,"ieee_address":"0x6
80ae2fffef50db3"},{"endpoint":1,"ieee_address":"0x680ae2fffef5055a"}]},{"friendly_name":"bathroom/lights","id":10108,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffead5e3c"},{"endpoint":1,"ieee_address":"0x680ae2fffead5c39"}]},{"friendly_name":"bedroom/light
s","id":10109,"members":[{"endpoint":1,"ieee_address":"0x680ae2fffe0e90bb"},{"endpoint":1,"ieee_address":"0x680ae2fffe11e3c3"},{"endpoint":1,"ieee_address":"0x680ae2fffe38ce81"},{"endpoint":1,"ieee_address":"0x680ae2fffe11daf7"}]},{"friendly_name":"office/lights"
,"id":10110,"members":[{"endpoint":1,"ieee_address":"0x14b457fffe301910"},{"endpoint":1,"ieee_address":"0x14b457fffe2fee7a"},{"endpoint":1,"ieee_address":"0xec1bbdfffe9f5f8f"},{"endpoint":1,"ieee_address":"0xec1bbdfffe189fde"}]},{"friendly_name":"office/floor_lig
ht","id":10111,"members":[{"endpoint":1,"ieee_address":"0x00158d0003618c46"}]},{"friendly_name":"office/desk_light","id":10112,"members":[{"endpoint":1,"ieee_address":"0x14b457fffe2bd760"}]},{"friendly_name":"balcony/light","id":10113,"members":[{"endpoint":1,"ie
ee_address":"0x680ae2fffe22fbb7"}]},{"friendly_name":14243,"id":14243,"members":[]},{"friendly_name":20861,"id":20861,"members":[]},{"friendly_name":20862,"id":20862,"members":[]},{"friendly_name":20863,"id":20863,"members":[]},{"friendly_name":22898,"id":22898,"
members":[]},{"friendly_name":54744,"id":54744,"members":[]}]'

I have a bunch empty groups I don't even remember creating. So I wanted to clean them up.

mosquitto_pub -t zigbee2mqtt/bridge/request/group/remove -m '{"id": "99"}'

Which results in:

Zigbee2MQTT:error 2020-08-20 19:48:46: Request 'zigbee2mqtt/bridge/request/group/remove' failed with error: 'Group '99' does not exist'
Zigbee2MQTT:info  2020-08-20 19:48:46: MQTT publish: topic 'zigbee2mqtt/bridge/response/group/remove', payload '{"data":{},"error":"Group '99' does not exist","status":"error"}'

I also tried with 99 as a number, same result.

Edit: the docs say I can also do it without a json payload, but that fails with a different error.

Removes a group. Allowed payloads are {"id": "groupID"} or groupID where groupID can be the groupID or friendly_name of the group. Example; request: {"id": "my_group"} or my_group, response: {"data":{"id": "my_group", "force": false},"status":"ok"}.

Zigbee2MQTT:error 2020-08-20 19:49:46: Request 'zigbee2mqtt/bridge/request/group/remove' failed with error: 'message.trim is not a function'
Zigbee2MQTT:info  2020-08-20 19:49:46: MQTT publish: topic 'zigbee2mqtt/bridge/response/group/remove', payload '{"data":{},"error":"message.trim is not a function","status":"error"}'

Edit 2:
Tried with a different group too as 99 is the default group for some of the tradfri remote I have, same result. Also i grabbed the db entry in case you need it.

{"id":18,"type":"Group","groupID":54744,"members":[],"meta":{}}

[Koenkk]

@sjorge you can only remove groups which are also in configuration.yaml. I expect that these groups are not created anymore, therefore I would just manually remove them from data/database.db for now.

[sjorge]

@Koenkk yeah that worked, but they only showed up with the new_api, perhaps we should purge those types of groups automatically when rewriting the database.

[pgScorpio]

Problem

The current zigbee2mqtt/bridge/# topics are a mess. They are not well structured and don't provide a good reply/response mechanism. This makes it hard for other applications to integrate with Zigbee2mqtt. Therefore I propose the revise the structure while keeping the current working (so no breaking change).

Hi Koenkk and others,

I just stumbled onto this thread and I'm glad to see you're working on this topic issue since I also experience a lot of troubles with handling zigbee2mqtt messages in a structured manner, especially since it is "postfixed" but the postfix is not always at the end of the topic !

Maybe I may propose some ideas too ?

First of all take a look at the Tasmota 'prefixed' topic format, since I found this one the most easy to process in a structured manner and so I also use this format (with some small extensions) for all my internal "homecontrol" mqtt topics.

So my standard topic format is:
MSG_TYPE/MSG_SOURCE/DEVICE_TYPE/[/DEVICE_LOCATION[/...]]/DEVICE_NAME[/:VALUE_NAME]

(of course MSG_SOURCE == MSG_DESTINATION for outgoing messages)

Where MSG_TYPE can be:
cmnd (command to control a device)
stat (device status report)
tele (status from sensors)
request (administrative request)
response (response to administrative request)
etc...

and MSG_SOURCE would be i.e.:
zigbee2mqtt
tasmota
google
tradfri
hue
homecontrol
etc...

and DEVICE_TYPE would be i.e.:
(used to determine the possible commands and/or values)
light
multimedia
switch
dimmer
sensor
script
etc...
This could be i.e. 'control', 'admin', 'config' or 'info' for zigbee2mqtt itself

and [/DEVICE_LOCATION[/...]/DEVICE_NAME would be the friendly name.
This could be just 'coordinator' for the zigbee2mqtt coordinator itself. (preferably configurable like any other device.)

finally the /:VALUE_NAME
used to identify non-JSON payloads (note that VALUE_NAME always starts with a colon ':' !)
so I could either have a message with topic: stat/zigbee2mqtt/light/ledstrip with payload {"state":"on", "brightness":50}
or a message with topic: stat/zigbee2mqtt/light/ledstrip/:state with payload "on" (much easier to handle if only one value has changed)

In this way I can simply subscribe to i.e.:
All zigbee2mqtt messages: "+/zigbee2mqtt/#"
All status reports from any source: "stat/#"
All status reports from a specific source: "stat/zigbee2mqtt/#"
All status reports from a specific device type: "stat/+/light/#"
or even all status reports from specific device types in a specific location: "stat/+/light/location/#"
and the same goes for commands etc.

In this way I can also easily split a topic into its parts (I use a subflow in node-red for this) and than dispatch based on these parts:
First find any "/:" and and if it exists split the topic string, anything after "/:" is VALUE_NAME.
Than split the remaining topic into it's parts separated by "/", Now: we should have at least 3 parts:
The first part of topic is MSG_TYPE
The second part of topic is MSG_SOURCE
The last part of topic is DEVICE_NAME
if we have more than 3 parts:
The third part of topic is DEVICE_TYPE
and if there is still anything left between DEVICE_TYPE and DEVICE_NAME this is all DEVICE_LOCATION

Please note that DEVICE_TYPE is very handy to have for structured handling of messages (as it is an indication for which commands/values to check), but it can also be optional since we just can include it in friendly_name too if needed (as I also do with all my tasmota devices)

As I said, just an idea, since it has proven to me it's quite structured and versatile.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Koenkk]

@kirovilya @nurikk I've added permit joining via a specific device. See https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/mqtt_topics_and_message_structure.md#general