Zigbee2mqtt: [PETITION] zigbee2mqtt as embeded library

[kirovilya]

Hi Koen, in first I want to thank you for the great improvement of the zigbee2mqtt project and the whole zigbee theme!

In view of the big refactoring #1888 of zigbee-herdsman and zigbee2mqtt, I want to remind you of some of the suggestions made earlier #873 and in private talks.

I will probably express the opinion of many zigbee-nodejs-developers: is very necessary the ability to use zigbee2mqtt as a library (not only zigbee-herdsman) and run a controller with replacing the extension of working with MQTT to a own-extension!
If this is possible, then many projects on nodejs will be able to work with zigbee without an intermediate mqtt-layer.

Of the projects that I can recall are:

• iobroker.zigbee https://github.com/ioBroker/ioBroker.zigbee
• node-red-contrib-zigbee https://github.com/hobbyquaker/node-red-contrib-zigbee
• homebridge-zigbee https://github.com/itsmepetrov/homebridge-zigbee
• mozilla-iot https://github.com/mozilla-iot/zigbee-adapter

This will allow them to work with a common base of devices and on a single code base.

I would like to get this opportunity and adapt to it with the release of a new version of zigbee-herdsman and zigbee2mqtt.

Thank you very much!

[kirovilya]

Or move part of the functionality to zigbee-herdsman :)

[Koenkk]

I prefer to move parts to zigbee-herdsman, but what exactly more should be there? As with the zigbee-herdsman refactor most of the parts did already move there.

[kirovilya]

I believe that extensions not related to HomeAssistant and Mqtt may be moved to the zigbee-herdsman controller. For example:

• Zigbee - controller with handlers and queue
• deviceAvailability - for ping devices and checking availability
• deviceConfigure - to apply device configuration
• deviceReport - for setting up reports
• livolo - for livolo devices
• Responder - for time cluster
• SoftReset - for regular reset
• Xiaomi - for ping

[Koenkk]

Overview:

Zigbee - controller with handlers and queue
-> complete queue has been pushed down, zigbee only contains zigbee2mqtt specific logic now: https://github.com/Koenkk/zigbee2mqtt/blob/zigbee-herdsman-controller/lib/zigbee.js

deviceAvailability - for ping devices and checking availability
-> Candidate

deviceConfigure - to apply device configuration
-> Candidate

deviceReport - for setting up reports
-> Candidate

livolo - for livolo devices
-> This extension is unclear to me, it seems that livolo devices still don't work properly (?)

Responder - for time cluster
-> moved to zigbee-herdsman

SoftReset - for regular reset
-> I may want to get rid of this, does this even work?

Xiaomi - for ping
-> Removed, I don't think it's needed

[kirovilya]

ok, I agree

[dzungpv]

Separate the core from MQTT is a good idea, like the ideas we can make component for home assistant directly, not over mqtt middleman and improve speed. This idea i have wrote from the beginning of this project on ha forum. And after version 1.6 i hope mqtt only an "Communication add on", now without mqtt, we could not do anything with ZNP.

[Koenkk]

@dzungpv your comment made me realize that we should actually look at things the other way around.

Instead of having multiple projects focusing on one communication protocol, we should have one project focusing on multiple communication protocols. Meaning that it should be able to talk e.g. MQTT, Home Assistant native API, iobroker, nodered, etc.

EDIT: this would e.g. also take away the work for @kirovilya to constantly migrate stuff, thus he can focus more on only the iobroker part.

[dzungpv]

@Koenkk yes, and zigbee2mqtt may have an API (Json, Xml../) and lib for native application using by third party and developer add to their exist application, if you do that it may take a difference name with out the mqtt suffix. And if you could separate Z-stack low level specific code, it could work with other stack from Silabs or NXP, it is posible because they all follow same standard from Zigbee Alliance. I am sure this is the goal of some guy make Zigbeer

[Koenkk]

@dzungpv z-stack specific code has already been separated in zigbee-herdsman. Adding support for conbee, xbee, etc. is just a matter of implementing this interface: https://github.com/Koenkk/zigbee-herdsman/blob/master/src/adapter/adapter.ts

[manup]

@dzungpv z-stack specific code has already been separated in zigbee-herdsman. Adding support for conbee, xbee, etc. is just a matter of implementing this interface: https://github.com/Koenkk/zigbee-herdsman/blob/master/src/adapter/adapter.ts

This is really awesome! Have you considered to let the interface be even more simple by just using the underlying APS request, confirm and indication primitives?

That way most of the interface functions like nodeDescriptor(), sendZclFrameNetworkAddress(), bind(), unbind(), etc. could be done directly in TS with a default implementation.

[Koenkk]

@manup nice to see you here!

I've currently modelled it biased on the Z-Stack API, but after reading the Conbee and XBee API documentation I also realised this.

[manup]

I'm watching the project since a while now :)

Yes the XBee API mode is quite similar to ConBee/RaspBee API, both provide the APS layer access.

https://www.digi.com/resources/documentation/DigiDocs/90002002/Default.htm

• Transmit Request Frame (0x10, 0x11) request
• Transmit Status Frame confirm
• Explicit Rx Indicator frame (0x91) indication

The Ember Silabs EZSP serial protocol looks more comprehensive but seems to have a APS interface too, I think this is used by the Telekom Zigbee USB dongle:

https://www.silabs.com/documents/public/user-guides/ug100-ezsp-reference-guide.pdf

What I'm missing in the current adapter interface is some kind of callback handler for incoming packets without a prior request — like an APS indication which might carry an ZCL frame or a ZGP frame handler.

[kirovilya]

I am very glad that you liked the idea.
Yes, it would be great to have a multi-protocol zigbee-center. Moreover, both from the client side and from the side of the implementation of zigbee-adapters (as for example, https://www.openhab.org/addons/bindings/zigbee/#supported-coordinators).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Hedda]

@dzungpv your comment made me realize that we should actually look at things the other way around.

Instead of having multiple projects focusing on one communication protocol, we should have one project focusing on multiple communication protocols. Meaning that it should be able to talk e.g. MQTT, Home Assistant native API, iobroker, nodered, etc.

EDIT: this would e.g. also take away the work for @kirovilya to constantly migrate stuff, thus he can focus more on only the iobroker part.

@Koenkk Is the idea then that it could also act as a replacement for the zigpy library by the ZHA integration component in Home Assistant so you could use ZHA's UI to control zigbee-herdsman?

https://www.home-assistant.io/integrations/zha/

Basically making zigbee-herdsman an optional direct replacement for zigpy and its dependencies?

https://github.com/zigpy/zigpy/

Maybe unstale this or does the discussion belong in zigbee-herdsman instead?

[Koenkk]

@Hedda I think the idea is good however the biggest problem I see is that zigbee-herdsman = javascript, while home assistant is python.

[Hedda]

I guess that depends on what 'native' bridge protocol ZHA would use on the client-side to talk with zigbee-herdsman, as you could if as one separate server-side 'bridge' project focusing on multiple communication protocols as you suggested above. The 'native' bridge protocol here just referring to something more 'native' and direct than MQTT so that you do not have to manage a non-native gateway using a third-party UI.

That is, it would be very nice to be able to use Home Assistant with ZHA's UI as a front for zigbee-herdsman using some kind of a 'native' protocol but still be able to run zigbee-herdsman remotely. Like for example running Home Assistant with ZHA in a virtual machine on a server in the basement and running zigbee-herdsman with a seperate 'bridge/gateway' software.

@Hedda biggest problem I see is that zigbee-herdsman = javascript, while home assistant is python

FYI, Home Assistant is already planning on migrating its Z-Wave from directly using OpenZWave to a backend based on qt-openzwave which is not either written in Python but instead is in C++

• https://github.com/OpenZWave/qt-openzwave

qt-openzwave is a QT Wrapper for OpenZWave and contains ozwdaemon - a service that allows you to remotely manage a Z-Wave Network via ozw-admin (or connect to a MQTT Broker if that is what you want).

You can read Home Assistant developers arguments here why they are moving to such architecture design

• https://community.home-assistant.io/t/the-future-of-z-wave-in-ha-qt-openzwave/148541/18

Note though that the information in that initial post is a little outdated as I understand that they are no longer planning on using that python-openzwave-mqtt by @cgarwood but instead use qt-openzwave directly somehow. Perhaps @cgarwood would care to explain it better?

Perhaps port qt-openzwave or use its concept to make a new bridge wrapper for zigbee-herdsman that could support multiple communication protocols as you suggested above, again not just MQTT.