Smarthome: [RFC] Mqtt binding(s)

[davidgraeff]

I have the following binding proposals and would like to have feedback if those are actually a good thing to have in ESH.

Broker configuration binding

_Proposed package name: MqttBroker_ (I am open for name suggestions)

At the moment it is possible to configure brokers in a textual way. Without intensively watching the log file, there is no way to tell if the provided configuration is correct and if the connection to the broker have been established, neither via the GUI (PaperUI) nor machine readable (REST interface).

I propose a binding to make the configuration of brokers more intuitive and just let the user add them as things (and sync them to the io.transport.mqtt MqttService). Features would be:

• Rich status feedback
• Auto discovery of brokers (mqtt-sn, zeroconf, best effort subnet scan)
• Configurable reconnect time (this is not possible via textual configuration yet)
• Certificate and public key pinning for ssl connections

Personally I would also like to embedded a simple mqtt server like Moquette and list this in the discovery as an option to the user. Setting up a working Mqtt Broker would be just a button click away for ESH users of this binding.

Mqtt Things

_Proposed package name: MqttThings_ (I am open for name suggestions)

At this very moment there exist no means to actually connect mqtt topics to ESH things. There are a lot of commercial and non-commercial (Owntracks, EasyESP) Mqtt products out there who deserve their own binding.

But for a start I propose a generic Mqtt binding where you can bind topics to things (very much like the Openhab 1.x Mqtt binding, but modern). This should be part of the ESH extensions for 1) it to serves as a implementation examples for other Mqtt binding implementers and 2) allow the ESH consumers to easily interact with one of the core IoT protocols. The following features would be part of the binding:

• List all configured brokers of the io.transport.mqtt MqttService and let the user add them as bridges.
• Manually add a thing (Switch-, Contact-, Number-, Range-thing) and bind it to a Mqtt topic state and command.
• Auto discover topics (if retain messages==true)

Mqtt2Eventbus

_Proposed package name: Mqtt2Eventbus_ (I am open for name suggestions)

It might be a good idea to have a way to synchronize two or more instances of ESH (forum post). Mqtt could be a perfect protocol to archive this.

I am looking forward to some feedback.

Cheers, David

[marcelrv]

Some time ago I gave this also a try. Unfortunately never gave it the time to clean it up and make it properly working.
https://github.com/marcelrv/smarthome/tree/mqtt

Indeed as proposed the connection to the broker to be configured as a bridge.
What I liked very much was to have multiple topics per thing. Hence in the thing, you could basically give the topics you wanted to subscribe to and it would discover all things that matched the desired topic. Than, assuming you have your topics organised, your thing will discover all channels belonging to it. (e.g. /thermostat/mode, /thermostat/setTemp //thermostat/temperature would be grouped in single thing with 3 channels)
My main challenge at the time was that I wanted to be very user-friendly switch between various item types, which did not work out well.

Anyway.. my suggestion: allow for multiple channels per thing, each channel representing a mqtt topic

[davidgraeff]

@marcelrv It would indeed be handy to have multiple topics grouped in a single thing. But I have no idea how to present a user-friendly configuration with means of the nowadays PaperUI. For now I would go with a direct 1:1 relation of topic<-->thing with one channel. The reasons are:

• Other GUIs like HabPanel or ClassicUI with sitemaps only care about channels and it doesn't matter if channels are provided by one or more things.
• Easier to implement. No dynamic channel creation, no custom ThingTypeProvider. The type ("string","number") / representation ("range","switch") of the statically provided things is selected by the user and we don't have to implement obscure heuristics.
• We can always extend the binding with an "advanced" thing later.

I plan on adding more specific Mqtt bindings to OpenHab for EasyESP, Homie and other "established" Mqtt IoT specifications. Ideally your Mqtt things are following one of those.

Cheers, David

[ThomDietrich]

Hello @davidgraeff,
this is far from a complete response to this discussion. While reading through this proposal I agreed, that simple [one topic; one thing; one channel] combinations would be sufficient for most use cases. However looking at how most devices communicate, one aspect of a device is all too often divided down to two MQTT topics (status, command). That should be covered by the binding... I'm here to tell you that I'm thrilled to see that you've already covered that aspect in the open PR. One 🏅 for you sir 😉 Keep up the great work!

@marcelrv it seems your thought goes into the direction of the specific MQTT conventions, which are trying to specify how topics are to be found and handled. Wdyt?
@davidgraeff I've got some experience with Homie and was involved in discussions about improvements. Maybe you can find some edge cases there or want to get involved in the discussions. Best!

[ThomDietrich]

@davidgraeff I want to add a third de-facto MQTT convention to the list: https://github.com/arendst/Sonoff-Tasmota/wiki/Commands

[htreu]

Hi @davidgraeff, I´m about to review your mqtt bindings but want to discuss some structural issues before:

• Its nice to have a separate bundle for the moquette broker, nice!
• Why did you separate the broker and things binding? With only the broker binding available we are able to define broker connections but not be able to do anything with them. Instead it would be nice to have this one binding where broker connections are the bridges and things are configured for this bridges.
• In addition (just like the structure of the KNX binding) it would be nice to have things only as a structural element (w/o any configuration) but then add many channels from different channel-types. The configuration which is currently on thing level would then be done on the channels.

I´m aware this will change the direction slightly. But openHAB1 users will have a xtext config, for configuration in PaperUI we need a new feature for adding arbitrary channels to a thing.

for reference: the discussion for the KNX binding: https://github.com/openhab/openhab2-addons/pull/21#issuecomment-258159717

wdyt?

[htreu]

btw, here is an old (but gold) thread about mqtt in ESH. unfortunately the original author did not file a PR. But the ideas are still valid: https://www.eclipse.org/forums/index.php?t=msg&th=942318&goto=1704186&#msg_1704186

(just in case you guys did not know of it)

Maybe some of the ideas can be recycled here. if not now maybe later.

[davidgraeff]

@htreu

Having two bindings

To me it makes sense to separate the configuration of broker connections and using Mqtt.

• Until now it was split as well: The textual configuration file + The OH1.9/Mqtt binding
• The configuration binding can be kept simple and focused. With your comments you haven proven that it's not easy to find consensus on how to map Mqtt topics to the ESH world.
• Some people may only want to configure brokers via a UI but keep their text-files for actual rules and item definitions. Some other people may want to configure their brokers textual and only want to use the MqttThings or similar binding with the PaperUI.
• The MqttThing binding is really just for a very generic use and might not even be used that often. Usually someone defines a Mqtt convention / protocol on top of Mqtt and this will be implemented in a separate binding anyway (with proper auto discovery, thing/channel mapping etc).
• We want to be at least as good as HomeAssistant and they separate configuration and usage too and it works quite well for them.

Things only as a structural element

The ESH UIs do not provide all means to realize this idea I think. With the upcoming Homie binding and other Mqtt convention bindings, I recon this MqttThing binding will only be used for very basic tasks anyway. It will be for example to just read a simple switch or sensor value, but nothing multi channel like.

[davidgraeff]

The forum topic talks about a "DeviceTemplate" concept. That's a nice idea to add auto-discovery. It's really sad, that Mqtt does not have the resource/topic discovery embedded in the standard, like in CoAP.

Personally I would vote to add this in a separate PR, because it needs discussion, and the KNX guys haven't finished and implemented their ideas yet.

[ThomDietrich]

@davidgraeff I'm with you with the separate broker. It will support all the future MQTT bindings and can be the base for other ideas like your MQTT-Eventbus.

One thing that confuses me is your constant reminder that parts might be configured via Ui OR config file. Please keep in mind that both should be possible interchangeably. So far all (most) addons/bindings can be configured via UI, utilizing auto discovery and higher logic built into the UI, or via configuration files following the information in the README files. We should hold on to that concept! I have to agree that at some point the latter might not be that popular anymore but that point was not reached yet and we should not break with the freedom users have in configuring their system currently!

Best! Thomas

[davidgraeff]

@ThomDietrich : I did not say that ^^. I said: Users may use configuration files OR GUI, depending on their preference. Personally I like to avoid text files in the 21st century for my Smarthome. And it should always be possible to use a GUI as well for all things that right now still need config files.

[hakan42]

With me, it is the complete opposite. A server system should be able to die and be recreated without human intervention. Best, of course, if the configuration files for this are human readable.

It should always be possible to use config files for all things that still needs manual GUI interaction 😛

[davidgraeff]

Using a GUI or text files has exactly nothing to do with backup and restore process. The jsondb with the GUI generated things and items can be backuped and restored fine :)

[kaikreuzer]

@davidgraeff Sounds all very reasonable to me (the split into two bindings). I assume from a Karaf feature perspective, we would not package the "mqtt bridge bundle" as a feature of its own, but only have a "Generic MQTT Binding", "Homie Binding", etc. as features, which each include the mqtt bridge bundle as a dependency (and thus it will be automatically installed), correct?

[davidgraeff]

@kaikreuzer Yes, sounds reasonable.

[htreu]

@davidgraeff Thank you very much for the great discussion! I get your point in separating broker configuration and things binding. I agree this makes sense and speaking in openHAB terms the broker binding might then be bundled (feature wise) with specific MQTT based bindings to provide the bridge(s). (sorry i´m late, this was open in my browser the whole weekend...)

Regarding the MQTT Thing binding the generic nature is another argument towards a more generic structure of things and channels imho. Would that change a lot from an implementation pov?
As you already pointed out, the ESH UIs are not yet capable of the suggested structure. But with KNX binding around the corner we need to provide this features anyway.

[davidgraeff]

@htreu If you follow the OH forum, there is another Mqtt setup question every week (sometimes more than one). I'd like to have the Mqtt Things binding merged rather soon that later. The current implementation works and has a test suite.
I thing we should add a thing with dynamic channels to this binding as soon as the UI is ready. We can hide the "old" things so that new users would only use the dynamic channels and we are still backwards compatible.

[kaikreuzer]

@davidgraeff I do not think that refactoring the binding to have a single Thing type and a list of channel types to choose from will be much effort and I believe it would be for the better since

• Users can structure their Things nicely and do not have the huge overhead where there is a Thing for every item.
• Users will anyhow use textual configuration for those, so it isn't really an issue that the Paper UI does not yet support adding channels.
• We won't need a breaking change later on. With the current Thing types, which would be removed in future (keeping "wrong" stuff eternally is not an option!), people would be forced to change their definitions.
• We follow the modelling guidelines and do not have a bad example in the repo, which others might copy.

[davidgraeff]

I agree with most of your points except that people will use textual files anyway. I'd like to use my binding as well and I don't use a single text file for thing and item configuration :D So I'd prefer, if we can extend paperui in the next weeks, and on my part I will adapt the binding accordingly.

[kaikreuzer]

Sounds like a plan :-) I anyhow forgot to mention the last benefit:

• put more pressure on us to finally implement this missing feature in the Paper UI 😆

[htreu]

I can already feel it :-D

[maggu2810]

I assume from a Karaf feature perspective, we would not package the "mqtt bridge bundle" as a feature of its own, but only have a "Generic MQTT Binding", "Homie Binding", etc. as features, which each include the mqtt bridge bundle as a dependency (and thus it will be automatically installed), correct?

Shouldn't we have a ESH Karaf feature for the MQTT bridge bundle, so e.g. openHAB and other ones could depend on that feature instead of starting to depend on the ESH bundle.
It has been the concept till now, that no user of ESH features needs to solve the dependencies of ESH itself but could rely on the features.

[kaikreuzer]

Yes, you are right. My main point was that there won't be a separate "openHAB binding feature" for the bridge bundle - I indeed didn't express myself correctly.

[davidgraeff]

@kaikreuzer Any progress here? I can't help with PaperUI unfortunately (Angular is the most unfriendly inconsistent web-developer framework in my opinion), but might help with the core if changes are necessary there too.

[kaikreuzer]

@davidgraeff Sorry, I didn't see any action on my side - I was actually waiting for the refactoring of https://github.com/eclipse/smarthome/issues/3812#issuecomment-320660894 to be done.
It seems there is some misunderstanding, when you were waiting at the same time for a new feature in the Paper UI for which there hasn't been even an issue so far (@htreu now just created #4142 for it).
Anyhow, there is no dependency for your feature on #4142 - as mentioned above, everything will work textually right from the start and as soon as #4142 is implemented, no changes will be required in the MQTT binding to benefit from it.

[davidgraeff]

@kaikreuzer: I thought at least the thing XML definition needs adaption. It should allow me to define possible but not mandatory channels.

[kaikreuzer]

You're right, such an addition makes sense - but this is completely unrelated to the Paper UI (despite the fact that @htreu listed it as one feature in #4142). Probably best to address this in an independent issue in the core framework: https://github.com/eclipse/smarthome/issues/4146.

[davidgraeff]

I will close this issue now, because we have individual PRs already for discussion.