Openhab-addons: [RfC] Remote openHAB Binding

[kaikreuzer]

This is still in an ideation state, but I'd like to hear your opinions on that, so that we can decide whether this is a way to move forward or not.

Problem Description

With openHAB 3, users lose the ability to run 1.x bindings on the system. This has already been discussed in depth and the general gist was that it would be very useful to have a separate (old) instance of openHAB running in parallel and being included into the openHAB 3 setup through the event bus. @rkoshak already did a great job in writing a tutorial on how to create such a setup with MQTT.

The problem I see with MQTT is that it requires a broker and it isn't completely straight-forward for users to set up.

Potential Solution

I was thinking about using standard mechanisms of openHAB to achieve the inclusion of remove openHAB instances.
The way I imagine this is by implementing a binding that does the following:

1. Discover openHAB instances through mDNS and show them in the Inbox.
2. Read all available items from their REST API and provide an according channel in the openHAB Thing for each of them.
3. Subscribe to the SSE REST endpoint to get item events and pass them on for every linked channel. Likewise send any local events of those channels to the remote instance through the REST API.

Additional Thoughts

I am not sure how stable the SSE connection is, but my hope would be that it can be used that way. If it turns out to be not good enough, we could think of writing a small bundle for openHAB 2, which would need to be deployed on the remote instance and which could offer a direct TCP connection that could be used. This could then hook into the OSGi event bus in the remote instance directly and would be able to guarantee a fast and stable communication.

We'd have to check whether this binding should support openHAB 1+2+3, 2+3 or only 2.

• Supporting openHAB 2 is imho most important as this is what people have currently running and hence would want to be able to keep that as a "legacy" system in place.
• Supporting openHAB 1 directly would be neat as the main purpose are openHAB 1 bindings, so whoever is still running a 1.9 instance (or would want to use a very small and lightweight remote instance) will find this useful. 1.x is lacking SSE support, though, and I am afraid subscribing through item events through the REST API isn't really feasible, so this might not be an option to put in scope.
• Supporting openHAB 3 will make this binding useful for distributed setups in general (independent of any 1.x-binding support discussions).

Wdyt @openhab/architecture-council, @openhab/core-maintainers, @openhab/distro-maintainers & @openhab/add-ons-maintainers?

[lolodomo]

I like the idea, and the ability to create distributed setups.
It could allow a faster migration to OH3 in case some OH2 bindings are not immediately working in initial OH3 snapshots.
Do you thing it could be possible to have 2 instances running on the same RPI ?
Edit: I answer to myself: not an OH2 instance + an OH3 instance on the same machine as both are requiring a different Java version.

[kaikreuzer]

You can potentially also use different JVMs on the same RPi. I'd think a RPi3 should be able to run two instances, but it is likely not an ideal setup.

[ghys]

Read all available items from their REST API and provide an according channel in the openHAB Thing for each of them.

To stick as much as possible to the thing paradigm, what about letting the administrator of the remote (OH2+) instance set up "virtual things" and "virtual channels" to group related items (bound to an OH1 binding) together, i.e. the item would be linked to both the OH1 binding and the virtual channel/thing.
The remote instance would then appear as a Bridge to the local (OH3) instance; after it's added all the Things defined in that remote instance would be discovered as "Remote Things" - including the virtual things but potentially not only.

I hope it makes sense, and don't know if it's technically feasible, but imho proxying the actual things from the remote instance seems more semantically correct than defining a single "remote instance" thing and pick & choose items from it.

[kaikreuzer]

We don't have such constructs as "virtual things" - things are always directly related to a binding. I wouldn't want to have a huge additional configuration to be done on the remote instances, this is very much against the general idea of the bindings.

Hence I'd prefer to treat a remote openHAB instance just like any other external smarthome system that has an API to connect to and to interact with. If you fear that a single Thing gets too huge, we can easily offer generic things that can be freely defined by the user, just like we have it with KNX.

[ghys]

Point taken about the virtual things. I still would appreciate having the Things on the remote instance autodiscovered and added to the Inbox (if possible), of course it doesn't solve the problem for OH1 bindings but perhaps it would ease the configuration in distributed setup kinds of use cases; for instance, you could have an openHAB instance dedicated to the communication with a particular device that needs a local or dedicated interface, like a serial port, but don't want it to be your primary one. If done correctly it would transparently work out of the box, like if the thing was defined on the client instance, automatically determining the remote linked items to talk to.

For OH1 bindings, or in general when you don't have a remote thing defined, you could indeed build a generic thing and configure channels as you like, only instead of KNX addresses or MQTT topics you would specify the remote item names.

[kaikreuzer]

I was also first thinking about exposing the remote Things to the local instance, but I think this would mean a huge additional complexity. You would have to keep the Thing definition in sync, find a way to deal with unlinked channels, have to figure out how to best define a configuration for them, etc. Even if all that were done, it does not bring and benefit for the initial main use case of 1.x bindings and it also would not support items that have no links, but are used through rules only.

I therefore decided to base the suggestion purely on integrating on item level, which should be fairly straight forward and hopefully possible to implement by the time openHAB 3 is released. I'm open to having an even deeper integration of multiple oh3 instances in the future, but this can imho be done later on and shouldn't have priority.

[TBail]

If someone still uses a 1.x installation than there are reasons for it and he will typically not move to openHAB 3. Based on this assumption: Cut old things like 1.x bindings and focus on the new things.

If the intention is to control distributed OH instances, go ahead and develop a solution

[kaikreuzer]

@TBail See the discussion I have linked above - having an openHAB 1 instances or at least openHAB 1.x bindings supported is one of the main use cases why this feature is desired. And I think it makes a lot of sense, since people can still move to openHAB 3, even if they might have some legacy stuff running that isn't supported anymore.

[rkoshak]

I'll offer some of my experience doing this with rules.

• I've done only minimal testing thus far, but the SSE seems to be pretty stable. I've a third party integration that listens for events on the SSE and publishes commands and updates through the REST API.

• You will have to go out of your way to make sure to avoid infinite loops. I've managed to avoid that by dedicating one instance of OH as the "leader" and the other as the "follower." The "leader" only sends commands to the "follower". The "follower" only sends updates to the "leader". This is configured on an Item by Item basis but I suppose it could be configured on a whole OH instance basis to make is simpler, though that makes the "follower" instance less of a peer. If you allow both updates and commands to flow both directions on the same Item it's far too easy to end up in an infinite loop. The "follower" typically is the OH instance that is directly connected to the device. Any other approach I've tried ends up being either too hard to code or too complex for the user to understand and configure, though my MQTT rules implementation right now support it.

• If there is a way to distribute rules like bindings at some point (a la the Marketplace), it might be simpler to implement using JavaScript JSONDB rules. There would be no requirements to install another language (at least until Nashorn is fully gone from the JRE) and the user can configure it through tags or Item metadata. Or it could be implemented using Rules DSL on the OH 3 Rule engine if that's more likely to always be there, though it'd be nice to have the same rule templates to deploy in both the OH 3 instance and the OH 2 instance. This is just a plan B if a binding turns out to be untenable.

[TBail]

@kaikreuzer I see your argumentation and do not disagree. But playing around with all the new concepts in OH3 bring me to the view that it is a better idea to focus on new things and leave the old ones behind. Having in Mind the german saying: "Lieber ein Ende mit Schrecken, als eine Schrecken ohne Ende."

Nevertheless the developers will decide what are the steps to be done first.

[lolodomo]

@kaikreuzer : regarding your initial proposal, I have 3 questions:

1. With your solution, I don't imagine how it could lead to infinite loop. But my feeling is that the binding should veto the update prediction mechanism for commands coming from the master instance. The item updates will then come only from the slave instance. Do you agree with that ?
2. Can you confirm that the REST APIs we need for this binding were unchanged betwwen all OH versions (OH1, OH2 and OH3) ?
3. I see discussions about REST API now requiring OAuth2 in OH3. Is the API to get all the items concerned by OAuth2 ? More generally, will we have to implement OAuth2 in the binding for slave OH3 instances ? Of course, this is not a problem with OH1 and OH2 instances.

We could support remote things in the binding in a second time but my feeling is that it could add a lot of complexity compared to Kai's proposal.

Is there already someone working on Kai's proposal ? Maybe yourself Kai ? If not, I could be interested to help on that binding. What could be the name of such binding ?

[lolodomo]

Reading again Kai's initial message, I understand we have no real solution with 1.x instances due to lacking of SSE support ? So what can we do for 1.x users ?

I am already thinking where we have useful pieces of existing code. For subscribtion to the SSE REST endpoint to get item events, do we have something already existing in our Java code ? Or more generally a Java example of subscription to a OH REST endpoint ?

[kaikreuzer]

Hey @lolodomo, let me try to answer your questions:

1. Good question, I didn't yet think this through. But assuming the users have already configured the remote instances in a way that they definitely receive updates (either through bindings or by auto-updates), I'd agree that the master instance (or let's switch the wording to "main instance") should veto local auto-updates.
2. Confirmed for OH2+OH3. For OH1 we did not have SSE in place, I mentioned this in my original post above: " 1.x is lacking SSE support, though, and I am afraid subscribing through item events through the REST API isn't really feasible, so this might not be an option to put in scope." It was possible to do long-polling in OH1, so it might be feasible to offer the same functionality by a different implementation, when a OH1 instance is recognized. But this is probably a lower prio and we should focus on OH2 for the start.
3. Authentication is only applied on administrative resources. The /items endpoint is available and there is hence no issue here.

Is there already someone working on Kai's proposal ? Maybe yourself Kai ?

No, I haven't yet started. I am currently working on the expire feature and planned to look at this binding afterwards. But if you are interested in starting here, I'd very much welcome it and will happily contribute to your efforts later on.

Wrt source code to start with, I am currently not aware of any Java implementation consuming openHAB SSE, but this could be a good starting point: https://www.baeldung.com/java-ee-jax-rs-sse#1-jax-rs-client-api-for-sse

[kaikreuzer]

What could be the name of such binding ?

Another good question. Most straight-forward name would be "openHAB Binding", but that could probably also be easily misunderstood (or rather "not understood"). How about "Remote openHAB (remoteopenhab) Binding? Or maybe "openHAB Fabric (openhabfabric) Binding"? I'm open for any good ideas here.

[wborn]

Java implementation consuming openHAB SSE

For an example on how to consume SSE in openHAB you can also have a look at the Nest Binding. It listens to the streaming Nest API events using SSE with this NestStreamingRestClient.

[lolodomo]

"Remote openHAB" looks a good name to me.

[ghys]

I'd offer "openHAB System Link" for the Xbox vibe but that's just me ;)
But in all fairness it does describe clearly what it is :)

[robnielsen]

Oops, accidently closed, I reopened but see it it has been moved from To Do and now is In progress. Can somebody fix it? Sorry....

[lolodomo]

Thank you for the link to the Nest binding, it could really help.

[lolodomo]

In few hours I made good progress. I hope I could publish the binding for review at the end of the week, at least a WIP version.

I would like to make OFFLINE remote OH 1.x instance things (not yet support in the initial version). How to detect OH 1.x server? I get a version from the REST API (3 with my current OH2 server). Was this version equal to 1 for OH1 and 2 or 3 for OH2?

Regarding listening items updates from a SSE link, it seems to work but I am not yet sure if the link will disconnect or not after a certain time? I could reopen the connection if no event was received since 10 minutes for example?

[wborn]

Many thanks for putting some effort into this @lolodomo! :+1:

it seems to work but I am not yet sure if the link will disconnect or not after a certain time?

IIRC the underlying SSE connection should automatically reconnect when it detects it is being disconnected. But it didn't work reliably with the Nest Binding so I added some logic to reconnect it. The SSE connection would not auto-reconnect if for instance you would disable your network adapter, restart your router or un-plug/plug-in your network cable. It could behave better now that the old Jersey SSE implementation has been replaced. But I haven't found time to retest this.

[lolodomo]

In my case, I would have to check what happens when the remote server is shutdown or becomes unreachable after an established SSE connection.

[kaikreuzer]

I would like to make OFFLINE remote OH 1.x instance things

What exactly do you mean by this?

Was this version equal to 1 for OH1 and 2 or 3 for OH2?

Afair, the version field didn't exist at all - so that would be the best way to identify an OH1 instance :-) .

[lolodomo]

I mean that à thing linked to a remote OH1 server will be set to OFFLINE by the binding ... at least until we find a working solution for OH1 (like long polling request as you suggested to replace the SSE request). As I have no running OH1 server, I will probably not implement this solution myself.

I have an important question: what is the syntax for an item name? Is it compatible with the syntax of a channel UID (last part) ?

[lolodomo]

I have now already something working but several remaining TODO tasks to finish the work, like supporting all item types for example.

One difficulty could be the detection of changes of items in the remote instance. I plan to add a channel to trigger a manual refresh / synchronization. Could it be detected simply by using another SSE request?

[lolodomo]

One difficulty could be the detection of changes of items in the remote instance. I plan to add a channel to trigger a manual refresh / synchronization. Could it be detected simply by using another SSE request?

I answer to myself. I see in the core framework core some additionnal events like ItemAddedEvent, ItemUpdatedEvent and ItemRemovedEvent. I assume I receive them through the SSE request. To be checked.

[lolodomo]

I just pushed a WIP version. This is not yet fully finished, few item types are still not yet supported, in particular number items with a dimension.
I tested with a OH2 server and a OH3 server (the topic changed, smarthome vs openhab), both are working. But most of my tests have been done with an OH2 server.

[lolodomo]

I have an important question: what is the syntax for an item name? Is it compatible with the syntax of a channel UID (last part) ?

I found that in our documentation: "The only characters permitted in an Item name are letters, numbers and the underscore character". I have not found the XSD to confirm the documentation. If that's correct, there is no problem to use the item name as channel ID.

[kaikreuzer]

If that's correct, there is no problem to use the item name as channel ID.

Yes, this is correct.
You can find it in the Xtext grammar definition and the ItemsValidator, which disallows the dash, which otherwise is still allowed in the grammar.

[lolodomo]

@kaikreuzer :regarding groups, my feeling is that we have to consider only the groups with a function defined. Standard groups will be ignored as they have no state value. Do you agree?

[kaikreuzer]

Sounds fair. If problematic, I would even say it's ok to ignore all group items, since group items can also easily be created in the main instance.

[lolodomo]

When I have a string "13.4 °C", is there a simple and generic way to build a QuantityType ?
If my value is rather "20 %", the code should build a QuantityType.
If my value is rather "2.5 cm", it should build a QuantityType.
And so one whatever the unit.

[wborn]

You might be able to use the same way used by the Karaf Console commands and persistence such as MapDB:

org.openhab.core.types.TypeParser.parseState(List<Class<? extends State>>, String)

[lolodomo]

@wborn : Thank you for the information.

[lolodomo]

I finished the work, the PR is ready for review.
The first version works on item level.
I will work on thing level in a next version, once this first version is merged.

[lolodomo]

If my local server has only Wifi connection, the discovery code (through mDNS) is discovering my IP (one IPv4 and one IPv6) through the wlan0 interface.
Of course it makes no sense to declare the local server as a remote server !
If possible, I should avoid adding things to the inbox for the local server. How could I detect that ?

If a thing is setup with the REST URL pointing to the local server, I should detect it too and put the thing OFFLINE. Is there something returned by the REST API that allows to identify the openHAB server, a kind of unique ID ?

[wborn]

If possible, I should avoid adding things to the inbox for the local server. How could I detect that ?

It might make sense to also add the OH instance UUID to the response in MDNSAnnouncer.

Is there something returned by the REST API that allows to identify the openHAB server, a kind of unique ID ?

I think you can get the openHAB UUID via /rest/uuid

[wborn]

I see /rest/uuid is secured in OH3. So if it needs to be secured it shouldn't end up in mDNS responses.

[lolodomo]

If possible, I should avoid adding things to the inbox for the local server. How could I detect that ?

It might make sense to also add the OH instance UUID to the response in MDNSAnnouncer.

Ok. Can someone do that ? I have no dev environment setup for the core framework...

Is there something returned by the REST API that allows to identify the openHAB server, a kind of unique ID ?

I think you can get the openHAB UUID via /rest/uuid

Ok I see it.

Unfortunately this API is protected. @ghys : could we make the /rest/uuid accessible to any user ?

@wborn : in my code, is there a simple way to retrieve this UID from the local server, I mean a simple method to call ?

[lolodomo]

I see /rest/uuid is secured in OH3. So if it needs to be secured it shouldn't end up in mDNS responses.

Why should it be secured ? It provides just an ID...

[wborn]

I mean a simple method to call ?

InstanceUUID.get()

It provides just an ID

I don't know the exact reason, IIRC @ghys secured it.

[lolodomo]

If this UID was in the MDNSAnnouncer, I could even use it as representation property. This will allow to have only one thing added to the inbox, rather than one for each interface and IP.

[wborn]

Even if it is added now to MDNSAnnouncer, you cannot use it for OH1/OH2 instances.

[lolodomo]

Waiting for an answer from @ghys . In case we can unsecure this API, I can make the discovery of the binding better and avoid that a user defines a remote thing for the local server.

[wborn]

Probably the add-on will only run on OH3, so you want to make sure you don't detect your own OH3 instance. :wink:

[lolodomo]

Even if it is added now to MDNSAnnouncer, you cannot use it for OH1/OH2 instances.

You're right. But I can call the /rest/uuid for OH2 and set it as representation property.

[lolodomo]

Probably the add-on will only run on OH3, so you want to make sure you don't detect your own OH3 instance. 😉

Yes, of course.

But there is 2 different subjects:

• Detect a thing is not linked to the local server (in practive, I need a solution working on OH3 as the binding will run only on OH3)
• Improve the discovery to avoid multiple entries in the inbox for the same remote server. This concerns OH2 and OH3 servers.

[wborn]

You could also have OH2 servers where the /rest/uuid resource is also secured when users run it behind their own proxy. I have it setup like that myself.

[ghys]

@ghys secured it.

I plead not guilty!
I think the rationale is here: https://github.com/openhab/openhab-core/pull/1570#issuecomment-672193949
Since it identifies an openHAB instance uniquely, it may be considered sensitive by some wrt tracking, so that's why's it's in its separate bundle and not even a mandatory endpoint.

[wborn]

So it's conflicting requirements, where not being able to be tracked... results in not being able to properly distinguish between OH instances in the Inbox. :wink:

[lolodomo]

I plead not guilty!
I think the rationale is here: openhab/openhab-core#1570 (comment)
Since it identifies an openHAB instance uniquely, it may be considered sensitive by some wrt tracking, so that's why's it's in its separate bundle and not even a mandatory endpoint.

Thank you @ghys ;)
@kaikreuzer : is it somlething we could change ?

[wborn]

Detect a thing is not linked to the local server (in practive, I need a solution working on OH3 as the binding will run only on OH3)

Did you also try if you can detect all local IP addresses and filter out responses coming from those IPs?
I.e. similar to the logic in NetUtil.

[lolodomo]

Doing more tests, I discovered two problems I have to analyze:

• Changes of dynamic state options are not detected
• State updated twice in a very short time and item with a visibility rule in sitemap => visibility sometimes wrong in OH3

[kaikreuzer]

@kaikreuzer : is it somlething we could change ?

I wouldn't really - as explained in the linked issue, it is an optional bundle, so you should not rely on its existence.
I'd in anycase prefer the IP address to be the distinguishing parameter and also the representation property (since I believe there is a much higher chance for people to know the IP address than knowing the UUID). You should be able to filter out the own instance by looking up the local IPs - or am I missing some issue here?

[lolodomo]

You should be able to filter out the own instance by looking up the local IPs - or am I missing some issue here?

For the discovery, yes, that would be the solution. I just have to search how to get the local IPs :)

For the initialization of the thing, as I just have an URL, this is not obvious, the URL does not necessarly contains the IP address when defined by the user (it could use a domain name). But if the URL contains one local IP, yes I could at least detect these cases.

[kaikreuzer]

My naive assumption was that there's only one IP broadcasted via mDNS and we could check it through NetworkAddressService.getPrimaryIpv4HostAddress() - but it seems it's not that simple...

I remember from the past that I always saw my IPv4 address in the mDNS broadcast. Just had a look in my mDNS browser and I just see 16 (!) openHAB instances with IPv6 address only - I am a bit puzzled 🙄 .

But afaik, the mDNS announcement should always contain the IP address so it should be straight forward to connect to it (after all, that's what the Android and iOS are doing as well). But even if it were a hostname, that should be ok to use as well, right?

[lolodomo]

As the initial version of the binding should be merged very soon, my intentions for a second version are:

• Automatic discovery of remote server things; they will be attached to the bridge thing representing the remote server.
• Matching the local thing status to the remote thing status
• Automatic creation of trigger channels on the local thing matching the remote channels of kind "trigger"

All this should allow to use the remote things in local rules through the local things.

This should be doable for the end of the week, I believe.