Hono: Implicit registration of devices connected via gateways

[mhemmeter]

As of today all devices that connect to Hono via a gateway are supposed to be explicitly registered in the device registry with the via flag pointing to the gateway device. In this setup only gateway devices are authenticated and not the devices themselves. In order to simplify the management of devices in such a setup an _implicit_ registration of devices behind a gateway would be beneficial form my perspective.

At first glance from security perspective an _implicit_ registration should be possible as after successful authentication of the gateway device a trust relation has been established.

For example such a feature would help in case devices connect via Bluetooth to a gateway that then communicates with Hono. Due to the features in Bluetooth devices may show up in a deployment dynamically and it may not be possible to "pre-register" the devices. It would be easier in a such a dynamic scenario if Hono registers these devices _implicitly_ upon connection instead requiring the gateway to invoke the Device Registry Management API for explicit registration.

[mhemmeter]

Remark: An interesting question is if and how an implicit registration can work in a setup where Eclipse Hono is used in combination with Eclipse Ditto (see Eclipse IoT Packages). In such a setup devices must not only be registered in the Eclipse Hono device registry but also as a Thing in Ditto.

[geglock]

Regarding the combination with Eclipse Ditto (via Eclipse IoT Packages or somehow else), a proposal could be to emit some kind of "welcome" message to the north-bound application for each implicit registration.
If such a message could be defined freely per tenant including placeholder replacement, then it could be a "Thing Create" message - see example here: https://www.eclipse.org/ditto/protocol-examples-creatething.html
This way each implicitly registered device on Hono level would also trigger the creation of a respective Thing on Ditto level.

[sophokles73]

I understand the use case and I also see the benefit of being able to register such devices on the fly. Devices being registered in this way will never be able to connect directly to a protocol adapter, though, because we would not register any credentials for them. This is consistent with the fact that the gateway device authenticates to the adapter and then acts on behalf of the edge device(s).
The biggest challenge in this context will be the management of the (edge) device identifiers.

I can imagine several approaches:

1. The gateway generates an identifier for the edge device to be registered and uses it when uploading data on behalf of the edge device. The protocol adapter and registry implementation determine that the device doesn't exist (yet) and then provision the device on the fly.
2. The gateway uses a distinct resource on the protocol adapter to register an edge device. The adapter and the registry implementation generate an identifier, register the device on the fly for the gateway and return the newly created identifier, which the gateway can then (locally) associate with the edge device.

It would also be possible to let the gateway use the gateway specific resource for uploading data on behalf of a new edge device and include a marker (header/query param etc) to indicate to the adapter that this data is for a new edge device which should be provisioned on the fly. However, it might be difficult to return the generated device identifier to the gateway in this case. In MQTT for example there is no response message which could be used to convey the information.

Regarding the additional provisioning of the device in Ditto, we should be able to use the same mechanism that we already use for devices that are being auto-provisioned based on a client certificate. See #1296 for a discussion of options. As part of that discussion we explicitly decided against emitting an event in that case for the time being. In any case, I would consider it an (optional) feature of a device registry implementation.

[sophokles73]

@calohmn @dejanb @ctron WDYT?

[avgustinmm]

If the device will have different id (chosen by gateway, adapter or registry) then it will be nice if the device has option to provide its own/internal id (if exists) in order to be incorporated in the generated id.
This way user could somehow link the device id in hono with the own device id.
If this hono id generation follows a pattern (e.g. :) then event this link could be made automatically be tools..

[calohmn]

FMPOV, the first approach about (the one where uploading an event/telemetry message for an unknown device triggers creation of the device) seems to be the one we should be aiming for. This would avoid having to define a separate new resource for each adapter and would also be a very intuitive solution.

I imagine, the right for a tenant to automatically create devices would have to be explicitly defined for such tenants, so that not every tenant is allowed to do that.
I could also imagine requiring an extra createIfMissing parameter for the event/telemetry requests to explicitly enable automatic device creation if needed.

Open questions:

• I guess there could be the scenario of a device being used with multiple gateways. First the device is connected with gateway A and gateway A triggers creation of the device in Hono. Then the device connects to gateway B. When gateway B wants to send an event/telemetry message on behalf of the device, that would normally get rejected because gateway A had this device be created with its own id in the device's via property. If gateway B has the same right to automatically create devices in Hono and also uses a createIfMissing parameter when sending the event/telemetry message, the protocol adapter could trigger the modification of the device entry to then also contain gateway B in the device's via list.
• Gateway groups: Support an extra request parameter with a gateway group to let it be used in the created device in a viaGroups property instead of the via property with the gateway id?
• Should there be support for letting automatically created devices also be _removed_ again from the device registry, triggered by some kind of request to the protocol adapter? Or automatically created devices get some lifespan after which they are deleted. If the device is still needed after that period, it can just get created again the same way it got created before.

[sophokles73]

I imagine, the right for a tenant to automatically create devices would have to be explicitly defined for such tenants, so that not every tenant is allowed to do that.

FMPOV this should even require explicit authorization at the gateway level.

I guess there could be the scenario of a device being used with multiple gateways. First the device is connected with gateway A and gateway A triggers creation of the device in Hono. Then the device connects to gateway B.

I guess the desired behavior depends on whether a physical device should always be represented by a single logical device ID in Hono. If the gateway generates an ID for the device then gateway A might generate a different ID than gateway B. Following @avgustinmm 's comment above, we would always end up with different device IDs for the same physical device, depending on which gateway the device is connected via. Maybe @avgustinmm and @mhemmeter can comment on whether this would be acceptable for the envisioned used cases and what those use cases are in the first place?

Should there be support for letting automatically created devices also be removed again from the device registry.

If we create device IDs scoped to the gateway IDs then I'd rather go for the limited life span approach instead of relying on the fact that a gateway would (eventually) unregister a device ...

[e-grigorov]

Usually, the device network provides enough information for the device identifier. For example, in Blueooth, it can be the device address. Of course, the different gateway vendors can use different identifiers.

The main point here is that the gateway knows the device and can recognize it with a given identifier. It'll be useful to reuse this identifier(or similar one) in the Hone device registry. In this way, the device can be easily found.

If we create device IDs scoped to the gateway IDs then I'd rather go for the limited life span approach instead of relying on the fact that a gateway would (eventually) unregister a device ...

From the user point of view, these devices are not so different. They just cannot speak Hono protocol but they are used as any other device. Such limited life will require a periodic "keep me" message and will complicate the provisioning.

[sophokles73]

For the sake of simplicity I suggest to leave the roaming use case out of this issue for the time being. Instead, let's focus on how a gateway can register a device on the fly. Consequently, unregistering such a device should also be of no concern right now.
What I can imagine for Hono is the following:

1. A new device is being connected to a gateway that is already registered in Hono.
2. The gateway publishes a telemetry message or an event on behalf of the new device to one of Hono's protocol adapters.
3. The protocol adapter tries to determine the registration status of the device and sends a corresponding request to the Device Registration service.
4. The Device Registration service implementation fails to assert the device's registration because the device i snot registered (yet).
5. The Device Registration service checks if the gateway is authorized to register device for its tenant on the fly. This might be done by means of a property in the gateway's registration info. If the gateway is not authorized, then a 404 status is returned to the adapter and the adapter rejects the message published by the gateway. Otherwise, the registration service creates a record for the new device using the device ID provided by the gateway and sends a response to the adapter that contains a 200 code and contains a property in the defaults which indicates that the device has been registered on-the-fly, e.g. hono_registration_status=new.
6. The protocol adapter creates the downstream AMQP message from the gateway's published data and includes the hono_registration_status property in the downstream message. The adapter then forwards the AMQP message downstream.
7. A downstream application can now determine that the device has been newly created in Hono and can act accordingly. For example, Ditto could use this flag to trigger creation of a new Thing for the device before processing the message payload.

This mechanism should work regardless of the payload of the message published by the gateway. However, the gateway might also publish a specific event in order to indicate the device's capabilities as discussed in #1618. In that case Ditto could use the information from the payload to create a typed Thing that reflects the capabilities of the device already. This, however, is completely optional.

@calohmn @mhemmeter @avgustinmm @e-grigorov @geglock @fkaltner WDYT?

[mhemmeter]

The approach makes sense to me. From a requirements perspective we can exclude the roaming use case and add it later on (if necessary at all). Supporting this new feature in MQTT Protocol Adapter has highest priority for me (compared to the other protocols supported by Eclipse Hono).

[fkaltner]

I think this is doable. Can you assign this issue to me?

[thjaeckle]

I wonder if this _hono_registration_status_ property should ever be transported via a QoS 0 telemetry message - if this message never reaches the downstream application, a corresponding thing will e.g. never be created in the downstream connected Ditto.

So FMPOV for telemetry messages there should be a separate QoS 1 event generated (ideally published before the causing telemetry message is published) in order to ensure that it eventually reaches a downstream connected consumer.
Or, alternatively, the telemetry message could be upgraded to a QoS 1 event message with message persistence, if this makes sense?

If the message from the device already was an event message, the information could be "piggy-backed" in the same QoS 1 event message via the mentioned property.

[fkaltner]

Wouldn't Ditto favor the "specific event" case Kai outlined above:

However, the gateway might also publish a specific event in order to indicate the device's capabilities as discussed in #1618. In that case Ditto could use the information from the payload to create a typed Thing that reflects the capabilities of the device already.

?

But in general you are right: for telemetry sent with QoS 0 there's a risk that the downstream application may miss that message. However, to my understanding, telemetry messages are forwarded by Hono having QoS 1 always (see: https://github.com/eclipse/hono/blob/cba0221c43ec2b93bfb436d438d8542f81f61022/client/src/main/java/org/eclipse/hono/client/impl/TelemetrySenderImpl.java#L109).
So I think the case you described shouldn't happen.

[thjaeckle]

Wouldn't Ditto favor the "specific event" case Kai outlined above:

Ah, the gateway would have to publish that event? Sure, that would also work.
I just thought that when Hono already "knows" that a new device connected via a gateway was created on-the-fly, an event could be created and published by Hono itself.

However, to my understanding, telemetry messages are forwarded by Hono having QoS 1 always ...
So I think the case you described shouldn't happen.

I don't think so. The sender link uses QoS 1 for telemetry messages as well, yes. However, there is no re-sending of negatively settled messages by the downstream consumer. Re-Sending does not happen for telemetry messages in Hono, even not for QoS 1 telemetry, as far as I know.
Such telemetry messages would have to be persisted in the broker I guess to be delivered with actual QoS 1 semantics.

[fkaltner]

Ah, the gateway would have to publish that event? Sure, that would also work.
Yes, that was my understanding. This made sense to me, since only the gateway knows what capabilities the device has.

I am not sure about the details about re-delivery - @calohmn can you help out?

[calohmn]

The sender link uses QoS 1 for telemetry messages as well, yes. However, there is no re-sending of negatively settled messages by the downstream consumer. Re-Sending does not happen for telemetry messages in Hono, even not for QoS 1 telemetry, as far as I know.

Yes. I agree that in case of a telemetry message we should send a separate _event_ message with the _hono_registration_status_ property then (before forwarding the telemetry message), in order to make sure that the message with that property gets brokered and will also get delivered if the downstream consumer is temporarily unavailable for example.

[fkaltner]

I am not sure why we would still need the hono_registration_status property then: how about sending a specific event in that case as well? Then we would not have two different approaches and a downstream application could always listen for those specific events in order to get notified about auto-provisioned devices.

Also this approach still has some edge cases: even if we send an event when a telemetry message of a not yet provisioned devices comes in and we wait for the event being settled we could lose the telemetry message if the application crashes at that very point of time.

[calohmn]

I am not sure why we would still need the hono_registration_status property then: how about sending a specific event in that case as well?

I think a specific event would be characterized by the very fact that it has that hono_registration_status property. That's also how the TimeUntilDisconnectNotification event messages are set up (sent out for example when a MQTT device subscribes for commands) - they are (usually) event messages containing a ttd property.

In that regard we could indeed follow the "piggy-back" approach of just adding that property if the message already is an event message. I think if the northbound application rather wants separate event messages with that status information or if it needs more information, then the gateways should be set up so that they publish specific events (#1618) in such a case.

Also this approach still has some edge cases: even if we send an event when a telemetry message of a not yet provisioned devices comes in and we wait for the event being settled we could lose the telemetry message if the application crashes at that very point of time.

But the delay in publishing the telemetry message by waiting for the disposition update for the event message first wouldn't make it more likely that the telemetry message gets lost, would it? The telemetry might get lost anyway and it's probably also not the last telemetry message of that device.

[thjaeckle]

It would be very helpful if the gateway ID could also be included as part of the message which indicates that a new device was registered.
That way downstream applications (like Ditto) could create a relation of an edge-device (or thing) to its gateway-device as well.

Maybe as additional defined property? What do you think?

[fkaltner]

Sounds reasonable and is easy to implement. So I guess there's no harm on Hono's side 😄 We could name the property hono_registration_via.

[sophokles73]

If we want to include the gateway identifier then FMPOV it should simply be gateway_id in analogy to the device_id property. In fact, we could/should include this property in every message published by a gateway on behalf of a device...

[sophokles73]

So, it seems like we include two properties in the downstream message:

1. gateway_id which contains the identifier of the gateway that acted on behalf of the device (if applicable) and
2. hono_registration_status which contains value NEW if the device has been provisioned on the fly by a gateway.

[fkaltner]

Yes, but: at the moment the _gateway_id_ will only be included when auto provisioning (and not for all messages) for an edge device sending data via a gateway. Obviously it will always be included in that case.

Intention was to keep the change small.

[sophokles73]

@fkaltner The intention of this issue is to discuss requirements/ideas for how this should be implemented. If multiple PRs are required in order to get the desired behavior, so be it. There is no need to stuff everything into a single PR. It is totally ok if we add the gateway_id property for all messages received via a gateway later ...

However, it would be good if we can agree on the desired behavior within this issue first.

[fkaltner]

Alright. In general I think the _gateway_id_ being set for all messages sent via a gateway makes sense!

Talking about ideas:
How about including the possibility to set the authority for a gateway device also for client-certificate based auto-provisioning?
By this we could combine both approaches: the only thing required for edge devices communicating through a gateway with Hono then would be a client certificate for the gateway.

[sophokles73]

How about including the possibility to set the authority for a gateway device also for client-certificate based auto-provisioning?
By this we could combine both approaches: the only thing required for edge devices communicating through a gateway with Hono then would be a client certificate for the gateway.

I am not sure if I understand correctly. Can you elaborate on what you want to achieve?

[fkaltner]

Say there are edge devices communicating with a gateway and the gateway itself is configured with a client certificate and is auto-provisioned.

In order to not have to do a registration of the gateway's edge devices one could use the implicit registration implemented in the course of this issue. However, to my understanding, we would still have to set the authority on the gateway device itself by updating it.

The idea was to be able to include the necessary authority for implicit device registration of the gateway into the client certificate based auto-provisioning of the gateway (not sure if we can put something like this as additional data into a client certificate...).

[sophokles73]

Ok, I see your point now. X.509 certificates may contain (non-critical) custom extensions that could be used for indicating such an authority. However, the overall approach taken by Hono (currently) is to not require any particular information and/or syntax in certificates in order to be able to use existing certificates issued by standard CAs. Introducing such an extension would require the client certificate(s) to be issued by a custom CA that is aware of the extension and includes it in the issued certificate(s).

[fkaltner]

Ok, I see your point now. X.509 certificates may contain (non-critical) custom extensions that could be used for indicating such an authority. However, the overall approach taken by Hono (currently) is to not require any particular information and/or syntax in certificates in order to be able to use existing certificates issued by standard CAs. Introducing such an extension would require the client certificate(s) to be issued by a custom CA that is aware of the extension and includes it in the issued certificate(s).

Good point. Another option would be to store this information (gateway should be granted auto-provisioning authority) along with the CA certificate in the device registry. By this we would be independent of the CA supporting custom extensions.

[fkaltner]

@thjaeckle While implementing (fyi, see: https://github.com/eclipse/hono/pull/2094) we stumbled upon a problem with the approach we devised, hence the following question arose:

How about _always_ (meaning not only when the device is not registered AND it is a telemetry message but for telemetry and event messages) sending an empty event with the application property hono_registration_status being set to NEW? This means in case of an event being sent as the device's first message you would get the empty event as well.

[sophokles73]

Another option would be to store this information (gateway should be granted auto-provisioning authority) along with the CA certificate in the device registry. By this we would be independent of the CA supporting custom extensions.

Can you give an example of where and how you want to store this information?

[sophokles73]

How about always (meaning not only when the device is not registered AND it is a telemetry message but for telemetry and event messages) sending an empty event with the application property hono_registration_status being set to NEW? This means in case of an event being sent as the device's first message you would get the empty event as well.

How would that help with the newly identified issue(s)?

[fkaltner]

How would that help with the newly identified issue(s)?

Well, this would make it easier to have the whole implementation in the device registry. At least that was my understanding of your comment in the PR 😅:

It doesn't need to because the registry would always publish an event indicating that the device has been provisioned while the protocol adapters would simply forward the telemetry message it received from the device as is.

Before making a decision I wanted to make sure if it is an option at all from Ditto's perspective.

On your concerns regarding robustness of the approach this would not help per se unless we implement a persistent flag about the notification having been sent to the application (but regarding that point we didn't have a consensus yet fmpov).

[thjaeckle]

How about _always_ (meaning not only when the device is not registered AND it is a telemetry message but for telemetry and event messages) sending an empty event with the application property hono_registration_status being set to NEW? This means in case of an event being sent as the device's first message you would get the empty event as well.

hi @fkaltner - this would work from our perspective as well

[sophokles73]

How would that help with the newly identified issue(s)?

Well, this would make it easier to have the whole implementation in the device registry. At least that was my understanding of your comment in the PR sweat_smile:

Ok, then I misunderstood your question. Let me try to re-phrase what I think we should do:

1. Gateway sends telemetry or event message for a device that does not exist yet.
2. Protocol adapter requests registration assertion from Device Registration service.
3. Device Registration service creates device without credentials and publishes an empty notification event containing the tenant_id, device_id, gateway_id and hono_registration_status=NEW properties.
4. Device Registration service returns registration assertion to protocol adapter
5. Protocol adapter forward data from device in downstream message as usual, i.e. the message does not contain the hono_registration_status property.

Would that work for everybody? @fkaltner @thjaeckle ?

[thjaeckle]

That works for us 👍 (see comment below clarifying)
Just for my understanding: Between step 3 and 4 - is the settlement of the empty notification event being waited for?

[fkaltner]

Protocol adapter forward data from device in downstream message as usual, i.e. the message does not contain the hono_registration_status property.

Actually I do think we need to include hono_registration_status property in order to support the "specific event use case":

(see above in https://github.com/eclipse/hono/issues/2053#issuecomment-660137717):

However, the gateway might also publish a specific event in order to indicate the device's capabilities as discussed in #1618. In that case Ditto could use the information from the payload to create a typed Thing that reflects the capabilities of the device already. This, however, is completely optional.

Since the specific event will be specific by

(..) the very fact that it has that hono_registration_status property
(see https://github.com/eclipse/hono/issues/2053#issuecomment-663605563)

[fkaltner]

Just for my understanding: Between step 3 and 4 - is the settlement of the empty notification event being waited for?

Yes, it is.

[thjaeckle]

Actually I do think we need to include hono_registration_status property in order to support the "specific event use case":

Yes .. if the gateway would already send this special "new device creation" event (e.g. also not being empty but with a custom payload) I would not see why there should be 2 events sent out.

[thjaeckle]

@sophokles73 sorry, I misunderstood before and answered to eagerly :)

what Ditto would need is to always receive only 1 event:

• containing the properties tenant_id, device_id, gateway_id and hono_registration_status=NEW
• the payload being:
  
  
  
  • either empty (event was created as a result of some other telemetry message sent by the newly created device)
  
  
  • or containing a custom sent payload (by e.g. the gateway, defining the Ditto policy, Vorto model, etc.)
  
  

FMPOV when the first message of a new device is already an event, the properties could just be added to that event.
And when the first message of a new device is telemetry, an additional empty notification event should be sent before sending the telemetry.

[sophokles73]

Just for my understanding: Between step 3 and 4 - is the settlement of the empty notification event being waited for?

Yes, it is.

To be absolutely clear: The Device Registration service will wait for the disposition from the Artemis broker in this case. This, however, does not mean that the event has been processed by any downstream application. It just means that the event has successfully been received and persisted by Artemis.

[sophokles73]

@thjaeckle

If we opt for the Device Registration service to publish the event, then I would rather not have the protocol adapters to distinguish multiple cases but instead simply always follow the same process, i.e. registry is responsible for (always) publishing event indicating newly created device while the protocol adapters simply forward the message received from a device as they always have before. The special event I have been talking about before would then simply contain information about the device's capabilities (info model) but would otherwise not be special in any kind and would therefore also not contain the hono_registration_status property.

[thjaeckle]

@sophokles73 I'm not sure if I can follow - as you described it there could be 2 events (the first being the empty notification and the second containing the device capabilities?), did I understand that correctly?

Ditto won't be able create a thing in "2 steps" (where the device's capabilities are contained in a second event) as Ditto's consumption of event/telemetry messages is stateless and can't wait for a second message to come.
Also, the second message could be received by another cluster node if more than one consumer is connected to Hono.

[sophokles73]

as you described it there could be 2 events (the first being the empty notification and the second containing the device capabilities?), did I understand that correctly?

Yes, that is correct. The first event would have been created by the Device Registration service and would indicate that the device has been created newly. The second one would be the event that had been uploaded by the gateway, indicating the device's capabiliites.

Why shouldn't Ditto be able to create a (default, empty) Thing based on the first event and then (later) update the Thing with information from the second event?

[fkaltner]

Also, the second message could be received by another cluster node if more than one consumer is connected to Hono.

@sophokles73 If more than one cluster node is connected then ordering in Artemis is not guaranteed afaik, right? If so, then the empty event could arrive after the device's first telemetry or event message if I am not mistaken.

[thjaeckle]

Why shouldn't Ditto be able to create a (default, empty) Thing based on the first event and then (later) update the Thing with information from the second event?

Several reasons, these come directly to my mind:

• the policy would not be the correct one when the empty thing is created first (the initial policy could only allow the channel to Hono to READ+WRITE the complete thing)
  
  
  
  • as a result, an application requiring to be notified of the "thing created" event would not get notified of the creation (because it is missing from the policy)
  
  
  • if an already existing policyId would should be re-used, Ditto would first implicitly create a policy which would be obsolete if the second update message would set the policyId to the existing one -> there would be a no longer referenced policy
  
  
• attributes from the first message (e.g. "gateway_id" or and indication "connectedViaGateway") would be overwritten in the second message (applying e.g. the Vorto based template)
• "cost": 2 DB operations instead of only 1

[fkaltner]

Based on the discussion, once again the proposed steps:

1. Gateway sends telemetry or event message for a device that does not exist yet.
2. Protocol adapter requests registration assertion from Device Registration service.
3. Device Registration service creates device without credentials and publishes an empty notification event containing the tenant_id, device_id, gateway_id and hono_registration_status=NEW properties.
4. The Device Registration service waits for the disposition from the Artemis broker of the empty notification.
5. Device Registration service returns registration assertion to protocol adapter
6. Protocol adapter forward data from device in downstream message as usual, i.e. the message does not contain the hono_registration_status property.
7. Ditto receives the event and creates a thing using the gateway's policy as a template

Consequently:

• There will be no Vorto model for the created thing (which is _currently_ not required for the use cases known to the participants of this discussion)
• There still may be edge cases due to missing message ordering guarantees for multiple consumers: the empty event may arrive at the application after the device's first telemetry or event message was processed. In this case it is acceptable for now to lose the telemetry or event message.

[joosdavid]

To guarantee a good user experience it would be an advantage if Things can send a response to the empty notification. For this it is necessary that Hub sets the "reply-to" header in the empty notification. Is this possible?

[sophokles73]

@joosdavid not sure what you mean with good user experience in this case. FMPOV your proposal would turn the event/notification that the registry sends in order to inform (arbitrary) other services about the newly registered device into a remote procedure call on exactly one service, thus increasing coupling ...