Hono: Reduce number of Command & Control receiver links

[calohmn]

Currently there is one Command & Control receiver link for each device, opened from the protocol adapter instance to the AMQP network.
This link is on the address control/$tenant/$device (or control/$tenant/$gateway respectively).

It would be good if we could reduce that potentially large number of links.

As proposed by @sophokles73, we could use receiver links for each protocol adapter instance (containing an instance uuid in the address) instead.

Taking on the example illustration from #930

this means replacing the red control/$tenant/$gateway links with just one for each protocol adapter instance, e.g. control_internal/$tenant/$protocolAdapterUUID.

Whenever a device/gateway subscribes to a protocol adapter for receiving Command & Control messages, we will store the mapping of device/gateway ID -> protocol adapter instance via the new Device Connection API. We will also make sure there is a receiver link control_internal/$tenant/$protocolAdapterUUID on that protocol adapter instance.
Upon receiving a command on the tenant-scoped receiver link, the protocol adapter will query the Device State API to get the corresponding protocol adapter uuid and forward the command message to control_internal/$tenant/$protocolAdapterUUID.

---

Concerning when to implement this:
we currently have 2 kinds of receiver links in the protocol adapters:

• control/$tenant (tenant-scoped link with gateway-mapping feature)
• control/$tenant/$device (legacy command link, also used now for forwarding the gateway-mapped commands internally)

The 2nd control/$tenant/$device link will be made into an only-used-internally kind of link at some point, removing the backwards compatibility for applications to send commands on that link.
When we do this, we could change the control/$tenant/$device link into the new
control_internal/$tenant/$protocolAdapterUUID link. That way, we don't have to deal with 3 kinds of receiver links at the same time.

[ctron]

Wouldn't it make sense to allow the client side to also re-use that information in some way. Like passing along the protocol adapter ID in the telemetry messages and allow the client to send a command directly to that protocol adapter.

That would allow to forward the device command (after an inbound message) directly to the correct protocol adapter and prevent us from doing a lookup in order to figure out the "correct" protocol adapter.

[calohmn]

I see that this would speed up routing of command messages.

However, I think we still need to support the use case that an application can send commands without having received the protocol adapter ID from Hono. E.g. there could be an MQTT device subscribed with ttd=-1 and the application could already have been restarted since getting the event with ttd=-1.

That means that this direct addressing of the correct protocol adapter would have to be a 2nd kind of option for applications to send commands (besides the existing way).
And that seems a bit odd, supporting 2 kinds of endpoints for sending commands (with the only difference that in one case the routing could be a bit quicker).
And this direct addressing endpoint would make the application depend on internal, technical implementation details. Maybe we later on introduce another way of routing from one adapter to the other which would make this obsolete.

Therefore I wouldn't describe this as an officially supported way.
(We could set the protocol adapter ID in the telemetry/event message along with the ttd property anyway and thereby enable applications to use it unofficially, though.)

[sophokles73]

IMHO it makes sense to work on this after we have removed the old control endpoint from the north bound C&C API (#1328), otherwise we will still have the old device scoped receiver links opened by the adapters ...

[calohmn]

In order to develop the requirements for the Device Connection service changes, here are different scenarios concerning routing of Command & Control messages:

Simple scenarios:

1. Device connected directly with adapter, subscribes for commands
   When handling a command subscription request for a device 4713, the protocol adapter instance #1 that received this request needs to store a mapping in the Device Connection service, assigning device 4713 to adapter instance #1.
   When a command for device 4713 gets received by adapter instance #2 on the tenant-scoped command consumer link, the adapter makes a request to the Device Connection service to retrieve the mapping information (getCommandDestination).
   As the mapped adapter instance is #1, the adapter forwards the command to that adapter instance on the adapter-instance specific command link (command_internal/$tenantId/$adapterInstanceId). Adapter instance #1 receives the command and sends it to the device.

---

Gateway scenarios:

1. Gateway subscribing for commands on behalf of all its devices
   A gateway gw-1 makes a request to adapter instance #1, subscribing for commands, using just its gateway device id, wanting to receive commands for all devices it acts on behalf of.
   With a request to the Device Connection service, the adapter establishes the mapping from gw-1 to adapter instance #1.

   • Device has already sent messages
     A device 4711 is connected to gw-1 and already has sent telemetry messages via the gateway. That means the "lastKnownGateway" mapping for device 4711 has been set to gw-1.
     When a command for device 4711 arrives at adapter instance #1, the adapter requests the relevant mapped adapter instance id from the Device Connection service.
     For device 4711 there is no direct mapping to an adapter instance. That means the "lastKnownGateway" mapping for device 4711 is taken into account. The found value of gw-1 is taken to get the corresponding mapped adapter instance.
   • Device hasn't sent messages yet
     This addresses #1712: If device 4711 hasn't sent telemetry/event messages yet, there is no "lastKnownGateway" mapping yet.
     If device 4711 has only one "via" gateway defined, this information can be used in the Device Connection service to return the protocol adapter instance mapped to that "via" gateway - if that gateway has indeed opened a command consumer, causing for such a mapping to exist.
     This all means that the adapter has to provide the device "via" information in the "getCommandDestination" request to the Device Connection service.
     If device 4711 has multiple "via" gateways defined, the Device Connection service can return a result of potentially _multiple_ mappings, containing the gateways that have opened command consumers.
     With this information, the protocol adapter can send the commmand to one of these gateways.

2. Gateway subscribing for commands of a specific device
   Gateway gw-2 subscribing for commands to 4712 when connected with adapter instance #2 means that in the Device Connection service a mapping from 4712 to adapter instance #2 is established.
   In the Device Connection service, when handling a "getCommandDestination" request for device 4712, it is first checked whether there is an adapter instance mapping for device 4712. Only if no such mapping is found, the "lastKnownGateway" or the supplied "via" gateways are checked.
   Note:
   If Gateway gw-2 closes the subscription for commands to 4712 and later subscribes just with its gateway id (ie. subscribing to commands of all devices), a still existing mapping of 4712 to gw-2 in the Device Connection Service would cause a wrong return value for "getCommandDestination". Therefore, when closing a subscription, a Device Connection Service request should be done to remove the mapping. To avoid concurrency issues, the request to remove a mapping should contain device id and adapter instance, so that the device connection service implementation can skip the removal operation if the mapped adapter instance value has changed already, meaning there was a concurrent request from another adapter instance that added a mapping.

---

Somewhat special scenarios:

1. Device connected directly with adapter, subscribes for commands; subscription already exists
   Suppose a device 4713 has subscribed to commands via MQTT (with rather long keepAlive interval defined on the connection), reaching adapter instance #1.
   The device discovers a disconnect and creates a new subscription, this time the request reaches adapter instance #2. The disconnect of the previous subscription has not yet been noticed by adapter instance #1. That means that when adapter instance #2 wants to add a mapping in the Device Connection Service from 4713 to instance "#2", there is already a mapping for 4713.
   The option to return an error in the Device Connection Service here would mean that stale entries could effectively block all further requests.
   So, the existing entry should rather be overwritten.
   Note the implication here: Consider the case that in the above scenario, adapter instance #1 still hasn't noticed the disconnect with the device. A command directed at device 4713 reaches adapter instance #1 by chance.
   As there still is a local command handler for device 4713, the command could be directly sent to the device, resulting in an error or a timeout as the connection isn't intact anymore.
   First asking the Device Connection Service for the current target adapter instance, even though there is a local handler for the command, would mitigate this, avoiding the error by allowing the command to be directed to the correct adapter instance.
   This would of course add extra overhead that isn't needed in "normal" scenarios. Once adapter instance #1 has noticed the disconnect and removed the local command handler, this also wouldn't be needed anymore.
   So the question is whether the extra Device Connection Service request should be done. I tend to prefer it to be done.

---

Ergo
New Device Connection Service operations:

• void setCommandHandlerAdapterInstance(String deviceId, String adapterInstanceId)
• void removeCommandHandlerAdapterInstance(String deviceId, String adapterInstanceId)
• JsonObject getCommandDestination(String deviceId, List<String> viaGateways)
  returns a JsonObject with something like { lastknownGateway: gw-1, adapterInstanceMapping: { gw-1: adapterInstance#1 } }

Further questions:

• What to do if the removal of a mapping entry in the Device Connection Service doesn't happen for some reason - how to get the stale mapping entry to be removed?
  => We could let the protocol adapter do a periodic refresh of the mapping information while the command subscription is still alive. The Device Connection Service could delete entries older than the refresh interval.
• Should we include the tenant id in the adapter-instance specific command link or should we use just command_internal/$adapterInstanceId?
  => IMHO, adding the tenant id could add some flexibility concerning tenant based limits, so I would opt for including the tenant id.

[sophokles73]

So the question is whether the extra Device Connection Service request should be done. I tend to prefer it to be done.

Adapter #1 could do the lookup request only if it is unable to send the command to the device. That way we would only do it if necessary but not under normal conditions. WDYT?

[sophokles73]

What to do if the removal of a mapping entry in the Device Connection Service doesn't happen for some reason - how to get the stale mapping entry to be removed?

Why not simply define a max age of entries in the device connection service?

Should we include the tenant id in the adapter-instance specific command link or should we use just command_internal/$adapterInstanceId?
=> IMHO, adding the tenant id could add some flexibility concerning tenant based limits, so I would opt for including the tenant id.

Can you give an example for when including the tenant ID would be helpful? FMPOV it only makes things more complicated than necessary. I particularly liked the idea of having just a single command receiver link per adapter instance ...

[calohmn]

So the question is whether the extra Device Connection Service request should be done. I tend to prefer it to be done.

Adapter #1 could do the lookup request only if it is unable to send the command to the device. That way we would only do it if necessary but not under normal conditions. WDYT?

I guess that would make the implementation more complicated. I would rather evaluate/implement this after the changes for this issue here.

What to do if the removal of a mapping entry in the Device Connection Service doesn't happen for some reason - how to get the stale mapping entry to be removed?

Why not simply define a max age of entries in the device connection service?

That was what I meant by saying the Device Connection Service could delete entries older than the refresh interval, ie. the max age.
Since we don't have a max age for command subscriptions, this makes it necessary for the protocol adapters to do a periodic refresh of the mapping information while the command subscription is still alive.

Should we include the tenant id in the adapter-instance specific command link or should we use just command_internal/$adapterInstanceId?
=> IMHO, adding the tenant id could add some flexibility concerning tenant based limits, so I would opt for including the tenant id.

Can you give an example for when including the tenant ID would be helpful? FMPOV it only makes things more complicated than necessary. I particularly liked the idea of having just a single command receiver link per adapter instance ...

Right, I didn't have a specific benefit of including the tenant ID in mind. So, yes, we should use the simple solution of using a single command receiver link per adapter instance.

[calohmn]

Another thought on this:

Should we include the tenant id in the adapter-instance specific command link or should we use just command_internal/$adapterInstanceId?
=> IMHO, adding the tenant id could add some flexibility concerning tenant based limits, so I would opt for including the tenant id.

Can you give an example for when including the tenant ID would be helpful? FMPOV it only makes things more complicated than necessary. I particularly liked the idea of having just a single command receiver link per adapter instance ...

Theoretically, using just command_internal/$adapterInstanceId would allow a (misbehaving) custom protocol adapter instance to open such a consumer link with an $adapterInstanceId already used by another protocol adapter instance, thereby potentially getting command messages not targeted at that adapter.
With an added tenant ID in the address, consumer creation of a custom protocol adapter could be restricted to addresses only containing specific tenants.
On the other hand, if we mandate usage of a UUID for the $adapterInstanceId, with a new UUID used with every adapter instance restart, the above shouldn't be a problem. So I think we can stick to a single command receiver link per adapter instance.

[calohmn]

Relevant PRs have been merged now.