Hono: Describe MQTT adapter compliance with MQTT specification

[dejanb]

As Hono is not a full MQTT broker, there are some restrictions in supporting all MQTT features. In particular:

• clean-session MUST be true. Connect requests with clean-session=false will be rejected.
• retain flag MAY be true. Messages will be confirmed if necessary, and an “x-opt-retain” application property will be added.
• Messages published with qos=2 are not acknowledged, discarded and tracked in malformed-messages Hono metric.
• LWT will be ignored if provided. Hono will log it and track it in “some” Hono metric.

We should document these restriction in MQTT protocol adapter user guide

[ppatierno]

retain flag MAY be true. Messages will be confirmed if necessary, and an “x-opt-retain” application property will be added.

Agree on that but in any case we don't use the MQTT adapter for subscriptions today. So how much is it useful to have this new annotation ? In any case it's something that we provided in the MQTT gateway implementation in EnMasse.

[pellmann]

This was brought up for the future command and control, where the device wants to subscribe for its commands.

[ppatierno]

In this case we are not going to send commands from MQTT devices so why the MQTT adapter should handle the retain flag for retaining a message from the MQTT device ?
Having the x-opt-retain application property on the AMQP command message could make sense for having that message immediately delivered to the MQTT device when it connects and subscribes (it's the reason for retain as "last well known" message).
But I don't see the reason for handling the retain flag from a publisher point of view to the MQTT adapter.

[sophokles73]

So how much is it useful to have this new annotation ?

For Hono it probably is completely useless (so far). The idea was to let downstream consumers decide whether they want to react to the fact that an MQTT device has published the message with retain=true. Hono itself doesn't make any guarantees regarding the delivery of such messages to new subscribers.

The alternatives would have been to either close the connection if an MQTT device publishes such a message or to silently discard it or to ignore the retain flag. This alternative seemed to be a good compromise but maybe you can think of a better alternative?

[ppatierno]

Hono itself doesn't make any guarantees regarding the delivery of such messages to new subscribers.

Ok but we have to document this somehow because, in a way, we are not honoring the spec (which already happens for other MQTT features). Who uses the MQTT Hono adapter should know that even if the message is published with retain=false, it's will not delivered to other devices which will connect later.

[sophokles73]

That's the overall intention of this whole issue ;-) Making sure that everybody understands in what way the MQTT adapter does not comply with/support the spec.

[sophokles73]

The MQTT 3.1.1 spec defines in section 3.2.2.2

If the Server accepts a connection with CleanSession set to 1, the Server MUST set Session Present to 0 in the CONNACK packet in addition to setting a zero return code in the CONNACK packet [MQTT-3.2.2-1].

If the Server accepts a connection with CleanSession set to 0, the value set in Session Present depends on whether the Server already has stored Session state for the supplied client ID. If the Server has stored Session state, it MUST set Session Present to 1 in the CONNACK packet [MQTT-3.2.2-2]. If the Server does not have stored Session state, it MUST set Session Present to 0 in the CONNACK packet. This is in addition to setting a zero return code in the CONNACK packet [MQTT-3.2.2-3].

The Session Present flag enables a Client to establish whether the Client and Server have a consistent
view about whether there is already stored Session state.

Once the initial setup of a Session is complete, a Client with stored Session state will expect the Server to maintain its stored Session state. In the event that the value of Session Present received by the Client from the Server is not as expected, the Client can choose whether to proceed with the Session or to disconnect. The Client can discard the Session state on both Client and Server by disconnecting,
connecting with Clean Session set to 1 and then disconnecting again.

My understanding of this is that the server uses the session present flag to indicate to the client if it has session state for the client's identifier or not. In Hono we do not maintain session state so we always return false as the value of this flag. However, I do not see a reason why we should reject a connection request with a clean session flag having value 0. Instead, we simply accept with session present set to false and let the client decide if this is acceptable for it (keep connection) or not (close connection).

WDYT @dejanb @ppatierno

[sophokles73]

Regarding the retain flag: I would like to restrict this to events and telemetry messages only, i.e. do not apply it to command response messages, and also would like to use an AMQP 1.0 message annotation instead of an application property. Thoughts @dejanb @ppatierno @pellmann ?

[pellmann]

@sophokles73 Why would you change x-opt-retain to a message annotation? Message annotations are for the infrastructure and it is not guaranteed that they will be available after some steps (like brokers). For my understanding we created this annotation, because we do not support this at the infrastructure-level and because of this, we give the information to the application level.

I agree with you that it makes no sense for command responses.

[dejanb]

@sophokles73 "session present" and ignoring retain for command responses make total sense to me

[sophokles73]

@pellmann

Message annotations are for the infrastructure and it is not guaranteed that they will be available after some steps (like brokers).

In fact, the AMQP 1.0 spec states that

The message-annotations section is used for properties of the message which are aimed at the infrastructure and SHOULD be propagated across every delivery step. Message annotations convey information about the message. Intermediaries MUST propagate the annotations unless the annotations are explicitly augmented or modified (e.g., by the use of the modified outcome).

FMPOV a broker would also be an intermediary because it clearly is not the target of a message (unless it uses a broker specific management address but this is not the point here). As such, it would have to propagate the message annotations.

IMHO you can also argue that the flag is indeed targeted at the infrastructure since its proper handling, i.e. retaining the message for delivery to new consumers, should not depend on the content and semantics of the message itself. In fact, message brokers usually provide such a feature out-of-the-box under the name persistent subscription (usually not limited to the last message though). That said, I would not insist on making x-opt-retain a message annotation but FMPOV these are good reasons for doing so.

[pellmann]

@sophokles73

For me, the word SHOULD says that there may be problems in an unknown AMQP 1.0 network with message annotations.

So you argue you want to use the x-opt-retain flag somewhere in the AMQP 1.0 network to deliver the feature of retain (e.g. with the Broker)? In this case you are right and we should use the message annotation. In the case that we want to let the handling to be done by the application (that was my impression) FMPOV it should be an application property.

But in most cases it does not really matter..

[sophokles73]

So you argue you want to use the x-opt-retain flag somewhere in the AMQP 1.0 network to deliver the feature of retain (e.g. with the Broker)?

Not necessarily, I am just saying that this kind of handling could be easily done in a generic way at the infrastructure level (whatever that means) instead of an application (which in this case would act as a consumer of the message). However, there is nothing that would prevent an application to react to this annotation if it wants to ...

[pellmann]

For my understanding, if we use a message annotation than the expectation is, that it is something that could (and should) be understood by the AMQP network. It is a feature/capability, that some component could understand and also we should define the meaning and put it in the registry. If we want to create such a retain feature at this level I think it is the right place and if not I would assume that we should not use message annotations.

AMQP 1.0 spec

The capabilities negotiated on link attach and on the source and target can be used to establish which annotations
a peer understands; however, in a network of AMQP intermediaries it might not be possible to know if
every intermediary will understand the annotation. Note that for some annotations it might not be necessary for
the intermediary to understand their purpose, i.e., they could be used purely as an attribute which can be filtered
on.
A registry of defined annotations and their meanings is maintained [AMQPMESSANN].

[sophokles73]

It is a feature/capability, that some component could understand and also we should define the meaning and put it in the registry.

I do not intend to define a retain feature in Hono, I merely want to allow downstream components to detect that the sender of the message intended to retain the message (in the MQTT sense). That's all. Whether any such downstream component actually makes use of that information (and in what way) is completely to the discretion of that component. The only reason to use a message annotation instead of an application property is that such a retain feature might very well be implemented in a generic way at the infrastructure level (not necessarily as part of the AMQP 1.0 Messaging Network).

[sophokles73]

Ok committers, let's have a vote on this one.

If you are in favor of adding the x-opt-retain flag as a message annotation, add your thumbs up to this comment.
If you would rather have the flag being added as an application property of the message, add your thumbs down to this comment.

@dejanb @ctron @ppatierno @sysexcontrol @pellmann
please cast your vote

[sophokles73]

@ctron @ppatierno can you please cast your vote or indicate that you'd like to abstain?

[ppatierno]

@sophokles73 sorry to be late but I am on vacation :P I have just voted. Using a message annotation is the same way we used for the MQTT gateway in EnMasse so I agree on that :)

[sophokles73]

There is one other (good) argument for add the flag as an application property instead of a message annotation: a JMS provider based consumer may not be able to read the flag's value if it is a message annotation because there is no mapping specified (yet) for annotations to JMS properties ...

[sophokles73]

Messages published with qos=2 are not acknowledged, discarded and tracked in malformed-messages Hono metric.

qos 2 messages are not acked and discarded. However, they are not yet tracked in a malformed messages metric. We will add such a generic metric for all protocol adapters after we have migrated to Micrometer #794