Azure-iot-sdk-c: Question: retry logic on device-to-cloud messages

[syedelec]

• OS and version used:
  Ubuntu 16.04
• SDK version used:
  release 2018-04-13

Description of the issue:

Hi,
I made some tests with AMQP protocol to send data from device to cloud using IoTHubClient_SendEventAsync. I defined inside the confirmation callback a status depending on the result IOTHUB_CLIENT_CONFIRMATION_RESULT
If the result is IOTHUB_CLIENT_CONFIRMATION_OK the status it's ok, otherwise it fails.

For my tests, I am continuously sending data to cloud, then I manually disable internet connection for 10 seconds, then I re-enable it and I see all the buffered (not sent) data are sent correctly.

According to this: https://github.com/Azure/azure-iot-sdk-c/issues/438
The IoTHubClient_SetRetryPolicy function defines retry policies to the reconnection only, not to the message re-send logic, so data will be re-sent despite the retry policy as soon as the reconnection occurs.

In my case, I prefer not to use the SDK's internal _message_ retry logic and have to handle this in my code. Basically, when I call IoTHubClient_SendEventAsync to send data, is it possible to get an immediate return from the confirmation callback, either succeeded or failed.

The whole process would be :

• Send data with IoTHubClient_SendEventAsync
• confirmation callback prints success (receiving IOTHUB_CLIENT_CONFIRMATION_OK)
• Disconnect internet
• confirmation callback prints fail (immediately for all messages during disconnection, receiving other result than IOTHUB_CLIENT_CONFIRMATION_OK)
• Re-connect internet
• confirmation callback prints success (receiving IOTHUB_CLIENT_CONFIRMATION_OK)

Is this behavior possible with the current implementation of azure ?

Thanks.

Code sample exhibiting the issue:

N/A

Console log of the issue:

N/A

[ewertons]

Hi @syedelec ,
good question. Very clear.

As you correctly assessed, the retry policy applies only to the reconnection.
And the reconnection code is separate from the telemetry messaging code. So there is no way to trigger a send-message failure directly by tweaking with the reconnection operation and options.

However, you can get close to your desired behavior if you play with the send-message timeout.
Messages will return to you through the callback in case sending succeeds, if sending explicitly fails sending a few times, or if the message times out being sent.

The default timeout for sending a telemetry message is 5 minutes.

You can reduce that to a very short amount of time (a few seconds) by setting this option:

 #include "iothub_client_options.h"
…
size_t telemetryTimeoutInSecs = 10;
IoTHubClient_SetOption(iotHubClientHandle, OPTION_EVENT_SEND_TIMEOUT_SECS, &telemetryTimeoutInSecs);

Would that work for you?

[syedelec]

Hi @ewertons sorry for late answer, this does work for me, thanks !

I have a question concerning the option OPTION_MESSAGE_TIMEOUT, in the first case I was using this option but it did not have any effect on the device to cloud messages. When does this option apply ?

[lucarv]

I have the same issue (sort of).
I would like to reveive a callback for message sending failure.

This is how we are setting this up, I have setup the simple telemetry sample to send messages over AMQP, and I receive the following:
_Confirmation callback received for message 1 with result IOTHUB_CLIENT_CONFIRMATION_OK_
I then disable the device on IoT Hub and I get absolutely no callback, whereas I would expect to receive
_Confirmation callback received for message 1 with result IOTHUB_CLIENT_CONFIRMATION_ERROR_

I have done some tracing and there is never a callback returned from the transport to the client. Is that expected?
I have lots of printf with errors, the last one being
_Error: Time:Wed Jul 11 14:00:59 2018 File:/Users/lucarv/source/c/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:IoTHubTransport_AMQP_Common_DoWork Line:1614 Device 'bwt-amqp-001' reported a critical failure; connection retry will be triggered._

[ewertons]

@syedelec , please take a look at OPTION_EVENT_SEND_TIMEOUT_SECS.

@lucarv , for how long did you wait after disabling the device?
Messages don't fail right away when they are added to the internal queue. They stay there for up to a certain amount of time (take a look at the same document above please), and if they don't get sent within that, they are returned with failure. Notice that during reconnections the SDK does not try to send messages on the wire, so you should not have an error for a few minutes while the client is trying to reconnect.
Another doc you could check is this one, it has a lot of details about reconnection.

[lucarv]

@ewertons we waited for a minimim of 6 minutes and a maximum of 60. We also had retry strategy set to none to absolute no difference (beyond the absence of retries).

[ewertons]

This will require some more investigation.
Since this is a "sort of" the same issue as @syedelec 's (actually it is not), it would have been better to open a separate issue, so we don't mix up. But let me verify here first on our side.

[syedelec]

@ewertons Thank you for the details. So the OPTION_MESSAGE_TIMEOUT option is now DEPRECATED. From my side the issue is resolved, thanks again.

[lucarv]

@ewertons you are absolutely right, let me know if i should created a new issue to be tracked separately

[jspaith]

@lucarv - yes, could you please create a new issue since unfortunately we lost the thread of both issues.

I have lots of printf with errors, the last one being - when you re-open the issue, could you please provide more of the logs just being careful no private information about the hub/connection is leaking out? The DoWork() error tends to be the last log message on the stack as the errors bubble up so we'll need a bit more context. Thanks

In meantime I'm going to close out this issue for clarity since @syedelec's initial problem has been addressed.

[az-iot-builder-01]

@syedelec, @lucarv, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey