Azure-iot-sdk-c: Question: Handling loss of network connectivity

[Patrik-Berglund]

Hi,

If the network connectivity is completely lost, should I destroy the client in that case? (Disconnected LAN cable, WIFI AP down or similar to that)

IoTHubClient_LL_Destroy(iotHubClientHandle);
platform_deinit();

And later when the network connectivity is recovered, could be seconds, hours, days or weeks later recreate it from scratch?

platform_init()
IoTHubClient_LL_CreateFromConnectionString
IoTHubClient_LL_SetDeviceTwinCallback
and so on....

Or does the SDK handle this by it self?

[jspaith]

One of the main advantages of using the C SDK versus bringing just a "raw" MQTT/AMQP library is that it handles all this retry for you automatically. So you shouldn't have to yank everything down. We've many fixes here especially around Dec/Jan, so you'd want to be on a newer SDK. We've never tested as far out as a week but we don't give up retry.

We in fact have fault injection tests that run as part of our gated checkin, where we have a back-channel communication to IoT Hub and it will kill TCP Connection on us, send random errors, kill AMQP links, etc. to make sure we recover. E.G. https://github.com/Azure/azure-iot-sdk-c/blob/master/iothub_client/tests/iothubclient_mqtt_e2e_sfc/iothubclient_mqtt_e2e_sfc.c

That said - there is at least one bug where we don't work well with http proxies and retry. This is being actively investigated, https://github.com/Azure/azure-iot-sdk-c/issues/264.

[zvrba]

Oh, that's nice to know. Thanks for this work, it's really a PITA to implement retry in the application.
Do you know by any chance whether the C# sdk also implements some retry mechanism?

[jspaith]

I know C# has some reconnection logic, but I don't know all the particulars. I'd checkout https://github.com/Azure/azure-iot-sdk-csharp/blob/master/.github/CONTRIBUTING.md#need-support or potentially they've doc'd this already.

[Patrik-Berglund]

Thanks for your quick answer @jspaith

How does the SDK handle outgoing messages (D2C, Device Twin updates) during these outages? Will it queue the messages and then send them when the connection is recovered?

[ewertons]

Hi @Patrik-Berglund ,

The D2C (aka telemetry) messages get queued and are re-sent (in the order they were passed to the device client C SDK) after a reconnection is completed.
If using AMQP, the client behaves the same way for Twin messages.
The MQTT transport does not resend twin messages (this one is a point of improvement that we are currently discussing).

[Patrik-Berglund]

Hi @ewertons

How deep is that queue, configurable? If full will it throw the oldest item?

[ewertons]

Hi @Patrik-Berglund ,
the queue has no limit (thus, it's also not configurable).

[Patrik-Berglund]

Ok,

So if you are running on a resource constrained device you will need to make sure that you don't run out of memory when the queue starts to fill up...

[ewertons]

Yes.
One option is to monitor when the client gets disconnected (through the StatusConnectionCallback function) and not send more messages until it gets invoked with status Authenticated (indicating the reconnection has succeeded).

[jspaith]

You could also count the number of callbacks you've received and not send more messages until the # drained off.

Because there's workarounds that are serviceable (albeit not eloquent) we've never implemented this in the SDK.

We understand the concerns here though, so if you want to create a new issue asking for a max-messages-to-queue we'd mark as an enhancement and get on our backlog.

[ewertons]

Hi @Patrik-Berglund ,
we will close this issue, but feel free to reopen it if you would like follow up on your question.

Thanks,
Azure IoT Team

[yzhong94]

@Patrik-Berglund @zvrba , thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

[ukluk]

@jspaith @ewertons This is not working for me as documented.

We have a cellular connection.
Each 24hrs the carrier disconnects.
It is a constrain device using the mBed-TLS and FreeRTOS

I am not able to recover unless a teardown occurs with the full IoTHubClient_LL_Destroy(iotHubClientHandle); and platform_deinit(); as @Patrik-Berglund suggested.

We are implementing the raw sockets using Telit stack, but it is very similar to the net socket code you have around WIFI.

After 100 errors the StatusConnectionCallbackis not changing. What should I change?

I would like to recommend passing an abort token to IoTHubClient_LL_DoWork(); and force the retry policy to give up and re-authenticate again. We can fork the iothub client and suggest a Pull Request if this is beneficial to the community.

Here some traces:

NO CARRIER
Network Disconnected
Time 2019-03-01T15:03:11
Cellular Recovery
Modem::Recovered
Time 2019-03-01T15:03:49

ERROR: net_sock_send_tcp_cellular L#204 Send failed.
ERROR: net_sock_send_tcp_cellular L#204 Send failed.
ERROR: net_sock_send_tcp_cellular L#204 Send failed.

[cjohnsonoem]

Having basically the same issue. The StatusConnectionCallback is not triggered when network loss is present. We are using a HTTP get request to the android generate 204 URL to signal network connection status back to our main controller.

When we lose internet connection the SDK never does a callback.

[jspaith]

@ukluk @cjohnsonoem - sorry for delays here. We don't really track closed issues so I'd recommend opening a new issue on this.

Please tag @jebrando as he happens to be looking at our retry logic now and may have some more insight.