Azure-iot-sdk-c: MQTT reconnect failed after network disconnection

[kevin0402]

• OS and version used:
  Debian 8, kernel version 4.1
• SDK version used:
  1.1.14

Description of the issue:

My device connects to internet via a Ethernet switch and keep sending messages to IoT Hub every 10 seconds.
[My Device --- Switch ---- Internet]

Data can be sent to and received from Azure IoT Hub without problems in normal case.
If I unplugged the Ethernet cable between switch and Internet, I found :

1. MQTT stack and underlying socket layer are too late aware that messages can not send out to IoT Hub. It takes about 15 minutes to sense that the network is disconnected.
   Wondering to know how you implement MQTT keep-alive feature ?
2. I plug the Ethernet cable back to the switch, but it is always failed when SDK tries to reconnect.
   You could refer to attached log for detailed error message. It seems that socket is in wrong state.

In the same test scenario , AMQP and HTTP can detect this kind of connection loss in a relative short time and reconnect to IoT Hub quickly.

Code sample exhibiting the issue:

Console log of the issue:

mqtt1.txt

[ollyw]

I think I am seeing similar issues when I block the TCP traffic to 8883 with TCP RST. The client doesn't seem to detect disconnection/reconnect. After a while, I get a segfault. I can't rule out something my end, but if I am interpreting the stack trace right, it seems to point to the file descriptor for the socket being closed, but a write happening anyway

Platform Ubuntu 14.04.5 LTS with statically compiled binaries with azure-iot-sdk-c at commit 9747a4dcdedfd1ed6923c08a4b3323e2b01d5804

```Program received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7ffff5fec700 (LWP 6241)]
0x00007ffff6dc995b in __libc_send (fd=13, buf=0x7fffe801d710, n=154, flags=-1) at ../sysdeps/unix/sysv/linux/x86_64/send.c:31
31 ../sysdeps/unix/sysv/linux/x86_64/send.c: No such file or directory.
(gdb) bt

0 0x00007ffff6dc995b in __libc_send (fd=13, buf=0x7fffe801d710, n=154, flags=-1) at ../sysdeps/unix/sysv/linux/x86_64/send.c:31

1 0x000000000046e617 in socketio_send ()

2 0x0000000000461ef2 in xio_send ()

3 0x0000000000468f7b in write_outgoing_bytes ()

4 0x000000000046a786 in tlsio_openssl_send ()

5 0x0000000000461ef2 in xio_send ()

6 0x00000000004728f8 in sendPacketItem ()

7 0x0000000000474584 in mqtt_client_publish ()

8 0x000000000045aa57 in publish_mqtt_telemetry_msg ()

9 0x000000000045ed3b in IoTHubTransport_MQTT_Common_DoWork ()

10 0x00000000004588cb in IoTHubTransportMqtt_DoWork ()

11 0x0000000000456dc2 in IoTHubClient_LL_DoWork ()

12 0x0000000000419234 in iothub_client_event_dispatch (

13 ....calling code from app

```

[kiranpradeep]

I can't rule out something my end,

I could reproduce the segfault happened for @ollyw on an Ubuntu 14.04 LTS. It is not specific to the machine.

[kevin0402]

@tameraw and @jebrando

Any progress in investigation ?
Can you reproduce this issue ?

[ollyw]

+1 for a progress report / fix

[jebrando]

Just to ensure I have the issues that I'm looking into.

1. The MQTT client takes 15 min to detect a socket disconnect.
2. The SDK does not connect successfully once the network is reconnected.

This is happening on Debian 8, kernel version 4.1.

I will look into this tonight and this weekend and report my findings (Hopefully with a fix).

[jebrando]

Ok, I believe I know what's going on. The OS does not signal that the network is down to the app on a socket level, so the sdk thinks that it is successfully sending the message. The SDK does not send MQTT keepAlive message since normal publish messages are being sent every few seconds, so it does not detect the a missed PINGRESP. There is a few ways to fix this, and I'll discuss with product owners how they would like to see this resolved.

As far as the reconnect is concerned I believe that the network is in a bad state and fixing the first issue should resolve the reconnect. I will talk to the correct people and report back.

Cheers.
Jelani

[tameraw]

@kevin0402 - Work in progress to provide a solution ASAP.

[tameraw]

@kevin0402 - This has been fixed and released. Let us know if still having any issues.

[kiranpradeep]

@tameraw Thanks for this fix. Usually a bug closure will have reference to a commit or vice versa. Could you please refer it here ?

[enuysa-msft]

@kiranpradeep
https://github.com/Azure/azure-iot-sdk-c/commit/d362248b17cbbd3a31a2c96f6acd802b5c5eb739 is commit id for fix provided by Azure IoT Hub C client SDK developer for this issue.