Azure-iot-sdk-csharp: Not reconnecting after IoT Hub failover when AmqpConnectionPoolSettings.Pooling is enabled

[martin-weber]

Description of the issue:

When using the DeviceClient with AmqpConnectionPoolSettings.Pooling enabled, the DeviceClient does not reconnect after a manual initiated failover of the IoT Hub.

With AmqpConnectionPoolSettings.Pooling disabled (= false) or other TransportType settings such as Mqtt the DeviceClient reconnects after failover and successfully continues sending messages.

Code sample exhibiting the issue:

The nuget package "Microsoft.Azure.Devices.Client" Version="1.23.2" was used to verify the behavior.

Initialization of the DeviceClient with AmqpConnectionPoolSettings.Pooling enabled (= true) leads to the error:

_deviceClient = DeviceClient.CreateFromConnectionString(
    _connectionString,
    new ITransportSettings[]
    {
        new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
        {
            AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
            {
                Pooling = true,
            }
        }
    });

Initialization of the DeviceClient with AmqpConnectionPoolSettings.Pooling disabled (=false), properly reconnects after the failover finished:

_deviceClient = DeviceClient.CreateFromConnectionString(
    _connectionString,
    new ITransportSettings[]
    {
        new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
        {
            AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
            {
                Pooling = false,
            }
        }
    });

Console log of the issue:

Calling await _deviceClient.SendEventAsync(message); throws during and then after the failover forever on each call an IotHubException with an inner exception of type AmqpException with message An AMQP error occurred (condition='com.microsoft:iot-hub-not-found-error'):

03/04/2020 12:10:35: IotHubException: error(condition:com.microsoft:iot-hub-not-found-error,info:[com.microsoft:tracking-id:9ca84927533546abb68ad3da7b6a0cf4-G:20-TimeStamp:04/03/2020 10:10:34,com.microsoft:is-filtered:True])
        03/04/2020 12:10:35: AmqpException: An AMQP error occurred (condition='com.microsoft:iot-hub-not-found-error').

AB#7253018

[abhipsaMisra]

iot-hub-not-found-error is an error that is thrown directly from the service when they are unable to connect to the specified iot hub instance. Please open a support ticket here, with relevant details, so that we can debug this from service-side.

[martin-weber]

@abhipsaMisra (or @barustum) : Not sure if its a problem of the service, as other implementations such as MQTT but without AMQP connection pooling can handle it and reconnect after the failover finished.

I'm not sure I understand the reason, why this must be handled on the service side.
Can you please give me some more details for the support ticket, why this can and should be handled by the service?

[abhipsaMisra]

@martin-weber You are correct, this is being handled in other protocols, and with pooling turned off for AMQP. This would indicate that there might be some difference in the request reaching the service for pooled AMQP connections, which is causing failover operation to fail. In order to investigate this we would need to track the request and response from the service, which requires information regarding your hub/ region etc. It is very possible that the SDK might not be sending the request as expected, causing service to throw iot-hub-not-found-error, but we would need to investigate this with service traces to root-cause the issue.

[martin-weber]

Please open a support ticket here, with relevant details, so that we can debug this from service-side.

@abhipsaMisra : Added a support ticket.

[abhipsaMisra]

@martin-weber Thanks. I'll keep this issue open so that we can update the finding/ resolution here as well.

[abhipsaMisra]

@martin-weber Were you able to get any traction on this issue via the support ticket created?

[martin-weber]

@abhipsaMisra : I added you to the internal conversation.

[abhipsaMisra]

The issue this fails over AMQP is because of the way the SDK handles the connection underneath for pooled scenarios. The client SDK does not initiate closure of the connection until every client instance on that connection has been disposed.
As a solution to this, we plan to map this error code to a particular exception type which will clean the closest upper layer before starting a reconnection. What this would mean is that, when session errors, it will trigger the closure of the connection, and the subsequent reconnection will trigger a fresh DNS lookup.
The concern about kicking of other devices multiplexed over the same connection is not a concern here, since every device would eventually receive an iot-hub-not-found-error on communicating with the older hub, so a fresh connection will bring them all back.

[az-iot-builder-01]

@abhipsaMisra, @martin-weber, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

[abhipsaMisra]

The fix has been checked into the master branch. I'll update this thread once we have a release out.

[abhipsaMisra]

I still see an issue where multiplexed devices don't recover immediately. I am investigating this issue; in the meanwhile, I'll remove the fix-checked-in label until I have more information.

[abhipsaMisra]

The current behavior from service does not allow for a reliable way for the SDK to retry on failover scenarios. I am working with the service team to establish the expected service behavior, irrespective of the transport protocol used/ multiplexing capabilities enabled. I'll update this thread as I get updates from them.

[abhipsaMisra]

The service currently returns the same error code for the scenario "iot hub does not exist" and "iot hub is not available since it is failing over". This complicates any retry logic that can be added into the SDK/ application. There is also a known issue where the hub might return the old IP address of the previously failed over hub, when doing a fresh DNS resolution. We are waiting on some clarification from the service team around these issues; I'll keep this thread updated.

[martin-weber]

@abhipsaMisra is there any work going on or should we close the issue?

[abhipsaMisra]

@martin-weber Unfortunately this issue has currently been placed on the backlog for the IoT Hub service team, due to other prioritization issues. There is a work item scheduled to update the entire failover-fallback implementation; however, I do not have an estimate for by when it will be done.
Until the service team addresses this issue at their end, there isn't anything the client library can do here.

If you are interested, I can provide you with some contacts on the service team, with whom you could follow this up with; and that might help you get some traction on this.

[martin-weber]

Thank you, @abhipsaMisra . This answer is fine for me. Just wanted to ask if it makes sense to close the issue here. In that case, I let it open.