Hazelcast: Extensible strategies for client reconnection

[thekalinga]

Hi,

Currently, if a client gets disconnected from the cluster, hazelcast retries based on the connectionAttemptLimit & connectionAttemptPeriod property of ClientConfig

The problem with this approach is, it disconnects after the specified limit. Currently there is no way to specify custom strategies, such as retry connecting forever till cluster is visible/exponential backoff reconnects.

It would be nice to have the user specify custom retry strategy.

Here is my sample implementation

interface ClientReconnectStrategy {
  boolean shouldRetry(long previousAttemptCount, long lastRetriedAt);
  long nextRetryAt(long previousAttemptCount, long lastRetriedAt); // or long can be replaced with LocalDateTime/Date
}

class CountAwareFixedDelayClientReconnectStrategy implements ClientReconnectStrategy {

  private long maxConnectionAttemptLimit;
  private long connectionAttemptPeriodInMillis;

  public CountAwareFixedDelayClientReconnectStrategy(long maxConnectionAttemptLimit, int connectionAttemptPeriodInMillis) {
    this.maxConnectionAttemptLimit = maxConnectionAttemptLimit;
    this.connectionAttemptPeriodInMillis = connectionAttemptPeriodInMillis;
  }

  public boolean shouldRetry(long previousAttemptCount, long lastRetriedAt) {
    return previousAttemptCount < maxConnectionAttemptLimit;
  }

  public long nextRetryAt(long previousAttemptCount, long lastRetriedAt) {
    return Clock.currentTimeMillis() + connectionAttemptPeriod;
  }

}

Existing ClusterListenerSupport.connectToCluster() can be replaced with

  public void connectToCluster() throws Exception {
    ownerConnectionAddress = null;

    final ClientNetworkConfig networkConfig = client.getClientConfig().getNetworkConfig();
    final int connAttemptLimit = networkConfig.getConnectionAttemptLimit();
    final int connectionAttemptPeriod = networkConfig.getConnectionAttemptPeriod();

    final int connectionAttemptLimit = connAttemptLimit == 0 ? Integer.MAX_VALUE : connAttemptLimit;

    Set<InetSocketAddress> triedAddresses = new HashSet<InetSocketAddress>();
    long previousRetryCount = 0;
    long lastRetriedAt = -1; // -1 If previousRetryCount is zero, it does not matter what we set for lastRetriedAt
    while (reconnectStrategy.shouldRetry(previousRetryCount, lastRetriedAt)) {
        if (!client.getLifecycleService().isRunning()) {
            if (logger.isFinestEnabled()) {
                logger.finest("Giving up on retrying to connect to cluster since client is shutdown");
            }
            break;
        }

        lastRetriedAt = Clock.currentTimeMillis();
        boolean isConnected = connect(triedAddresses);

        if (isConnected) {
            return;
        }
        previousAttemptCount++;
        long nextTryAt = reconnectStrategy.nextRetryAt(previousAttemptCount, lastRetriedAt);
        long remainingTime = nextTryAt - Clock.currentTimeMillis();
        logger.warning(
                String.format("Unable to get alive cluster connection, try in %d ms later, attempt #%d",
                        Math.max(0, remainingTime), previousAttemptCount));

        if (remainingTime > 0) {
            try {
                Thread.sleep(remainingTime);
            } catch (InterruptedException e) {
                break;
            }
        }
    }
    throw new IllegalStateException("Unable to connect to any address in the config! "
            + "The following addresses were tried:" + triedAddresses);
  }

[nlwillia]

It would also be useful to be able to configure different parameters depending on whether this is the first connection attempt or a reconnect. The first connect might be a check-if-it's-there-else-fallback strategy while a reconnect might prefer to block until it's available again. The only way to do that now is to connect with fail-fast parameters and if it succeeds, immediately disconnect then reconnect with parameters geared towards longer term stability.

edit: It would also be useful to be able to configure different behavior for operations than for the client connection itself. I'd generally like for the client connection to keep trying indefinitely, but that doesn't mean I want operations to block forever. I can configure the client to eventually shutdown, but that requires more diligence at the code level to deal with handles that could permanently expire (HazelcastClientNotActiveException) between uses.

[dbrimley]

Hi,

As of 3.9 we now have client connection strategies which allow for non blocking during disconnect.
http://docs.hazelcast.org/docs/3.9.2/manual/html-single/index.html#configuring-client-connection-strategy

I'd be interested if this covers your use case?

We are introducing exponential back-off of retries in 3.11 (summer 2018).
We're also considering exposing the connection retry interface that is part of client connection strategies.

[nlwillia]

The strategy approach will address most cases, but I think there will still be situations where it's useful to be able to plug in custom logic, which sounds like where you're going. Connection config is entirely static right now, so the only way to adapt is to leave and rejoin with different parameters which is awkward.

[mmedenjak]

Hi all, Hazelcast 3.11 has been released and you can now use the exponential backoff client reconnect strategy: https://docs.hazelcast.org/docs/latest/manual/html-single/#configuring-client-connection-retry.