Hazelcast: Joining members cannot form cluster when existing members are killed

Created on 26 Oct 2018 · 6Comments · Source: hazelcast/hazelcast

Config :

Config config = new Config();
JoinConfig join = config.getNetworkConfig().getJoin();
join.getMulticastConfig().setEnabled(false);
join.getAwsConfig().setEnabled(false);
join.getTcpIpConfig().setEnabled(true);
join.getTcpIpConfig().addMember("127.0.0.1:5701");
join.getTcpIpConfig().addMember("127.0.0.1:5702");
join.getTcpIpConfig().addMember("127.0.0.1:5703");
join.getTcpIpConfig().addMember("127.0.0.1:5704");

Member A and Member B up and running,
Start Member C and Member D
Kill Member A and B before C and D actually join
C and D will not form cluster

Easily reproducible on my local.

@Danny-Hazelcast also reproduced the issue here :
https://hazelcast-l337.ci.cloudbees.com/view/kill/job/cluster-formation/4/console

Logs from Member C like :

FINE: [127.0.0.1]:5703 [dev] [3.11] Will send master question to each address in: [[127.0.0.1]:5702, [127.0.0.1]:5704, [127.0.0.1]:5701]
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {[127.0.0.1]:5704=false}
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {[127.0.0.1]:5704=false}
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:46 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:47 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:47 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpAcceptor
FINE: [127.0.0.1]:5703 [dev] [3.11] Accepting socket connection from /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/127.0.0.1:5703->/127.0.0.1:60399}.inboundPipeline, NioChannel{/127.0.0.1:5703->/127.0.0.1:60399}.outboundPipeline
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Established socket connection between /127.0.0.1:5703 and /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Initialized new cluster connection between /127.0.0.1:5703 and /127.0.0.1:60399
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINEST: [127.0.0.1]:5703 [dev] [3.11] Binding Connection[id=3, /127.0.0.1:5703->/127.0.0.1:60399, endpoint=null, alive=true, type=MEMBER] to [127.0.0.1]:5704, reply is true
Oct 26, 2018 9:50:47 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5704 is removed from the blacklist.
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
FINEST: [127.0.0.1]:5703 [dev] [3.11] Sending bind packet to [127.0.0.1]:5704
Oct 26, 2018 9:50:47 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] Resetting connection monitor for endpoint [127.0.0.1]:5704
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Connection[id=2, /0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701, endpoint=[127.0.0.1]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
java.io.EOFException: Remote socket closed!
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:116)
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:368)
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:353)
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:279)
    at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:234)

Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.io.EOFException {Remote socket closed!}, Error-Count: 1
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Removed connection to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] NOT sending master question to blacklisted endpoints: {}
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5702
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5704
Oct 26, 2018 9:50:48 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending master question to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:53193->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:53193->null}.outboundPipeline
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
FINE: [127.0.0.1]:5703 [dev] [3.11] Handling master response [127.0.0.1]:5701 from [127.0.0.1]:5702
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Setting master address to [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.cluster.impl.ClusterJoinManager
WARNING: [127.0.0.1]:5703 [dev] [3.11] Could not create connection to possible master [127.0.0.1]:5701
Oct 26, 2018 9:50:48 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:47511->/127.0.0.1:5701}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:53193->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:53193->null}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:49 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 2
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [127.0.0.1]:5703 [dev] [3.11] Connection[id=1, /0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702, endpoint=[127.0.0.1]:5702, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
java.io.EOFException: Remote socket closed!
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:116)
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:368)
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:353)
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:279)
    at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:234)

Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5702 Cause => java.io.EOFException {Remote socket closed!}, Error-Count: 1
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Removed connection to [127.0.0.1]:5702
Oct 26, 2018 9:50:49 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:39283->/127.0.0.1:5702}.outboundPipeline
Oct 26, 2018 9:50:49 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:49 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:49 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true


---------------------------------


FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:54 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:54 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Joining to master [127.0.0.1]:5701
Oct 26, 2018 9:50:54 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:54 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:54 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:50683->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:50683->null}.outboundPipeline
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:55 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:50683->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:50683->null}.outboundPipeline
Oct 26, 2018 9:50:55 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 8
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5703 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 9
Oct 26, 2018 9:50:55 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:55 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:55 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5703 [dev] [3.11] Joining to master [127.0.0.1]:5701
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:55 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:55 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:58555->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:58555->null}.outboundPipeline
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5703 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:56 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5703 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:58555->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:58555->null}.outboundPipeline
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5703 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:56 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5703 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5703 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 9
Oct 26, 2018 9:50:56 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5703 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 10
Oct 26, 2018 9:50:56 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5703 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:56 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5703 [dev] [3.11] Scanning all invocations

Logs from Member D like :

INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:50 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:44785->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:44785->null}.outboundPipeline
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:44785->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:44785->null}.outboundPipeline
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:51 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5704 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5704 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 3
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.cluster.impl.operations.JoinMastershipClaimOp
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending 'false' for master claim of node: [127.0.0.1]:5703
Oct 26, 2018 9:50:51 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5704 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:51 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:51 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:51 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:33433->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:33433->null}.outboundPipeline
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:52 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:33433->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:33433->null}.outboundPipeline
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701


-----------------------------------


FINE: [127.0.0.1]:5704 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.
Oct 26, 2018 9:50:52 AM com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor
FINEST: [127.0.0.1]:5704 [dev] [3.11] Scanning all invocations
Oct 26, 2018 9:50:52 AM com.hazelcast.cluster.impl.TcpIpJoiner
FINE: [127.0.0.1]:5704 [dev] [3.11] Sending join request to [127.0.0.1]:5701
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Starting to connect to [127.0.0.1]:5701
Oct 26, 2018 9:50:52 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Connecting to /127.0.0.1:5701, timeout: 0, bind-any: true
Oct 26, 2018 9:50:52 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Adding pipelines: NioChannel{/0:0:0:0:0:0:0:0:32933->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:32933->null}.outboundPipeline
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [127.0.0.1]:5704 [dev] [3.11] Could not connect to: /127.0.0.1:5701. Reason: SocketException[Connection refused to address /127.0.0.1:5701]
Oct 26, 2018 9:50:53 AM com.hazelcast.internal.networking.nio.iobalancer.IOBalancer
FINEST: [127.0.0.1]:5704 [dev] [3.11] Removing pipelines: NioChannel{/0:0:0:0:0:0:0:0:32933->null}.inboundPipeline, NioChannel{/0:0:0:0:0:0:0:0:32933->null}.outboundPipeline
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnector
FINEST: [127.0.0.1]:5704 [dev] [3.11] Connection refused to address /127.0.0.1:5701
java.net.SocketException: Connection refused to address /127.0.0.1:5701
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at com.hazelcast.internal.networking.AbstractChannel.connect(AbstractChannel.java:129)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.tryToConnect(TcpIpConnector.java:179)
    at com.hazelcast.nio.tcp.TcpIpConnector$ConnectTask.run(TcpIpConnector.java:113)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Oct 26, 2018 9:50:53 AM com.hazelcast.cluster.impl.TcpIpJoiner
INFO: [127.0.0.1]:5704 [dev] [3.11] [127.0.0.1]:5701 is added to the blacklist.
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
FINEST: [127.0.0.1]:5704 [dev] [3.11] An error occurred on connection to [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 5
Oct 26, 2018 9:50:53 AM com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [127.0.0.1]:5704 [dev] [3.11] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 6
Oct 26, 2018 9:50:53 AM com.hazelcast.internal.cluster.ClusterService
FINE: [127.0.0.1]:5704 [dev] [3.11] Cannot suspect [127.0.0.1]:5701, since it's not a member.

High Internal Core Defect

Source