Spring-boot: RedisHealthIndicator failed，when add a slaver to redis cluster

[dyc87112]

Hi，I use spring-boot-starter-data-redis to connect redis cluster. When I use command redis-trib.rb add-node --slave 127.0.0.1:7006 127.0.0.1:7000 add a slave, RedisHealthIndicator check health failed：

2017-08-11 14:58:02,899 WARN  [http-nio-10020-exec-3] org.springframework.boot.actuate.health.RedisHealthIndicator - Health check failed
java.lang.IllegalArgumentException: Node 127.0.0.1:7006 is unknown to cluster
    at org.springframework.data.redis.connection.jedis.JedisClusterConnection$JedisClusterNodeResourceProvider.getResourceForSpecificNode(JedisClusterConnection.java:4133)
    at org.springframework.data.redis.connection.jedis.JedisClusterConnection$JedisClusterNodeResourceProvider.getResourceForSpecificNode(JedisClusterConnection.java:4107)
    at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:145)
    at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnSingleNode(ClusterCommandExecutor.java:128)
    at org.springframework.data.redis.connection.ClusterCommandExecutor.executeCommandOnArbitraryNode(ClusterCommandExecutor.java:116)
    at org.springframework.data.redis.connection.jedis.JedisClusterConnection.clusterGetClusterInfo(JedisClusterConnection.java:3955)

application.properties：

spring.redis.cluster.nodes=localhost:7000,localhost:7001,localhost:7002
spring.redis.pool.max-wait=-1
spring.redis.pool.min-idle=8
spring.redis.pool.max-idle=100
spring.redis.pool.max-active=1000

redis cluster status

C:\redis-cluster>redis-trib.rb check 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 8950ba4f601b268a3b1a3ce2d29eeb455e05fa09 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
M: 9dfec9c2dc07756e96a299f7ac8f1b2324c7ad42 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 6c4366629841e68839b1a6c143a0cadc04c92530 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 8950ba4f601b268a3b1a3ce2d29eeb455e05fa09
S: 5bfc4589811c7ee30fce574c07538dfc915f8526 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 9dfec9c2dc07756e96a299f7ac8f1b2324c7ad42
M: e6ac116ccb06dba47ebb1477c2f5dfdd1b2349f5 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 49c8248fef51b9c22ad58868410ce3347f2eedf4 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 8950ba4f601b268a3b1a3ce2d29eeb455e05fa09
S: a4bed00362d785fa46b9680a2c0a8e5aa5cdf136 127.0.0.1:7005
   slots: (0 slots) slave
   replicates e6ac116ccb06dba47ebb1477c2f5dfdd1b2349f5
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

[wilkinsona]

I don't think this has anything to do with Spring Boot itself. The health indicator just creates a connection:

RedisConnection connection = RedisConnectionUtils
        .getConnection(this.redisConnectionFactory);

And then, if it's a cluster connection, gets the cluster info:

ClusterInfo clusterInfo = ((RedisClusterConnection) connection)
        .clusterGetClusterInfo();

You haven't provided much of a description about the nature of the failure. For example, does the health check fail once and then start working, fail continually, only fail if it's called immediately after adding the node to the cluster, something else?

My suspicion is that either Jedis can't cope with dynamically adding a node to the cluster, or there's a race between the node being added to the cluster and Jedis' view of the cluster catching up with that addition.

I think you'd be better bringing this to the attention of the Spring Data team. Either by raising a Spring Data Redis issue or by chatting with them on Gitter. Whichever approach you take, please take the time to provide a detailed description of how to reproduce the problem.

If it turns out that a change is needed in Spring Boot, we can re-open this issue.

[dyc87112]

Sorry，spring-data-redis project has no way to create an issue...

[wilkinsona]

They use JIRA.

[dyc87112]

Thanks!

[rud]

Link to the JIRA issue, for completeness: https://jira.spring.io/browse/DATAREDIS-679