Redis: Redis Cluster in broken state after resharding

[eitijupaenoithoowohd]

I have a cluster based on Redis 3.0.7 and have attempted to reshard it with the script ./redis-trib.rb from all of the three versions: 3.0.7, 3.2 and last unstable. Results were the same in any combinations:
"[ERR] ERR syntax error" or
"[ERR] Calling MIGRATE: ERR syntax error"

I have found issues like: #1616 but it have no useful solution to make resharding correctly.

./redis-trib.rb reshard --timeout 60000 10.60.42.33:6000

Performing Cluster Check (using node 10.60.42.33:6000)
M: 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0 10.60.42.33:6000
slots:5468-10978 (5511 slots) master
2 additional replica(s)
S: 0cfdcc039f6cf5bd8ce4d98a7760cd74e5ae90a0 10.60.41.220:6002
slots: (0 slots) slave
replicates 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
S: c02254dc537643e87abe3074cdde3a10b3baf846 10.60.41.220:6001
slots: (0 slots) slave
replicates 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
S: 721d5a3b02b8b8db38c375bc2333c4c2140a0527 10.60.42.33:6001
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
S: cf2711f88d22110835a7b7f28675fbb7b7647247 10.60.42.33:6002
slots: (0 slots) slave
replicates 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71
S: 93927624b09be7b45c09e07852d5eca9a4de6a4f 10.60.44.98:6002
slots: (0 slots) slave
replicates 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71
S: 3ec89bf8d686693d0c48832c8e55ab4b488d7805 10.60.44.98:6001
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
M: 2ad5df37693741de21d7a230e5eb7cd9ee721ed8 10.60.41.220:6000
slots:63-5467 (5405 slots) master
3 additional replica(s)
S: 3a0c1b5ff7f4c8ccdd43b629ac9b0e4e154d5eb5 10.60.43.115:6379
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
M: 953d1819ec96d6cc9255e1ea762e3eb48f21fa57 10.60.41.79:6379
slots:0-62 (63 slots) master
0 additional replica(s)
M: 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71 10.60.44.98:6000
slots:10979-16383 (5405 slots) master
2 additional replica(s)
[OK] All nodes agree about slots configuration.
Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all

Ready to move 4096 slots.
Source nodes:
M: 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0 10.60.42.33:6000
slots:5468-10978 (5511 slots) master
2 additional replica(s)
M: 953d1819ec96d6cc9255e1ea762e3eb48f21fa57 10.60.41.79:6379
slots:0-62 (63 slots) master
0 additional replica(s)
M: 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71 10.60.44.98:6000
slots:10979-16383 (5405 slots) master
2 additional replica(s)
Destination node:
M: 2ad5df37693741de21d7a230e5eb7cd9ee721ed8 10.60.41.220:6000
slots:63-5467 (5405 slots) master
3 additional replica(s)
Resharding plan:
Moving slot 5468 from 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
Moving slot 5469 from 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
...
...
Moving slot 21 from 953d1819ec96d6cc9255e1ea762e3eb48f21fa57
Moving slot 22 from 953d1819ec96d6cc9255e1ea762e3eb48f21fa57
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 5468 from 10.60.42.33:6000 to 10.60.41.220:6000:
[ERR] ERR syntax error
52034@P0021:~/data/programs/src/redis-3.0.7/src$ ./redis-trib.rb fix 10.60.41.79:6379>>> Performing Cluster Check (using node 10.60.41.79:6379)
M: 953d1819ec96d6cc9255e1ea762e3eb48f21fa57 10.60.41.79:6379
slots:0-62 (63 slots) master
0 additional replica(s)
M: 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71 10.60.44.98:6000
slots:10979-16383 (5405 slots) master
2 additional replica(s)
S: 3a0c1b5ff7f4c8ccdd43b629ac9b0e4e154d5eb5 10.60.43.115:6379
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
S: 0cfdcc039f6cf5bd8ce4d98a7760cd74e5ae90a0 10.60.41.220:6002
slots: (0 slots) slave
replicates 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
S: 721d5a3b02b8b8db38c375bc2333c4c2140a0527 10.60.42.33:6001
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
S: 3ec89bf8d686693d0c48832c8e55ab4b488d7805 10.60.44.98:6001
slots: (0 slots) slave
replicates 2ad5df37693741de21d7a230e5eb7cd9ee721ed8
M: 2ad5df37693741de21d7a230e5eb7cd9ee721ed8 10.60.41.220:6000
slots:63-5467 (5405 slots) master
3 additional replica(s)
S: 93927624b09be7b45c09e07852d5eca9a4de6a4f 10.60.44.98:6002
slots: (0 slots) slave
replicates 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71
M: 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0 10.60.42.33:6000
slots:5468-10978 (5511 slots) master
2 additional replica(s)
S: cf2711f88d22110835a7b7f28675fbb7b7647247 10.60.42.33:6002
slots: (0 slots) slave
replicates 6338c8642e0c27d3ecd6050292e1e4d56cd3cc71
S: c02254dc537643e87abe3074cdde3a10b3baf846 10.60.41.220:6001
slots: (0 slots) slave
replicates 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
[OK] All nodes agree about slots configuration.

Check for open slots...
[WARNING] Node 10.60.41.220:6000 has slots in importing state (5468).
[WARNING] Node 10.60.42.33:6000 has slots in migrating state (5468).
[WARNING] The following slots are open: 5468
Fixing open slot 5468
Set as migrating in: 10.60.42.33:6000
Set as importing in: 10.60.41.220:6000
Moving slot 5468 from 10.60.42.33:6000 to 10.60.41.220:6000:
[ERR] ERR syntax error

[eitijupaenoithoowohd]

I'm doing manually all that user's manual said: http://redis.io/topics/cluster-spec
redis-cli -v
redis-cli 3.0.7

redis-cli -h 10.60.41.79 -p 6379 CLUSTER SETSLOT 5468 IMPORTING 953d1819ec96d6cc9255e1ea762e3eb48f21fa57
OK
redis-cli -h 10.60.42.33 -p 6000 CLUSTER SETSLOT 5468 MIGRATING 7743c9fda07c6818e8d0cc8189f58d0cab47c4f0
OK
redis-cli -h 10.60.42.33 -p 6000 CLUSTER GETKEYSINSLOT 5468 1000
1) "p6459f9617d9dfe97d776bfd50a984f3c19f012feM"
redis-cli -h 10.60.42.33 -p 6000 MIGRATE 10.60.41.79 6379 "" 0 6000 KEYS p6459f9617d9dfe97d776bfd50a984f3c19f012feM
(error) ERR syntax error

What am I doing wrong?

[mjh1]

I seem to have the same issue, I'm testing with latest stable, 4.0.1.

• I create a cluster of 3 nodes
• add one item to the cluster
• add a fourth node
• when i try to reshard to balance the keys across all 4, I get the error:

Moving slot 12182 from 127.0.0.1:7002 to 127.0.0.1:7003:
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

Commands used (as in https://redis.io/topics/cluster-tutorial using port number as directory name):

cd 7000
~/Downloads/redis-4.0.1/src/redis-server redis.conf &
cd ../7001
~/Downloads/redis-4.0.1/src/redis-server redis.conf &
cd ../7002
~/Downloads/redis-4.0.1/src/redis-server redis.conf &
cd ../7003
~/Downloads/redis-4.0.1/src/redis-server redis.conf &
~/Downloads/redis-4.0.1/src/redis-trib.rb create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002
~/Downloads/redis-4.0.1/src/redis-cli -c -p 7000 set foo bar
~/Downloads/redis-4.0.1/src/redis-trib.rb add-node 127.0.0.1:7003 127.0.0.1:7000
~/Downloads/redis-4.0.1/src/redis-trib.rb reshard --from all --to XXXX --slots 4096 --yes 127.0.0.1:7000

If i try the above with an empty redis (leaving out the set command) it works.

@antirez is there something i'm doing wrong here?

[mjh1]

From https://github.com/antirez/redis/issues/4272 it seems the problem is coming from the particular version of the redis gem i was using (4.0.0), I've switched to using an older version of the gem for the time being.

[antirez]

That's one of the many reasons redis-trib in Redis 4.2 will be C-coded and part of redis-cli. The ruby ecosystem is not focused on reliability / stability enough IMHO to write system programming tools in Ruby easily. Evolution of software projects even breaking compatibility with the past is taken in higher consideration, which is good in certain ways, bad in others.