Keda: Redis Cluster e2e are unstable

Created on 8 Jan 2021 · 9Comments · Source: kedacore/keda

Redis Cluster e2e are unstable, most of the e2e tests run in parallel, so it might be worth checking whether there is not a problem. See https://github.com/kedacore/keda/runs/1668376800?check_suite_focus=true or nightly runs.

Logs of redis-cluster-streams test:

2021-01-08T12:10:20.0827162Z >>> scalers/redis-cluster-streams.test.ts.log <<<
2021-01-08T12:10:20.0841853Z 
2021-01-08T12:10:20.0845915Z namespace/redis-cluster created
2021-01-08T12:10:20.0846647Z "bitnami" has been added to your repositories
2021-01-08T12:10:20.0847428Z NAME: redis-cluster
2021-01-08T12:10:20.0847989Z LAST DEPLOYED: Fri Jan  8 12:04:01 2021
2021-01-08T12:10:20.0848719Z NAMESPACE: redis-cluster
2021-01-08T12:10:20.0849274Z STATUS: deployed
2021-01-08T12:10:20.0849744Z REVISION: 1
2021-01-08T12:10:20.0850198Z TEST SUITE: None
2021-01-08T12:10:20.0857494Z NOTES:
2021-01-08T12:10:20.0858138Z ** Please be patient while the chart is being deployed **
2021-01-08T12:10:20.0858593Z 
2021-01-08T12:10:20.0858840Z 
2021-01-08T12:10:20.0859286Z To get your password run:
2021-01-08T12:10:20.0860787Z     export REDIS_PASSWORD=$(kubectl get secret --namespace redis-cluster redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
2021-01-08T12:10:20.0861666Z 
2021-01-08T12:10:20.0862965Z You have deployed a Redis Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis cluster:
2021-01-08T12:10:20.0865965Z 
2021-01-08T12:10:20.0866751Z 1. Run a Redis pod that you can use as a client:
2021-01-08T12:10:20.0875291Z kubectl run --namespace redis-cluster redis-cluster-client --rm --tty -i --restart='Never' \
2021-01-08T12:10:20.0876436Z  --env REDIS_PASSWORD=$REDIS_PASSWORD \
2021-01-08T12:10:20.0877440Z --image docker.io/bitnami/redis-cluster:6.0.9-debian-10-r36 -- bash
2021-01-08T12:10:20.0877980Z 
2021-01-08T12:10:20.0878454Z 2. Connect using the Redis CLI:
2021-01-08T12:10:20.0878832Z 
2021-01-08T12:10:20.0879569Z redis-cli -c -h redis-cluster -a $REDIS_PASSWORD
2021-01-08T12:10:20.0880667Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0892705Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0893960Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0898519Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0910293Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0911629Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0912841Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0914039Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0915479Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0917525Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0919744Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0922782Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0925469Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0927094Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0928310Z Error from server (NotFound): statefulsets.apps "redis-cluster" not found
2021-01-08T12:10:20.0929444Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0930512Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0931587Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0932657Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0935105Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0936330Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0938451Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0941049Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0944213Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0945606Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0946681Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0947765Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0948830Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0949901Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0950970Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0953993Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0956595Z namespace/redis-cluster-streams created
2021-01-08T12:10:20.0960217Z secret/redis***
2021-01-08T12:10:20.0962497Z triggerauthentication.keda.sh/keda-redis-stream-triggerauth created
2021-01-08T12:10:20.0964131Z deployment.apps/redis-streams-consumer created
2021-01-08T12:10:20.0965463Z scaledobject.keda.sh/redis-streams-scaledobject created
2021-01-08T12:10:20.0966730Z   ✖ before hook Redis is not in a ready state
2021-01-08T12:10:20.0967892Z scaledobject.keda.sh "redis-streams-scaledobject" deleted
2021-01-08T12:10:20.0970606Z triggerauthentication.keda.sh "keda-redis-stream-triggerauth" deleted
2021-01-08T12:10:20.0973370Z secret "redis-password" deleted
2021-01-08T12:10:20.0976110Z deployment.apps "redis-streams-consumer" deleted
2021-01-08T12:10:20.0978751Z namespace "redis-cluster-streams" deleted
2021-01-08T12:10:20.0980844Z Error from server (NotFound): namespaces "redis-cluster" not found
2021-01-08T12:10:20.0982051Z   ✖ 2 tests remaining in scalers/redis-cluster-streams.test.ts
2021-01-08T12:10:20.0982672Z 
2021-01-08T12:10:20.0983078Z   1 hook failed
2021-01-08T12:10:20.0983371Z 
2021-01-08T12:10:20.0983780Z   before hook
2021-01-08T12:10:20.0984074Z 
2021-01-08T12:10:20.0985063Z   /__w/keda/keda/tests/scalers/redis-cluster-streams.test.ts:34
2021-01-08T12:10:20.0985702Z 
2021-01-08T12:10:20.0986137Z    33:     }                                                            
2021-01-08T12:10:20.0988056Z    34:     t.is('6', redisReplicaCount, 'Redis is not in a ready state')
2021-01-08T12:10:20.0988794Z    35:                                                                  
2021-01-08T12:10:20.0989099Z 
2021-01-08T12:10:20.0989760Z   Redis is not in a ready state
2021-01-08T12:10:20.0990764Z 
2021-01-08T12:10:20.0991645Z   Difference:
2021-01-08T12:10:20.0992706Z 
2021-01-08T12:10:20.0993956Z   - '6'
2021-01-08T12:10:20.0995190Z   + ''

Logs of redis-cluster-lists test:

2021-01-08T12:10:20.1388710Z >>> scalers/redis-cluster-lists.test.ts.log <<<
2021-01-08T12:10:20.1389362Z 
2021-01-08T12:10:20.1390380Z Error from server (AlreadyExists): namespaces "redis-cluster" already exists
2021-01-08T12:10:20.1391418Z "bitnami" already exists with the same configuration, skipping
2021-01-08T12:10:20.1392440Z Error: cannot re-use a name that is still in use
2021-01-08T12:10:20.1393003Z 6
2021-01-08T12:10:20.1393387Z 6
2021-01-08T12:10:20.1393763Z 6
2021-01-08T12:10:20.1394143Z 6
2021-01-08T12:10:20.1394524Z 6
2021-01-08T12:10:20.1394906Z 6
2021-01-08T12:10:20.1395303Z 6
2021-01-08T12:10:20.1395680Z 6
2021-01-08T12:10:20.1396064Z 6
2021-01-08T12:10:20.1396439Z 6
2021-01-08T12:10:20.1396818Z 6
2021-01-08T12:10:20.1397195Z 6
2021-01-08T12:10:20.1397579Z 6
2021-01-08T12:10:20.1397959Z 6
2021-01-08T12:10:20.1398340Z 6
2021-01-08T12:10:20.1398720Z 6
2021-01-08T12:10:20.1399101Z 6
2021-01-08T12:10:20.1399493Z 6
2021-01-08T12:10:20.1399870Z 6
2021-01-08T12:10:20.1400257Z 6
2021-01-08T12:10:20.1400639Z 6
2021-01-08T12:10:20.1401020Z 6
2021-01-08T12:10:20.1401413Z 6
2021-01-08T12:10:20.1401854Z 6
2021-01-08T12:10:20.1402237Z 6
2021-01-08T12:10:20.1402616Z 6
2021-01-08T12:10:20.1402998Z 6
2021-01-08T12:10:20.1403393Z 6
2021-01-08T12:10:20.1403772Z 6
2021-01-08T12:10:20.1404152Z 6
2021-01-08T12:10:20.1404541Z 10.0.114.99
2021-01-08T12:10:20.1405445Z namespace/redis-cluster-lists-test created
2021-01-08T12:10:20.1406918Z secret/redis***
2021-01-08T12:10:20.1408332Z triggerauthentication.keda.sh/keda-redis-cluster-list-triggerauth created
2021-01-08T12:10:20.1409726Z secret/redis-config created
2021-01-08T12:10:20.1411346Z triggerauthentication.keda.sh/keda-redis-cluster-list-triggerauth-host-port created
2021-01-08T12:10:20.1413191Z deployment.apps/redis-worker-test-hostport created
2021-01-08T12:10:20.1414731Z scaledobject.keda.sh/redis-worker-test-hostport created
2021-01-08T12:10:20.1416138Z deployment.apps/redis-worker-test-address created
2021-01-08T12:10:20.1417485Z scaledobject.keda.sh/redis-worker-test-address created
2021-01-08T12:10:20.1419047Z deployment.apps/redis-worker-test-hostport-triggerauth created
2021-01-08T12:10:20.1420813Z scaledobject.keda.sh/redis-worker-test-hostport-triggerauth created
2021-01-08T12:10:20.1422242Z   ✖ before hook creating a Redis cluster should work.
2021-01-08T12:10:20.1423477Z Error from server (NotFound): jobs.batch "redis-writer-host-port-ref" not found
2021-01-08T12:10:20.1424887Z Error from server (NotFound): jobs.batch "redis-writer-address-ref" not found
2021-01-08T12:10:20.1426446Z Error from server (NotFound): jobs.batch "redis-writer-host-port-trigger-auth" not found
2021-01-08T12:10:20.1428012Z scaledobject.keda.sh "redis-worker-test-hostport" deleted
2021-01-08T12:10:20.1429423Z scaledobject.keda.sh "redis-worker-test-address" deleted
2021-01-08T12:10:20.1431026Z scaledobject.keda.sh "redis-worker-test-hostport-triggerauth" deleted
2021-01-08T12:10:20.1432952Z Error from server (NotFound): triggerauthentications.keda.sh "keda-redis-list-triggerauth" not found
2021-01-08T12:10:20.1435110Z Error from server (NotFound): triggerauthentications.keda.sh "keda-redis-list-triggerauth-host-port" not found
2021-01-08T12:10:20.1437141Z deployment.apps "redis-worker-test-address" deleted
2021-01-08T12:10:20.1438664Z deployment.apps "redis-worker-test-hostport-triggerauth" deleted
2021-01-08T12:10:20.1441041Z deployment.apps "redis-worker-test-hostport" deleted
2021-01-08T12:10:20.1442243Z secret "redis-password" deleted
2021-01-08T12:10:20.1443206Z namespace "redis-cluster-lists-test" deleted
2021-01-08T12:10:20.1444161Z namespace "redis-cluster" deleted
2021-01-08T12:10:20.1445222Z   ✖ 6 tests remaining in scalers/redis-cluster-lists.test.ts
2021-01-08T12:10:20.1445808Z 
2021-01-08T12:10:20.1446215Z   1 hook failed
2021-01-08T12:10:20.1446503Z 
2021-01-08T12:10:20.1446905Z   before hook
2021-01-08T12:10:20.1447194Z 
2021-01-08T12:10:20.1448087Z   /__w/keda/keda/tests/scalers/redis-cluster-lists.test.ts:31
2021-01-08T12:10:20.1448674Z 
2021-01-08T12:10:20.1449633Z    30:     sh.exec(`helm repo add bitnami https://charts.bitnami.com/bitnami`)  
2021-01-08T12:10:20.1450521Z    31:     t.is(0,                                                              
2021-01-08T12:10:20.1451558Z    32:       sh.exec(`helm install ${redisClusterName} --namespace ${redisNames…
2021-01-08T12:10:20.1452132Z 
2021-01-08T12:10:20.1452636Z   creating a Redis cluster should work.
2021-01-08T12:10:20.1453031Z 
2021-01-08T12:10:20.1453450Z   Difference:
2021-01-08T12:10:20.1453755Z 
2021-01-08T12:10:20.1454253Z   - 0
2021-01-08T12:10:20.1454826Z   + 1

bug testing

Source

zroubalik

All 9 comments

CC @goku321

zroubalik on 8 Jan 2021

👍1

@zroubalik Thank you for providing the details. I will start on this.

goku321 on 8 Jan 2021

@goku321 thanks!

zroubalik on 8 Jan 2021

The stability has definitely improved, but redis-cluster-streams seems to still very often fail. @goku321 could you please take a look?

Logs:

>>> scalers/redis-cluster-streams.test.ts.log <<<
2021-01-12T01:28:54.4182936Z 
2021-01-12T01:28:54.4184294Z namespace/redis-cluster-streams created
2021-01-12T01:28:54.4185100Z "bitnami" has been added to your repositories
2021-01-12T01:28:54.4185944Z NAME: redis-cluster-streams
2021-01-12T01:28:54.4186595Z LAST DEPLOYED: Tue Jan 12 01:06:46 2021
2021-01-12T01:28:54.4187407Z NAMESPACE: redis-cluster-streams
2021-01-12T01:28:54.4188039Z STATUS: deployed
2021-01-12T01:28:54.4188501Z REVISION: 1
2021-01-12T01:28:54.4188947Z TEST SUITE: None
2021-01-12T01:28:54.4194066Z NOTES:
2021-01-12T01:28:54.4195923Z ** Please be patient while the chart is being deployed **
2021-01-12T01:28:54.4197467Z 
2021-01-12T01:28:54.4197846Z 
2021-01-12T01:28:54.4198250Z To get your password run:
2021-01-12T01:28:54.4200037Z     export REDIS_PASSWORD=$(kubectl get secret --namespace redis-cluster-streams redis-cluster-streams -o jsonpath="{.data.redis-password}" | base64 --decode)
2021-01-12T01:28:54.4201034Z 
2021-01-12T01:28:54.4201993Z You have deployed a Redis Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis cluster:
2021-01-12T01:28:54.4202936Z 
2021-01-12T01:28:54.4203411Z 1. Run a Redis pod that you can use as a client:
2021-01-12T01:28:54.4208919Z kubectl run --namespace redis-cluster-streams redis-cluster-streams-client --rm --tty -i --restart='Never' \
2021-01-12T01:28:54.4210896Z  --env REDIS_PASSWORD=$REDIS_PASSWORD \
2021-01-12T01:28:54.4213152Z --image docker.io/bitnami/redis-cluster:6.0.9-debian-10-r59 -- bash
2021-01-12T01:28:54.4213698Z 
2021-01-12T01:28:54.4214305Z 2. Connect using the Redis CLI:
2021-01-12T01:28:54.4214655Z 
2021-01-12T01:28:54.4215523Z redis-cli -c -h redis-cluster-streams -a $REDIS_PASSWORD
2021-01-12T01:28:54.4216180Z 6
2021-01-12T01:28:54.4216594Z 6
2021-01-12T01:28:54.4217105Z 6
...
2021-01-12T01:28:54.4235082Z 6
2021-01-12T01:28:54.4235500Z 6
2021-01-12T01:28:54.4235894Z 6
2021-01-12T01:28:54.4236297Z 10.0.62.135
2021-01-12T01:28:54.4237305Z namespace/redis-cluster-streams-test created
2021-01-12T01:28:54.4238872Z secret/redis***
2021-01-12T01:28:54.4243919Z triggerauthentication.keda.sh/keda-redis-stream-triggerauth created
2021-01-12T01:28:54.4245492Z deployment.apps/redis-streams-consumer created
2021-01-12T01:28:54.4247857Z scaledobject.keda.sh/redis-streams-scaledobject created
2021-01-12T01:28:54.4284374Z 1
2021-01-12T01:28:54.4285416Z   ✔ Deployment should have 1 replica on start (159ms)
2021-01-12T01:28:54.4286332Z job.batch/redis-streams-producer created
2021-01-12T01:28:54.4286909Z 1
2021-01-12T01:28:54.4287221Z 1
2021-01-12T01:28:54.4287546Z 1
2021-01-12T01:28:54.4287870Z 1
2021-01-12T01:28:54.4288191Z 1
2021-01-12T01:28:54.4288513Z 4
2021-01-12T01:28:54.4288832Z 4
2021-01-12T01:28:54.4289155Z 4
2021-01-12T01:28:54.4289473Z 4
2021-01-12T01:28:54.4289778Z 5
...
2021-01-12T01:28:54.4319645Z 5
2021-01-12T01:28:54.4319964Z 5
2021-01-12T01:28:54.4320286Z 5
2021-01-12T01:28:54.4320605Z 5
2021-01-12T01:28:54.4321711Z   ✖ Deployment should scale to 5 with 100 messages and back to 1 Replica count should be 1 within 10 minutes
2021-01-12T01:28:54.4323880Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4324548Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4325368Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4326076Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4326765Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4327479Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4328187Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4328888Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4329589Z     ℹ (scale up) replica count is:5
2021-01-12T01:28:54.4330307Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4331023Z     ℹ (scale down) replica count is:5
...
2021-01-12T01:28:54.4377942Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4378661Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4379377Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4380078Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4382150Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4382944Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4383664Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4384736Z scaledobject.keda.sh "redis-streams-scaledobject" deleted
2021-01-12T01:28:54.4386376Z triggerauthentication.keda.sh "keda-redis-stream-triggerauth" deleted
2021-01-12T01:28:54.4387784Z secret "redis-password" deleted
2021-01-12T01:28:54.4388795Z deployment.apps "redis-streams-consumer" deleted
2021-01-12T01:28:54.4389946Z job.batch "redis-streams-producer" deleted
2021-01-12T01:28:54.4391056Z namespace "redis-cluster-streams-test" deleted
2021-01-12T01:28:54.4392147Z release "redis-cluster-streams" uninstalled
2021-01-12T01:28:54.4393148Z namespace "redis-cluster-streams" deleted
2021-01-12T01:28:54.4393648Z 
2021-01-12T01:28:54.4394049Z   1 test failed
2021-01-12T01:28:54.4394343Z 
2021-01-12T01:28:54.4394918Z   Deployment should scale to 5 with 100 messages and back to 1
2021-01-12T01:28:54.4395741Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4396466Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4397189Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4397891Z     ℹ (scale up) replica count is:1
2021-01-12T01:28:54.4398613Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4399319Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4400012Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4400725Z     ℹ (scale up) replica count is:4
2021-01-12T01:28:54.4401440Z     ℹ (scale up) replica count is:5
2021-01-12T01:28:54.4402175Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4402899Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4403625Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4404352Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4405072Z     ℹ (scale down) replica count is:5
...
2021-01-12T01:28:54.4457143Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4457870Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4458586Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4459297Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4459995Z     ℹ (scale down) replica count is:5
2021-01-12T01:28:54.4460356Z 
2021-01-12T01:28:54.4461288Z   /__w/keda/keda/tests/scalers/redis-cluster-streams.test.ts:103
2021-01-12T01:28:54.4461926Z 
2021-01-12T01:28:54.4462359Z    102:                                                                         
2021-01-12T01:28:54.4463311Z    103:   t.is('1', replicaCount, 'Replica count should be 1 within 10 minutes')
2021-01-12T01:28:54.4463958Z    104: })                                                                      
2021-01-12T01:28:54.4464251Z 
2021-01-12T01:28:54.4464729Z   Replica count should be 1 within 10 minutes
2021-01-12T01:28:54.4465104Z 
2021-01-12T01:28:54.4465486Z   Difference:
2021-01-12T01:28:54.4465770Z 
2021-01-12T01:28:54.4466223Z   - '1'
2021-01-12T01:28:54.4466669Z   + '5'

zroubalik on 12 Jan 2021

👍1

Sure @zroubalik I am already trying to figure this one out. I also noticed that redis-streams used to fail randomly in previous builds, not sure if this is related. Anyways, will try to fix it asap.

goku321 on 12 Jan 2021

❤1

Needs to be reopened 😅

This is fascinating. I couldn't reproduce the failure even for once. I'm surely missing something, will try to figure it out.

goku321 on 28 Jan 2021

😄1

Searching through the logs, I found few issues with the tests:

Sometimes Redis cluster fails to start with the default timeout, which is 300 seconds. Increasing the timeout might help.

I see MOVED errors in few logs coming from Redis cluster scaler. This happens when a key is moved(to a different node) from the node being queried. The redis client is supposed to handle this redirection automatically but the default value for max redirections is 3. That means, after 3 unsuccessful redirections, redis client would throw a MOVED error. Theoretically, there should be only one redirection. I'm still trying to educate myself on this.
In one of the logs I saw CLUSTERDOWN error. It could be due to the fact that cluster was still starting up (issue 1) OR one or more cluster nodes were down. Currently, we have one replica per master (3 master, 3 replicas). I think, increasing the number of replicas per master node might help. Though, this situation is very rare in the test environment.

I will create a fix for the first issue soon and I believe it might solve the third one as well. Let's see if it improves the stability of the tests.

goku321 on 2 Feb 2021

@goku321 thanks for the investigation! Btw, could be there a problem that all the redis tests run in parallel? Couldn't each test try to deploy and undeploy redis cluster?

zroubalik on 3 Feb 2021

👍1

@zroubalik Thanks for the suggestion. I verified that parallel tests won't create problems. Creating a cluster once would do.

goku321 on 3 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings