Ioredis: Failed to refresh slots cache

[tvb]

Hi,

We are running ioredis 4.0.0 against an AWS ElastiCache replication group with cluster mode on (2 nodes, 1 shard) and using the cluster configuration address to connect to the cluster.

client = new Redis.Cluster([config.redis]);

and the config part:

  redis: Config({
    host: {
      env: 'REDIS_HOST',
      ssm: '/elasticache/redis/host'
    },
    port: {
      env: 'REDIS_PORT',
      ssm: '/elasticache/redis/port'
    }
  }),

Every time we start the application the logs shows us:

{
    "stack": "Error: Failed to refresh slots cache.\n    at tryNode (/app/node_modules/ioredis/built/cluster/index.js:371:25)\n    at /app/node_modules/ioredis/built/cluster/index.js:383:17\n    at Timeout.<anonymous> (/app/node_modules/ioredis/built/cluster/index.js:594:20)\n    at Timeout.run (/app/node_modules/ioredis/built/utils/index.js:144:22)\n    at ontimeout (timers.js:486:15)\n    at tryOnTimeout (timers.js:317:5)\n    at Timer.listOnTimeout (timers.js:277:5)",
    "message": "Failed to refresh slots cache.",
    "lastNodeError": {
        "stack": "Error: timeout\n    at Object.exports.timeout (/app/node_modules/ioredis/built/utils/index.js:147:38)\n    at Cluster.getInfoFromNode (/app/node_modules/ioredis/built/cluster/index.js:591:34)\n    at tryNode (/app/node_modules/ioredis/built/cluster/index.js:376:15)\n    at Cluster.refreshSlotsCache (/app/node_modules/ioredis/built/cluster/index.js:391:5)\n    at Cluster.<anonymous> (/app/node_modules/ioredis/built/cluster/index.js:171:14)\n    at new Promise (<anonymous>)\n    at Cluster.connect (/app/node_modules/ioredis/built/cluster/index.js:125:12)\n    at new Cluster (/app/node_modules/ioredis/built/cluster/index.js:81:14)\n    at Object.<anonymous> (/app/node_modules/@myapp/myapp-core/core/redis.js:10:12)\n    at Module._compile (module.js:652:30)",
        "message": "timeout"
    },
    "level": "error"
}

What more information do you need so we can get this fixed?

[luin]

Please enable the debug mode DEBUG=ioredis:* node yourapp.js and post the logs here.

[tvb]

@luin here you go

2018-09-28T19:00:43.820Z ioredis:cluster status: [empty] -> connecting
2018-09-28T19:00:43.822Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:43.823Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:43.823Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:43.825Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:43.825Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:47.540Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> connecting
2018-09-28T19:00:47.540Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> connecting
{
    "stack": "Error: Failed to refresh slots cache.\n    at tryNode (/app/node_modules/ioredis/built/cluster/index.js:371:25)\n    at /app/node_modules/ioredis/built/cluster/index.js:383:17\n    at Timeout.<anonymous> (/app/node_modules/ioredis/built/cluster/index.js:594:20)\n    at Timeout.run (/app/node_modules/ioredis/built/utils/index.js:144:22)\n    at ontimeout (timers.js:486:15)\n    at tryOnTimeout (timers.js:317:5)\n    at Timer.listOnTimeout (timers.js:277:5)",
    "message": "Failed to refresh slots cache.",
    "lastNodeError": {
        "stack": "Error: timeout\n    at Object.exports.timeout (/app/node_modules/ioredis/built/utils/index.js:147:38)\n    at Cluster.getInfoFromNode (/app/node_modules/ioredis/built/cluster/index.js:591:34)\n    at tryNode (/app/node_modules/ioredis/built/cluster/index.js:376:15)\n    at Cluster.refreshSlotsCache (/app/node_modules/ioredis/built/cluster/index.js:391:5)\n    at Cluster.<anonymous> (/app/node_modules/ioredis/built/cluster/index.js:171:14)\n    at new Promise (<anonymous>)\n    at Cluster.connect (/app/node_modules/ioredis/built/cluster/index.js:125:12)\n    at new Cluster (/app/node_modules/ioredis/built/cluster/index.js:81:14)\n    at Object.<anonymous> (/app/node_modules/@myapp/myapp-core/core/redis.js:10:12)\n    at Module._compile (module.js:652:30)",
        "message": "timeout"
    },
    "level": "error"
}
2018-09-28T19:00:47.573Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:47.709Z ioredis:redis status[172.16.32.178:6379]: connecting -> connect
2018-09-28T19:00:47.709Z ioredis:redis queue command[0] -> info()
2018-09-28T19:00:47.710Z ioredis:redis status[172.16.32.178:6379]: connect -> close
2018-09-28T19:00:47.710Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:47.710Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:47.711Z ioredis:cluster status: connecting -> close
2018-09-28T19:00:47.711Z ioredis:cluster status: close -> reconnecting
2018-09-28T19:00:47.803Z ioredis:redis status[172.16.32.178:6379]: connecting -> connect
2018-09-28T19:00:47.804Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:47.805Z ioredis:redis status[172.16.32.178:6379]: connect -> ready
2018-09-28T19:00:47.814Z ioredis:cluster Cluster is disconnected. Retrying after 102ms
2018-09-28T19:00:47.814Z ioredis:cluster status: reconnecting -> connecting
2018-09-28T19:00:47.814Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:47.814Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:47.814Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:47.815Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:47.815Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:47.834Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:47.834Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:47.835Z ioredis:redis status[172.16.34.8:6379]: connect -> ready
2018-09-28T19:00:47.943Z ioredis:redis status[172.16.32.178:6379]: connecting -> connect
2018-09-28T19:00:47.943Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:47.943Z ioredis:redis status[172.16.32.178:6379]: connect -> ready
2018-09-28T19:00:47.944Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:47.944Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:47.945Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:47.945Z ioredis:cluster:connectionPool Connecting to 172.16.32.178:6379 as master
2018-09-28T19:00:47.945Z ioredis:redis status[172.16.32.178:6379]: [empty] -> wait
2018-09-28T19:00:47.945Z ioredis:cluster:connectionPool Connecting to 172.16.34.8:6379 as slave
2018-09-28T19:00:47.946Z ioredis:redis status[172.16.34.8:6379]: [empty] -> wait
2018-09-28T19:00:47.946Z ioredis:cluster status: connecting -> connect
2018-09-28T19:00:47.947Z ioredis:redis queue command[0] -> cluster(info)
2018-09-28T19:00:47.947Z ioredis:redis status[172.16.32.178:6379]: ready -> close
2018-09-28T19:00:47.947Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:47.947Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:47.948Z ioredis:redis status[172.16.34.8:6379]: wait -> connecting
2018-09-28T19:00:47.949Z ioredis:cluster Ready check failed (Error: Connection is closed.). Reconnecting...
2018-09-28T19:00:47.949Z ioredis:cluster status: connect -> disconnecting
2018-09-28T19:00:47.949Z ioredis:cluster:connectionPool Disconnect 172.16.32.178:6379 because the node does not hold any slot
2018-09-28T19:00:47.949Z ioredis:redis status[172.16.32.178:6379]: wait -> close
2018-09-28T19:00:47.949Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:47.949Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:47.949Z ioredis:cluster:connectionPool Disconnect 172.16.34.8:6379 because the node does not hold any slot
2018-09-28T19:00:47.949Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:47.949Z ioredis:redis queue command[0] -> info()
2018-09-28T19:00:47.950Z ioredis:redis status[172.16.34.8:6379]: connect -> close
2018-09-28T19:00:47.950Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:47.950Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:47.950Z ioredis:cluster status: disconnecting -> close
2018-09-28T19:00:47.950Z ioredis:cluster status: close -> reconnecting
2018-09-28T19:00:48.054Z ioredis:cluster Cluster is disconnected. Retrying after 104ms
2018-09-28T19:00:48.054Z ioredis:cluster status: reconnecting -> connecting
2018-09-28T19:00:48.055Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:48.055Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:48.055Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:48.055Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:48.055Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:48.160Z ioredis:redis status[172.16.32.178:6379]: connecting -> connect
2018-09-28T19:00:48.160Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:48.161Z ioredis:redis status[172.16.32.178:6379]: connect -> ready
2018-09-28T19:00:48.161Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:48.161Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:48.163Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:48.163Z ioredis:cluster:connectionPool Connecting to 172.16.32.178:6379 as master
2018-09-28T19:00:48.164Z ioredis:redis status[172.16.32.178:6379]: [empty] -> wait
2018-09-28T19:00:48.164Z ioredis:cluster:connectionPool Connecting to 172.16.34.8:6379 as slave
2018-09-28T19:00:48.164Z ioredis:redis status[172.16.34.8:6379]: [empty] -> wait
2018-09-28T19:00:48.164Z ioredis:cluster status: connecting -> connect
2018-09-28T19:00:48.164Z ioredis:redis queue command[0] -> cluster(info)
2018-09-28T19:00:48.165Z ioredis:redis status[172.16.32.178:6379]: ready -> close
2018-09-28T19:00:48.165Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.165Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:48.165Z ioredis:redis status[172.16.34.8:6379]: wait -> connecting
2018-09-28T19:00:48.165Z ioredis:cluster Ready check failed (Error: Connection is closed.). Reconnecting...
2018-09-28T19:00:48.165Z ioredis:cluster status: connect -> disconnecting
2018-09-28T19:00:48.165Z ioredis:cluster:connectionPool Disconnect 172.16.32.178:6379 because the node does not hold any slot
2018-09-28T19:00:48.166Z ioredis:redis status[172.16.32.178:6379]: wait -> close
2018-09-28T19:00:48.166Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.166Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:48.166Z ioredis:cluster:connectionPool Disconnect 172.16.34.8:6379 because the node does not hold any slot
2018-09-28T19:00:48.166Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.166Z ioredis:redis queue command[0] -> info()
2018-09-28T19:00:48.166Z ioredis:redis status[172.16.34.8:6379]: connect -> close
2018-09-28T19:00:48.166Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.167Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.167Z ioredis:cluster status: disconnecting -> close
2018-09-28T19:00:48.167Z ioredis:cluster status: close -> reconnecting
2018-09-28T19:00:48.273Z ioredis:cluster Cluster is disconnected. Retrying after 106ms
2018-09-28T19:00:48.273Z ioredis:cluster status: reconnecting -> connecting
2018-09-28T19:00:48.273Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:48.274Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:48.274Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:48.274Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:48.274Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:48.392Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.392Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:48.393Z ioredis:redis status[172.16.34.8:6379]: connect -> ready
2018-09-28T19:00:48.393Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:48.393Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:48.399Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:48.399Z ioredis:cluster:connectionPool Connecting to 172.16.32.178:6379 as master
2018-09-28T19:00:48.399Z ioredis:redis status[172.16.32.178:6379]: [empty] -> wait
2018-09-28T19:00:48.399Z ioredis:cluster:connectionPool Connecting to 172.16.34.8:6379 as slave
2018-09-28T19:00:48.400Z ioredis:redis status[172.16.34.8:6379]: [empty] -> wait
2018-09-28T19:00:48.400Z ioredis:cluster status: connecting -> connect
2018-09-28T19:00:48.400Z ioredis:redis queue command[0] -> cluster(info)
2018-09-28T19:00:48.400Z ioredis:redis status[172.16.34.8:6379]: ready -> close
2018-09-28T19:00:48.400Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.400Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.400Z ioredis:redis status[172.16.34.8:6379]: wait -> connecting
2018-09-28T19:00:48.401Z ioredis:cluster Ready check failed (Error: Connection is closed.). Reconnecting...
2018-09-28T19:00:48.401Z ioredis:cluster status: connect -> disconnecting
2018-09-28T19:00:48.401Z ioredis:cluster:connectionPool Disconnect 172.16.32.178:6379 because the node does not hold any slot
2018-09-28T19:00:48.401Z ioredis:redis status[172.16.32.178:6379]: wait -> close
2018-09-28T19:00:48.401Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.401Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:48.401Z ioredis:cluster:connectionPool Disconnect 172.16.34.8:6379 because the node does not hold any slot
2018-09-28T19:00:48.401Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.401Z ioredis:redis queue command[0] -> info()
2018-09-28T19:00:48.401Z ioredis:redis status[172.16.34.8:6379]: connect -> close
2018-09-28T19:00:48.401Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.401Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.402Z ioredis:cluster status: disconnecting -> close
2018-09-28T19:00:48.402Z ioredis:cluster status: close -> reconnecting
2018-09-28T19:00:48.510Z ioredis:cluster Cluster is disconnected. Retrying after 108ms
2018-09-28T19:00:48.510Z ioredis:cluster status: reconnecting -> connecting
2018-09-28T19:00:48.510Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:48.510Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:48.510Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:48.510Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:48.511Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:48.619Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.619Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:48.620Z ioredis:redis status[172.16.34.8:6379]: connect -> ready
2018-09-28T19:00:48.620Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:48.620Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:48.629Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:48.630Z ioredis:cluster:connectionPool Connecting to 172.16.32.178:6379 as master
2018-09-28T19:00:48.630Z ioredis:redis status[172.16.32.178:6379]: [empty] -> wait
2018-09-28T19:00:48.630Z ioredis:cluster:connectionPool Connecting to 172.16.34.8:6379 as slave
2018-09-28T19:00:48.631Z ioredis:redis status[172.16.34.8:6379]: [empty] -> wait
2018-09-28T19:00:48.631Z ioredis:cluster status: connecting -> connect
2018-09-28T19:00:48.631Z ioredis:redis queue command[0] -> cluster(info)
2018-09-28T19:00:48.631Z ioredis:redis status[172.16.34.8:6379]: ready -> close
2018-09-28T19:00:48.631Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.631Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.632Z ioredis:redis status[172.16.34.8:6379]: wait -> connecting
2018-09-28T19:00:48.632Z ioredis:cluster Ready check failed (Error: Connection is closed.). Reconnecting...
2018-09-28T19:00:48.632Z ioredis:cluster status: connect -> disconnecting
2018-09-28T19:00:48.632Z ioredis:cluster:connectionPool Disconnect 172.16.32.178:6379 because the node does not hold any slot
2018-09-28T19:00:48.632Z ioredis:redis status[172.16.32.178:6379]: wait -> close
2018-09-28T19:00:48.632Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.632Z ioredis:redis status[172.16.32.178:6379]: close -> end
2018-09-28T19:00:48.632Z ioredis:cluster:connectionPool Disconnect 172.16.34.8:6379 because the node does not hold any slot
2018-09-28T19:00:48.632Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.633Z ioredis:redis queue command[0] -> info()
2018-09-28T19:00:48.633Z ioredis:redis status[172.16.34.8:6379]: connect -> close
2018-09-28T19:00:48.633Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.633Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.633Z ioredis:cluster status: disconnecting -> close
2018-09-28T19:00:48.633Z ioredis:cluster status: close -> reconnecting
2018-09-28T19:00:48.744Z ioredis:cluster Cluster is disconnected. Retrying after 110ms
2018-09-28T19:00:48.744Z ioredis:cluster status: reconnecting -> connecting
2018-09-28T19:00:48.744Z ioredis:cluster:connectionPool Connecting to redis.<<MY_DOMAIN>>:6379 as master
2018-09-28T19:00:48.744Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: [empty] -> wait
2018-09-28T19:00:48.744Z ioredis:cluster getting slot cache from redis.<<MY_DOMAIN>>:6379
2018-09-28T19:00:48.744Z ioredis:redis status[redis.<<MY_DOMAIN>>:6379]: wait -> connecting
2018-09-28T19:00:48.744Z ioredis:redis queue command[0] -> cluster(slots)
2018-09-28T19:00:48.851Z ioredis:redis status[172.16.34.8:6379]: connecting -> connect
2018-09-28T19:00:48.851Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:48.852Z ioredis:redis status[172.16.34.8:6379]: connect -> ready
2018-09-28T19:00:48.852Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:48.852Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:48.853Z ioredis:cluster:connectionPool Disconnect redis.<<MY_DOMAIN>>:6379 because the node does not hold any slot
2018-09-28T19:00:48.853Z ioredis:cluster:connectionPool Connecting to 172.16.32.178:6379 as master
2018-09-28T19:00:48.853Z ioredis:redis status[172.16.32.178:6379]: [empty] -> wait
2018-09-28T19:00:48.853Z ioredis:cluster:connectionPool Connecting to 172.16.34.8:6379 as slave
2018-09-28T19:00:48.853Z ioredis:redis status[172.16.34.8:6379]: [empty] -> wait
2018-09-28T19:00:48.853Z ioredis:cluster status: connecting -> connect
2018-09-28T19:00:48.853Z ioredis:redis status[172.16.32.178:6379]: wait -> connecting
2018-09-28T19:00:48.853Z ioredis:redis queue command[0] -> cluster(info)
2018-09-28T19:00:48.854Z ioredis:redis status[172.16.34.8:6379]: ready -> close
2018-09-28T19:00:48.854Z ioredis:connection skip reconnecting since the connection is manually closed.
2018-09-28T19:00:48.854Z ioredis:redis status[172.16.34.8:6379]: close -> end
2018-09-28T19:00:48.854Z ioredis:redis status[172.16.32.178:6379]: connecting -> connect
2018-09-28T19:00:48.854Z ioredis:redis write command[0] -> info()
2018-09-28T19:00:48.855Z ioredis:redis status[172.16.32.178:6379]: connect -> ready
2018-09-28T19:00:48.855Z ioredis:connection send 1 commands in offline queue
2018-09-28T19:00:48.855Z ioredis:redis write command[0] -> cluster(info)
2018-09-28T19:00:48.856Z ioredis:cluster status: connect -> ready
2018-09-28T19:00:49.593Z ioredis:redis write command[0] -> get(ed7fc182-5a18-4c92-8d62-7c140be54545:orders)
2018-09-28T19:00:49.594Z ioredis:redis write command[0] -> get(ed7fc182-5a18-4c92-8d62-7c140be54545:positions)
2018-09-28T19:00:53.857Z ioredis:cluster getting slot cache from 172.16.32.178:6379
2018-09-28T19:00:53.857Z ioredis:redis write command[0] -> cluster(slots)
2018-09-28T19:00:58.857Z ioredis:cluster getting slot cache from 172.16.32.178:6379
2018-09-28T19:00:58.858Z ioredis:redis write command[0] -> cluster(slots)

after which the application logs are starting

[tvb]

@luin perhaps I don't need to use the new Redis.Cluster with Amazon ElastiCache?

[tvb]

@luin bump. Can we look into this together?

[tvb]

We still have this issue 😭

[luin]

Sorry for the late response. Redis.Cluster is required for the cluster mode in ElastiCache. What about disable the enableReadyCheck? new Redis.Cluster([/*..*/], {enableReadyCheck: false})

[luin]

This may be related with issue https://github.com/luin/ioredis/issues/701 & https://github.com/luin/ioredis/issues/709

[liam-murray-xealth]

I see a similar issue. In my case it appears to be a race condition because the connection error is intermittent. When it fails I would only see the following error message (we are using an encryption enabled elasticache cluster).

Error: 140434768993160:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1216:

Once I enabled DEBUG I could see that the connection failure would always be associated with a bunch of "Disconnect because the node does not hold any slot" messages. (I tried enableReadyCheck set true and false and it made no difference.)

2018-10-06T18:24:38.297Z ioredis:cluster:connectionPool Disconnect xxx.usw2.cache.amazonaws.com:6379 because the node does not hold any slot

I tried redis-clustr with node-redis and am able to connect fine with those.

[luin]

@liam-murray-xealth Thank you for pointing this out. What's the result of the command CLUSTER SLOTS? "Disconnect xxx.usw2.cache.amazonaws.com:6379 because the node does not hold any slot" happens when the result of CLUSTER SLOTS doesn't contain that host.

[luin]

@tvb What about using the configuration endpoint with new Redis() (instead of new Redis.Cluster())? I don't have a running ElastiCache cluster instance so I can't test with it. I'll set up one soon.

[liam-murray-xealth]

I think I found something that will help. I have been passing the single Configuration Endpoint URL to Redis.Cluster() (as the original author of this issue has been doing).

{
host: clustercfg.mycluster.usw2.cache.amazonaws.com
port: 6379
}

I tried instead passing the endpoints directly and it appears to work.

const nodes = [
{
host: 'my-cluster-0001-001.xxx.usw2.cache.amazonaws.com',
port: 6379,
},
{
host: 'my-cluster-0001-002.xxx.usw2.cache.amazonaws.com',
port: 6379,
},
{
host: 'my-cluster-0001-003.xxx.usw2.cache.amazonaws.com',
port: 6379,
},
]

I obtained the above by running the following (I only have one cluster with three nodes).

aws elasticache describe-cache-clusters --show-cache-node-info | jq -r '.CacheClusters[].CacheNodes[].Endpoint'

Funny thing is that it ("Redis.Cluster") sometimes does work with the configuration endpoint--it just seems to have some race condition that prevents it from always working.

Out of curiosity I tried using "new Redis()" passing the cluster configuration endpoint and got:

ReplyError: MOVED 11958 'my-cluster-0001-001.xxx.usw2.cache.amazonaws.com':6379

[tvb]

Correct. We see the same MOVED message if we connect without .Cluster config.

Providing individual node endpoints is not an option for us.

[luin]

This issue should be solved in v4.2.0. Using configuration endpoint as startup nodes for Cluster. Feel free to reopen this issue if the problem still exists.

[TNieminen]

Just noticed that this fix will not work on Redis engine 5.0 on Elasticache.
Downgrade to 4.0.10 to make it work.

[dko-slapdash]

Unfortunately the problem still exists. Version 4.6.2, exactly same symptoms.
ElastiCache with TLS, Engine Version Compatibility: 5.0.3

[krunkosaurus]

Still happening for me as well. Elasticache engine 5.0.3 (cluster) and [email protected]

[FelipeTaiarol]

Same problem for me with Elasticache engine 5.0.3 (cluster) and [email protected]

This is what I get with debug enabled:

2019-04-23T07:50:48.508Z ioredis:cluster getting slot cache from 172.20.4.199:6379
2019-04-23T07:50:48.508Z ioredis:redis write command[172.20.4.199:6379]: 0 -> cluster([ 'slots' ])
2019-04-23T07:50:48.509Z ioredis:cluster cluster slots result count: 1
2019-04-23T07:50:48.509Z ioredis:cluster cluster slots result [0]: slots 0~16383 served by 172.20.5.116:6379,172.20.4.199:6379
2019-04-23T07:50:48.509Z ioredis:cluster:connectionPool Reset with [ { host: '172.20.5.116', port: 6379, readOnly: false },
  { host: '172.20.4.199', port: 6379, readOnly: true } ]
2019-04-23T07:50:51.833Z 'Waiting for read_model_sync message'
2019-04-23T07:50:51.833Z ioredis:redis write command[172.20.4.199:6379]: 0 -> bzpopmax([ 'read_model_sync', '60' ])
2019-04-23T07:50:51.834Z 'Error doing \'bzpopmax\' on read_model_sync' { ReplyError: MOVED 13851 172.20.5.116:6379
    at parseError (/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:302:14)
  command: { name: 'bzpopmax', args: [ 'read_model_sync', '60' ] } }

I am starting the connection doing this:

new Redis.Cluster([{ port: 6379, host: ***.clustercfg.euw1.cache.amazonaws.com }]);

[FelipeTaiarol]

@luin , mentioning you just to make sure you are aware of the issue as we are not able to reopen the ticket ourselves.

[FelipeTaiarol]

Sorry, correction to what I said before, passing the cluster nodes to the Redis.Cluster constructor worked.

[astra1]

I was able to connect to aws with increased slotsRefreshTimeout

const cacheCluster = new IORedis.Cluster(
    [
        {
            host: process.env.elasticache_host,
            port: process.env.elasticache_port,
        },
    ],
    {
        slotsRefreshTimeout: 2000,
        redisOptions: {
            tls: {},
            password: process.env.elasticache_password,
        },
    }
);

[FelipeTaiarol]

Correction again, it works in the beginning, but after some time it stops working and I start getting these errors:

2019-04-23T09:34:26.319Z ioredis:cluster getting slot cache from 172.20.5.116:6379
2019-04-23T09:34:26.320Z ioredis:redis write command[172.20.5.116:6379]: 0 -> cluster([ 'slots' ])
2019-04-23T09:34:26.320Z ioredis:cluster getting slot cache from 172.20.4.199:6379
2019-04-23T09:34:26.320Z ioredis:redis write command[172.20.4.199:6379]: 0 -> cluster([ 'slots' ])
2019-04-23T09:34:26.321Z ioredis:cluster cluster slots result count: 1
2019-04-23T09:34:26.321Z ioredis:cluster cluster slots result [0]: slots 0~16383 served by 172.20.5.116:6379,172.20.4.199:6379
2019-04-23T09:34:26.321Z ioredis:cluster:connectionPool Reset with [ { host: '172.20.5.116', port: 6379, readOnly: false },
  { host: '172.20.4.199', port: 6379, readOnly: true } ]
2019-04-23T09:34:26.322Z ioredis:cluster cluster slots result count: 1
2019-04-23T09:34:26.322Z ioredis:cluster cluster slots result [0]: slots 0~16383 served by 172.20.5.116:6379,172.20.4.199:6379
2019-04-23T09:34:26.322Z ioredis:cluster:connectionPool Reset with [ { host: '172.20.5.116', port: 6379, readOnly: false },
  { host: '172.20.4.199', port: 6379, readOnly: true } ]
2019-04-23T09:34:27.090Z 'Waiting for read_model_sync message'
2019-04-23T09:34:27.090Z ioredis:redis write command[172.20.4.199:6379]: 0 -> bzpopmax([ 'read_model_sync', '60' ])
2019-04-23T09:34:27.091Z 'Error doing \'bzpopmax\' on read_model_sync' { ReplyError: MOVED 13851 172.20.5.116:6379
    at parseError (/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:302:14)
  command: { name: 'bzpopmax', args: [ 'read_model_sync', '60' ] } }

[FelipeTaiarol]

increasing slotsRefreshTimeout worked for me as well, thanks!

[tuananh]

any idea what cause the slot refresh timeout so high on aws elasticache?

[tong3jie]

set slotsRefreshTimeout:5000 ，but timeout again

[floridait]

I am also having this issue with the latest ioredis. I have tried setting slotsRefreshTimeout to 2000 and 5000. I have tried adding the cluster endpoints directly. Tried a combination of all. I have tried without cluster config and nothing is working for me. I am using redis 5.0.4 on ElastiCache on AWS. Has anyone gotten this working already? I even tried the redis-clustr package but it does not work either.

[JakeFDev]

Here is what I used to get a redis cluster connection using aws elasticache with auth.

const client = new Redis.Cluster(
      [
        {
          host: process.env.REDIS_HOST,
          port: process.env.REDIS_PORT,
        },
      ],
      {
        slotsRefreshTimeout: 2000,
        dnsLookup: (address, callback) => callback(null, address),
        redisOptions: {
          tls: {},
          password: process.env.REDIS_AUTH_TOKEN,
        },
      },
    )

[sidgujrathi]

Any updates on this issue, I am facing the same issue and above solution with slotsRefreshTimeout:2000 doesn't work for me.

[technicallyfeasible]

None of the suggested solutions above worked for me. I noticed that the disconnect error only happens for the first command executed on a node.

If I catch that and then call node.connect(), all subsequent commands will work.

I also came across this thread again when investigating why this happens with one of my ElastiCache clusters but not the other. The only difference I found so far was a minor version difference:

• working cluster: 5.0.3
• broken cluster: 5.0.0

An upgrade to version 5.0.4 has not solved the problem.

[floridait]

const Redis = require('ioredis');
function createCache(host, port) {
  return new Promise((resolve, reject) => {
    const client = new Redis.Cluster([{host, port}], {
      dnsLookup: (address, callback) => callback(null, address),
      slotsRefreshTimeout: 3000,
    });
    client.on('connect', () => {
      console.log('[redis] connected');
      resolve(client);
    });
    client.on('error', err => {
      console.error('[redis] error ', err);
      reject(err);
    });
  });
}
module.exports = createCache;

This works for me for AWS ElastiCache using a cluster configuration. Maybe it will help you.

[sidgujrathi]

const Redis = require('ioredis');
function createCache(host, port) {
  return new Promise((resolve, reject) => {
    const client = new Redis.Cluster([{host, port}], {
      dnsLookup: (address, callback) => callback(null, address),
      slotsRefreshTimeout: 3000,
    });
    client.on('connect', () => {
      console.log('[redis] connected');
      resolve(client);
    });
    client.on('error', err => {
      console.error('[redis] error ', err);
      reject(err);
    });
  });
}
module.exports = createCache;

This works for me for AWS ElastiCache using a cluster configuration. Maybe it will help you.

This I have tried already but not able to get connection. Can you please help us with telling what version you are using on Elasticache and ioredis ?

[sidgujrathi]

I found the solution 😄

Since AWS ElasticCache provides a single endpoint to connect to the cluster we can not use new Redis.Cluster([{HOSt:'', PORT:''}, {HOSt:'', PORT:''}]) and also not use new Redis(ENDPOINT)

but I manage to get a connection by

new Redis.Cluster(['${redisAddress}:${redisPort}'], { scaleReads: 'slave' });

[Nalgmann]

Used @sidgujrathi 's workaround but I'm still unable to connect to AWS ElasticCache service.
This is my debug:

  ioredis:cluster status: [empty] -> connecting +0ms
  ioredis:cluster resolved hostname <<domain>>.cvc7yh.ng.0001.use1.cache.amazonaws.com to IP 172.20.100.54 +6s
  ioredis:cluster:connectionPool Reset with [ { host: '172.20.100.54', port: 6739 } ] +0ms
  ioredis:cluster:connectionPool Connecting to 172.20.100.54:6739 as master +2ms
  ioredis:redis status[172.20.100.54:6739]: [empty] -> wait +0ms
  ioredis:cluster getting slot cache from 172.20.100.54:6739 +3ms
  ioredis:redis status[172.20.100.54:6739 (ioredisClusterRefresher)]: [empty] -> wait +1ms
  ioredis:redis status[172.20.100.54:6739 (ioredisClusterRefresher)]: wait -> connecting +1ms
  ioredis:redis queue command[172.20.100.54:6739 (ioredisClusterRefresher)]: 0 -> cluster([ 'slots' ]) +0ms
  ioredis:cluster:subscriber selected a subscriber 172.20.100.54:6739 +0ms
  ioredis:redis status[172.20.100.54:6739 (ioredisClusterSubscriber)]: [empty] -> wait +1ms
  ioredis:cluster:subscriber started +0ms

ClusterAllFailedError: Failed to refresh slots cache.
    at tryNode (/Volumes/SSD128HFS/Users/rax/skyalert/skyalert3-be/node_modules/ioredis/built/cluster/index.js:359:31)
    at /project/node_modules/ioredis/built/cluster/index.js:376:21
    at Timeout.duplicatedConnection.cluster.utils_2.timeout (/project/node_modules/ioredis/built/cluster/index.js:623:24)
    at Timeout.run (/project/node_modules/ioredis/built/utils/index.js:156:22)
    at ontimeout (timers.js:438:13)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)

I'm using the single endpoint to connect

[sidgujrathi]

I have found another workaround, somehow above mentioned solution stop working for me as well.

Can you please try the following solution and let us know?

new Redis(6379, <redisAddress>);

[Nalgmann]

Hi. I tried it but didn't work. But I know why I wasn't unable to connect:
ElasticCache won't connect to a place outside AWS. If I want to connect to the redis server I need to forward a SSH connection:

ssh -i "certificate-delivered-by-aws.pem" -f -N -L6379:aws.redis.server.cache.amazonaws.com:6379 [email protected]

This way I can do new Redis() and get a direct connection to ElasticCache.

I don't know if my case is isolated or a common issue related with AWS and not with ioredis.

Thanks for the help

[rabbitfufu]

Unfortunately this is still an issue.

As per the documentation, I am using the following code to connect to an aws elasticache redis cluster:

new Redis.Cluster([{
    host: AWSREDISHOST,
    port: AWSREDISPORT
}], {
    dnsLookup: (address, callback) => callback(null, address),
    redisOptions: {
        tls: { cert }
    }
})

As reported above, this code works great if you are using <= 4.0.10 of ioredis. However upgrading to the latest version of ioredis on aws causes this same code to error out with "ClusterAllFailedError: Failed to refresh slots cache." None of the suggested fixes above worked for me.

[SebC99]

@luin could we open this issue again?
Apparently no one manages to connect to AWS Elasticache Clusters...

[technicallyfeasible]

@luin could we open this issue again?
Apparently no one manages to connect to AWS Elasticache Clusters...

I am running lambda functions successfully connecting to ElastiCache redis. Running into timeouts once in a while but nothing I can't deal with.

[rjhilgefort]

Had this issue and @JakeFDev's solution worked for me. Specifically, the dnsLookup was required.

https://github.com/luin/ioredis/issues/711#issuecomment-503238221

[Dartv]

Nothing of the above worked for me

[xtianus79]

@luin this seems to be a major issue and needs attention as it is random and a hard error perhpas more descriptive error handling would help.

[murtaza-webonise]

increasing slotsRefreshTimeout worked for me as well, thanks!

can you please share the version of redis on ELastic Cache and of ioredis too?

[Girija-Shankar-Swain]

The following works for me.

Redis version: _5.0.4_ on AWS ElastiCache Clustered with TLS and AUTH enabled.
ioredis version: _4.16.0_

Code to connect:

const redis = new Redis.Cluster(
            [{ "host": <ELASTI_CACHE_CONFIGURATION_ENDPOINT> }], {
                dnsLookup: (address, callback) => callback(null, address),
                redisOptions: {
                    tls: true,
                    password: <ELASTI_CACHE_REDIS_AUTH>
                }
            });

Make sure you are in the right VPC, Subnet and Security group to connect to ElastiCache along with the correct IAM role.

[amirmuminovic]

Any updates on this? I'm still experiencing this issue with Redis version 5.0.5 on AWS ElastiCache Clustered, ioredis version: 4.16.1

Edit: Previously said I was using 4.16.0 version of ioredis while it was actually 4.16.1.

[visitsb]

Works 👍

• ElastiCache Redis in Cluster Mode
  Engine Version Compatibility: 5.0.6
Encryption in-transit: yes
  (hence tls: true needed for redisOptions shown in Lambda snippet below)
  Nodes: 9
Shards: 3
Primary Endpoint:- (since this is a Cluster, so Configuration Endpoint is relevant instead)

• My Lambda running on NodeJS 12x,
  [email protected]

Lambda code used

const Redis= require('ioredis');
const client = new Redis.Cluster([{ 
    host: process.env.REDIS_CLUSTER_CONFIGURATION_ENDPOINT 
}], {
    dnsLookup: (address, callback) => callback(null, address),
    redisOptions: {
        tls: true,
        password: process.env.REDIS_CLUSTER_AUTH
    }
});

exports.handler = async (event) => new Promise((resolve, reject) => {
    client.ping('Hello from ElastiCache Redis').then(resolve).catch(reject);
});

I get the below response for my Lambda execution

Response:
"Hello from ElastiCache Redis"

Request ID:
"769caff3-159c-41df-932b-e9134eaf8f6a"

Function Logs:
START RequestId: 769caff3-159c-41df-932b-e9134eaf8f6a Version: $LATEST
END RequestId: 769caff3-159c-41df-932b-e9134eaf8f6a
REPORT RequestId: 769caff3-159c-41df-932b-e9134eaf8f6a  Duration: 483.84 ms Billed Duration: 500 ms Memory Size: 128 MB Max Memory Used: 76 MB  Init Duration: 210.92 ms

Tip:

• Ensure your ElastiCache and Lambda both use the _same_ VPC, and Security Group. Otherwise your Lambda will keep failing on timeouts (including the error messaged Failed to refresh slots cache) since it cannot connect to ElastiCache anyways. Needless to say slotsRefreshTimeout was un-necessary in my case.
• REDIS_CLUSTER_CONFIGURATION_ENDPOINT, REDIS_CLUSTER_AUTH passed as environment variables to Lambda. REDIS_CLUSTER_AUTH can be empty if no redis AUTH is set on ElastiCache Redis cluster.
• I've use Lambda Layer for ioredis dependency in my snippet so I can update to _newer_ versions of ioredis without breaking my Lambda unintentionally.

Hope this helps!

[ion-willo]

Our config:
ElastiCache Redis in Cluster Mode
Engine Version Compatibility: 5.0.6
Encryption in-transit: no
Nodes: 3
Shards: 1
Configuration Endpoint: - <--- Is this maybe the issue?

My Lambda running on NodeJS 10.7.0,
[email protected]

Code:

const Redis = require("ioredis");

var cluster = new Redis.Cluster([
  {
    host: process.env.REDIS_CLUSTER_PRIMARY_ENDPOINT, port: "6379"
  }
], {
  dnsLookup: (address, callback) => callback(null, address),

  scaleReads: 'slave'
})

Fails with:

[ioredis] Unhandled error event: ClusterAllFailedError: Failed to refresh slots cache.
    at tryNode (/tmp/test/node_modules/ioredis/built/cluster/index.js:359:31)
    at /tmp/test/node_modules/ioredis/built/cluster/index.js:376:21
    at duplicatedConnection.cluster.utils_2.timeout (/tmp/test/node_modules/ioredis/built/cluster/index.js:624:24)
    at run (/tmp/test/node_modules/ioredis/built/utils/index.js:156:22)
    at tryCatcher (/tmp/test/node_modules/standard-as-callback/built/utils.js:11:23)
    at promise.then (/tmp/test/node_modules/standard-as-callback/built/index.js:30:51)
    at process._tickCallback (internal/process/next_tick.js:68:7)

REDIS_CLUSTER_PRIMARY_ENDPOINT points to Primary Endpoint

Note that connecting normally to the primary node (new Redis() works, so issues of VPCs and SecurityGroups should not apply.

Any advice would be greatly appreciated

[visitsb]

@ion-willo your configuration appears to be with Cluster mode disabled. In that case, you should directly connect to the Reader Endpoint using new Redis(...) and _not_ use new Redis.Cluster(...). That will do the split for you to evenly read from all read replicas. If you want to update the cache, then using Primary Endpoint is the correct one to use for write operations.

This information is available on the docs too-
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Endpoints.html

You can try setting up another ElastiCache Redis cluster but make sure to select Cluster Mode enabled (Scale Out). Then try with the Configuration Endpoint using new Redis.Cluster(...) and you should be good to go.

Tip: As @luin suggested in some other replies, you can try to connect to redis using redis-cli on a temporary EC2 instance within the same VPC, Security Group as your ElastiCache Redis. Then try to issue a CLUSTER SLOTS command. This command will work only if you use Configuration Endpoint available in Cluster Mode enabled. But this will not work on a Primary Endpoint nor Reader Endpoint since you are directly talking to a single redis node.

Hope this helps.

[MatteoGioioso]

@visitsb First of all, thanks for clarifying this.
I have one question, what you say for the cluster mode disabled is to have two separate clients and manually configure reads and writes?
For example:

const writer = new Redis({
  port: process.env.REDIS_PRIMARY_ENDPOINT,
})

const reader = new Redis({
  port: process.env.REDIS_READER_ENDPOINT,
})

and then

module.exports = {
    getAsync: reader.get,
    setAsync: writer.set,
}

It this right?

[visitsb]

@visitsb First of all, thanks for clarifying this.
I have one question, what you say for the cluster mode disabled is to have two separate clients and manually configure reads and writes?
For example:

const writer = new Redis({
  port: process.env.REDIS_PRIMARY_ENDPOINT,
})

const reader = new Redis({
  port: process.env.REDIS_READER_ENDPOINT,
})

and then

module.exports = {
    getAsync: reader.get,
    setAsync: writer.set,
}

It this right?

@MatteoGioioso Yes, that is correct. The below blog has this outlined as well as the docs.

Redis Cluster mode: Use Cluster-aware Redis clients and connect to the cluster using the configuration endpoint. This allows the client to automatically discover the shard and slot mappings. Redis Cluster mode also provides online resharding (scale in/out) for resizing your cluster, and allows you to complete planned maintenance and node replacements without any write interruptions. The Redis Cluster client can discover the primary and replica nodes and appropriately direct client-specific read and write traffic.

Non-Redis Cluster mode: Use the primary endpoint for all write traffic. During any configuration changes or failovers, Amazon ElastiCache ensures that the DNS of the primary endpoint is updated to always point to the primary node. Use the reader endpoint to direct all read traffic. Amazon ElastiCache ensures that the reader endpoint is kept up-to-date with the cluster changes in real time as replicas are added or removed. Individual node endpoints are also available but using reader endpoint frees up your application from tracking any individual node endpoint changes. Hence, it’s best to use primary endpoint for writes and single reader endpoint for reads.

Configuring the Redis client
https://aws.amazon.com/blogs/database/configuring-amazon-elasticache-for-redis-for-higher-availability/

[daparthi001]

Did any one had slots issue closer to elastic cache maintenance ?

@visitsb Have your issue was resolved with above configuration ?Is that ok if run 2 nodes and 1 shard ?

[visitsb]

@daparthi001 yes, the above configuration worked for me without problems. If you still face issues, perhaps open a new issue since this one is closed.