Ioredis: ETIMEDOUT connecting to AWS Elasticache (non-cluster)

Created on 10 Jan 2020  路  15Comments  路  Source: luin/ioredis

Trying to connect to an AWS Elasticache instance and running in to this problem.

Verifying that Redis connection works from EC2 instance;

/usr/src/app/packages/api # redis-cli -h <AWS_URL> -p 6379
<AWS_URL>:6379> SET a b
OK
<AWS_URL>:6379> GET a
"b"

Connecting like this:

const getRedisConfig = () => {
  const cfg = {
    host: process.env.REDIS_HOST
  };
  if (process.env.NODE_ENV === 'production') {
    cfg['tls'] = {};
  }
  return cfg;
};
// ...
this.connection = new Redis(getRedisConfig());
// ...

Recieving ETIMEDOUT error when connection through ioredis;

/usr/src/app/packages/api # DEBUG=ioredis:* yarn start
yarn run v1.21.1
warning package.json: No license field
$ node dist/server/index.js
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +0ms
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +2ms
ioredis:redis queue command[<AWS_URL>:6379]: 0 -> set([ 'a', 'b' ]) +1ms
redis cfg =>  {
  host: '<AWS_URL>>',
      port: 6379,
      tls: {}
}
ioredis:redis status[<AWS_URL>:6379]: [empty] -> connecting +2ms
ioredis:redis queue command[<AWS_URL>:6379]: 0 -> subscribe([ 'ASSESSMENTS' ]) +0ms
ioredis:redis status[pad-app-api-redis.dj3r40.0001.euw1.cache.amazonaws.com:6379]: [empty] -> connecting +1ms
ioredis:redis status[10.0.0.33:6379]: connecting -> connect +546ms
ioredis:redis write command[10.0.0.33:6379]: 0 -> info([]) +1ms
ioredis:redis status[10.0.0.33:6379]: connect -> ready +4ms
(node:843) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
    ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +2ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +10s
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +1ms
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +1ms
ioredis:connection reconnect in 50ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +48ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +2ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +10s
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +0ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:connection error: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
  errorno: 'ETIMEDOUT',
      code: 'ETIMEDOUT',
      syscall: 'connect'
} +1ms
    [ioredis] Unhandled error event: Error: connect ETIMEDOUT
at TLSSocket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/index.js:282:31)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:223:5)
at TLSSocket.Socket._onTimeout (net.js:474:8)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +10s
ioredis:connection reconnect in 100ms +1ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +1ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +0ms
ioredis:connection reconnect in 100ms +0ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: connecting -> close +0ms
ioredis:connection reconnect in 100ms +0ms
ioredis:redis status[<AWS_URL>:6379]: close -> reconnecting +0ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +101ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms
ioredis:redis status[<AWS_URL>:6379]: reconnecting -> connecting +1ms

Timeouts continue indefinitely like above.

Out of ideas with this one so if anyone could help I would really appreciate it.

wontfix
Source
picture chrisfinch
👍2

Most helpful comment

@chrisfinch
I have an Elasticache with encryption in-transit and at-rest, with TLS. Here is my implementation:

this._client = new IORedis({
  tls: {},
  port: '6379',
  host: process.env.REDIS_HOST,
  password: process.env.REDIS_AUTH,
  keyPrefix: `${process.env.LAMBDA_ENV}-`,
  connectTimeout: 17000,
  maxRetriesPerRequest: 4,
  retryStrategy: (times) => Math.min(times * 30, 1000),
  reconnectOnError: (error) => {
    const targetErrors = [/READONLY/, /ETIMEDOUT/];

    targetErrors.forEach((targetError) => {
      if (targetError.test(error.message)) {
        return true;
      }
    });
  }
});

You can try it out and check if it works ;)

picture vcalmic on 17 Jan 2020
👍5

All 15 comments

Checked timeout setting on Elasticache parameters - is set to 0 so doesn't look like that's the problem.

chrisfinch picture chrisfinch on 10 Jan 2020

+1 Some of the time, having this error in AWS lambdas connecting to elasticache

ERROR   [ioredis] Unhandled error event: Error: connect ETIMEDOUT
    at Socket.<anonymous> (/var/task/src/SQS-consumer/node_modules/ioredis/built/redis/index.js:282:31)
    at Object.onceWrapper (events.js:286:20)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at Socket._onTimeout (net.js:443:8)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
simondutertre picture simondutertre on 13 Jan 2020

I have encountered the same problem. Do you have a solution?

wjt382063576 picture wjt382063576 on 14 Jan 2020

@simondutertre @wjt382063576 I'm still trying to find a solution - if I find anything I will post it here - hopefully the maintainers of ioredis will be able to help soon.

chrisfinch picture chrisfinch on 14 Jan 2020
👍4

I managed to find a working stopgap solution as follows;

In AWS Elasticache Redis:

  • Turn OFF Encryption in-transit
  • Turn OFF Encryption at-rest

In code - connect like this;

const connection = new Redis({
    host: "my_aws_redis_instance_url"
});

This is only a temporary solution so I can continue development - encryption will be required for production deployments of my code so a solution for this timeout problem will still need to be found.

chrisfinch picture chrisfinch on 14 Jan 2020
👍2

@chrisfinch
I have an Elasticache with encryption in-transit and at-rest, with TLS. Here is my implementation:

this._client = new IORedis({
  tls: {},
  port: '6379',
  host: process.env.REDIS_HOST,
  password: process.env.REDIS_AUTH,
  keyPrefix: `${process.env.LAMBDA_ENV}-`,
  connectTimeout: 17000,
  maxRetriesPerRequest: 4,
  retryStrategy: (times) => Math.min(times * 30, 1000),
  reconnectOnError: (error) => {
    const targetErrors = [/READONLY/, /ETIMEDOUT/];

    targetErrors.forEach((targetError) => {
      if (targetError.test(error.message)) {
        return true;
      }
    });
  }
});

You can try it out and check if it works ;)

vcalmic picture vcalmic on 17 Jan 2020
👍5

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

stale[bot] picture stale[bot] on 16 Feb 2020

FYI, you have to run your Lambdas in your own VPC to be able to connect to Elasticache. Double check subnets and security groups.

aalimovs picture aalimovs on 31 Mar 2020

I've already double-checked everything you mentioned in this thread but I keep getting timeout errors and I don't know where to look at!

demian85 picture demian85 on 20 May 2020

@demian85 is your Lambda in your own VPC with the right subnet group?

aalimovs picture aalimovs on 20 May 2020

Which should be the right subnet group?
The Redis instance is in 2 subnets, the same for the lambda function. The same security group also, allowing inbound and outbound connections for everyone in the same group.

demian85 picture demian85 on 20 May 2020

Did anyone find a solution for this? I am seeing the same issue. @luin

ShraddhaMantri picture ShraddhaMantri on 5 Aug 2020

Same issue as well.

ColeTownsend picture ColeTownsend on 16 Oct 2020

I'm still running into the same issue and I don't know which is the proper solution to Lambda aggressively killing the socket and making the connection not reusable. Is that the problem, right?
So:

  • If I use lazyConnect and disconnect at the end of the lambda execution, will this work when reusing the lambda instance later?
  • Can I use the client that was previously closed and connect implicitly when sending a command? Should I connect explicitly always?

Thanks.

demian85 picture demian85 on 22 Oct 2020

Hi guys,

I'm still having this problem on AWS Elasticache with encryption in-transit and at-rest, with TLS. Did you find any solution? If I turn encryption off on AWS, it works, but I need encryption for my production environment.

CostaFernando picture CostaFernando on 22 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lakamsani picture lakamsani  路  3Comments
haoxins picture haoxins  路  5Comments
ORESoftware picture ORESoftware  路  5Comments
jamesdixon picture jamesdixon  路  4Comments
dfee picture dfee  路  4Comments