Sequelize: Preferred way to handle reconnect

Created on 4 Aug 2014  路  79Comments  路  Source: sequelize/sequelize

In case the db connection is lost, we need to reconnect to the database. I couldn't find a preferred way to handle this issue. Do you have a best practice? The issue can be solved on various ways:

  • handle in sequelize
  • handle in db module like pg
  • handle this issue outside of nodejs with connection handler

Thanks in advance

feature

Most helpful comment

Anyone has informations about this feature ? The issue has been created the 4 Aug 2014 and we are the 15 June 2017.

All 79 comments

I believe this would be best handled in sequelize - at least we could retry a number of times and otherwise give up. I believe we could handle this rather nicely actually. Each dialect returns an instance of Sequelize.ConnectionLost when the connector lib gives and error, and the abstract connector manager handles trying to reconect. It should be as simple as just trying the query again, since that should try to fetch a new connection (provided that the previous query flagged its connection as broken properly).

Flagging this as a feature request

@janmeier Thanks that would be amazing.

+

+1 This is definitely something desired. Thanks!

:+1: for this feature.

:+1: Waiting too

+1 I'm gonna add a bounty for this one

+1

This feature is so important in actual production setting, where databases run on completely different hosts, and having those hosts hiccup completely wreck everything because sequelize won't reconnect even though it should be able to.

I can't even emphasize how much this should be the most important feature to add as soon as possible, even just an autoreconnect: true to make Sequelize try to reconnect without any kind of checking what actually went wrong would make DB interfacing with it _vastly_ more useful =)

@Pomax so effectively retry queries anytime we get a connection type error? (Retrying a query should try to initiate a new connection assuming the old one was marked invalid which it should have).

That would necessarily be so hard to implement, could happen at query level and look for a global or local option.

(if you are willing to be a guinea pig i'd be happy to write something like that, we've just been hesitant since it would be hard to have unit or regression tests for, so we need someone to battle test it, and for some reason i've never really encountered it).

Small issue of transactions of course, they would likely need to be retried at the transaction level rather than the query level.

Yeah, as just a simplest option "retry ad infinitum, or the user explicitly set a retry count" would be amazingly useful already. My api.nihongoresources.com has a database hosted not just on a different machine, but by a different organization, and it drops every so often because the machine it's on power cycles fairly often (few weeks, if not days sometimes), which absolutely destroys API functionality.

If you want a guinea pig: it can't get worse than what it's doing right now, hook me up ;) This would be amazing to have:

var reconnectOptions = {
  max_retries: 999,
  onRetry: function(count) {
    console.log("connection lost, trying to reconnect ("+count+")");
  }
};

var sequelize = new Sequelize(database,username,password, {
                  dialect: "mysql",
                  port: 3306,
                  logging: false,
                  reconnect: reconnectOptions || true
                });

("or true" because I'll even just take a plain old "just try to reconnect forever" over "all things now broken" =D)

Transaction failure could be left up to the user really:

var reconnectOptions = {
  retry_on_reconnect: {
    transactions: true,
    ...
  },
  max_retries: 999,
  onRetry: function(count) {
    console.log("connection lost, trying to reconnect ("+count+")");
  }
};

Great :) I'll see if i can take a look at this soon.
The problem with transactions is that for PG a failed query will take down the entire transaction, so you can't retry a transaction alone, i think i'd like to build functionality to retry the entire transaction.

that's how transactions should work though, if it's a true transaction? if any part of a transaction fails, up to and including the commit, it should be discarded.

@Pomax Right but query retrying would have to be disabled inside a transaction then :) And then it might make sense to retry the entire transaction if one query had a connection error.

Although now that i'm thinking about it queries inside a transaction run on the same connection so might not be an issue.

Ahh, yes, very true. If the DBMS is smart enough to roll back on a broken connection or timeout, that should be fine (not sure off-hand what pg and mysql do there)

+1

+1 this would be great

Since this is still open any suggestions on how we should handle a database dropping out until it's implemented into Sequelize?

@Joshua-F my solution was to use a fairly dumb try/catch probe (like a select 1+1), and rebuilding the sequelize instance when it throws. That still fails when the database server is entirely unavailable (i.e. it plain old just drops instead of the connection being flaky) but when that happens you have a completely different set of problems anyway.

@Pomax that's a great idea! Do you have an example of how you rebuild the sequelize instance? Do you essentially wrap the construction in a factory function:

var Sequelize = require('sequelize');

module.exports = function buildSequelize() {
  var sequelize = new Sequelize('username', 'password');
  sequelize.define('User', {
    // ...
  });
  return sequelize;
};

Or are you using an easier method that simply closes and reopens sequelize? (I would prefer that method if it exists)

nope, that's what I do =)

@mickhansen is there an updated solution or fix for this pending? I'm having this issue as well...great to see that it's been well captured in this thread.

The 'wait_timeout' variable may be relevant as well, see:
https://github.com/felixge/node-mysql/issues/1070

@dxdc I originally thought I had this problem and commented on this thread like two days ago, but deleted my comment when I found that the latest sequelize does tend to do this automatically every time that a query is requested. At least I found this true with using sequelize with mysql.

thanks @AndrewFarley ; I'm still having this issue. As far as I can tell, it's connected to the wait_timeout variable in the MySQL settings. The default is 28800 (8 h), but this is too high for common web applications where a more realistic recommendation is 30-300 sec.

@dxdc Hmm, are you using 2.0? And can you give a code or pseudo code example of the problem you're having? I'm just starting in on a larger project with Sequelize at the core, and I've done a fair bit of testing with databases going away and timing out, and it seems to perform quite admirably for me. If it is doing this, I'd like to know and am driven to help you figure it out for both our sakes.

Thanks @AndrewFarley; I'm using 3.6 (latest version). If I change the wait_timeout to 28800, this completely avoids the problem.

e.g., this will fail:
mysql -uroot -p -e"SET GLOBAL wait_timeout=60;"

but, this will work:
mysql -uroot -p -e"SET GLOBAL wait_timeout=28800;"

Here is the code I've used to initialize sequelize.

"use strict";

// http://sequelize.readthedocs.org/en/1.7.0/articles/express/

var fs        = require("fs");
var path      = require("path");
var Sequelize = require("sequelize");
var basename  = path.basename(module.filename);
var env       = process.env.NODE_ENV || "development";
var config    = require(__dirname + '/../config/config.json')[env];
var sequelize = new Sequelize(config.database, config.username, config.passwor
d, config);
var db        = {};

fs
  .readdirSync(__dirname)
  .filter(function(file) {
    return (file.indexOf(".") !== 0) && (file !== basename);
  })
  .forEach(function(file) {
    var model = sequelize["import"](path.join(__dirname, file));
    db[model.name] = model;
  });

Object.keys(db).forEach(function(modelName) {
  if ("associate" in db[modelName]) {
    db[modelName].associate(db);
  }
});

db.sequelize = sequelize;
db.Sequelize = Sequelize;

module.exports = db;

The config file is just:

{
  "development": {
    "username": "root",
    "password": null,
    "database": "development",
    "host": "127.0.0.1",
    "dialect": "mysql"
  }
}

It seems that mysql pooling is a preferred fix to deal with automatic reconnects, but not sure exactly where the breakdown may be:

https://github.com/felixge/node-mysql#pooling-connections
http://stackoverflow.com/questions/13018227/reproduce-mysql-error-the-server-closed-the-connection-node-js?rq=1

Ah yeah, that may be why I don't have that problem. I remember reading somewhere that they recommend pooling on any production-level endpoint. I use the same index.js provided template as you to load sequelize, but my config looks like the following and I have no problems with timeout or reconnection...

{
  "development": {
    "username": "root",
    "password": "root",
    "database": "app_development",
    "host": "127.0.0.1",
    "dialect": "mysql",
    "pool": {
      "maxConnections": 100,
      "maxIdleTime": 1000
    },
    "define": {
      "underscored": true,
      "freezeTableName": true,
      "paranoid": true,
      "charset": "utf8"
    }
  }
}

Thanks @AndrewFarley ... I just gave that a try. Seems it's still relying on the wait_timeout specified in the mysql config (my.cnf) file and not utilizing these settings, because it's still timing out using the wait_timeout. Did you have to specify a separate section to catch the error?

The errors are always the same, and posted previously (re-pasted below).

events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Connection lost: The server closed the connection.
    at Protocol.end (../node_modules/mysql/lib/protocol/Protocol.js:103:13)
    at Socket.<anonymous> (../node_modules/mysql/lib/Connection.js:102:28)
    at Socket.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

@dxdc Ah, no I don't have a section to catch the error... but I see your point, but I've bypassed the need to "retry" completely because after a discussion with a fellow node engineer, I came to the conclusion that a retry mechanism isn't really asynchronous friendly. I'm making an API which responds to clients and has some backend repeating/scheduled jobs at intervals.

So, instead of a retry mechanism, on every attempt to use the database I'm simply catch()-ing errors. If it was on the request from a client, I send a HTTP 503 back to the client to tell them to try again later. If it was a backend/scheduled task, I merely defer it to try again later, with an exponential back-off algorithm. This overcomes the need to engineer a retry mechanism, and it allows for faster responses to requests. This approach may not work for you depending on your platform though, but I generally feel this is the best approach as it won't keep around connections that hang and wait for the database to keep re-trying to become available. Because at scale, having tons of requests waiting around sounds dangerous and can lead to that stampede effect when your database becomes available and you have tons of requests waiting to process.

If needed I can go cook up some code samples, but I think you get the gist?

Cheers!

Thanks @AndrewFarley. Seems I'm still stuck though with this, but not sure what the issue is..will continue to investigate. Appreciate your help.

Also, to add to my comments above, whenever it re-attempts to use the database on a subsequent request, the framework (unsure if it's sequelize or mysql) retries the connection to the db.

Anyway, good luck hope you figure it out @dxdc

Hi,

After setting wait_timeout to 30, I too face this problem. My app exits after throwing the error. Looks like my setup is similar to @dxdc

Would be helpful if someone shows how to handle this.

I'm on sequelize 3.6.0

Thanks,
mano

I set the wait_timeout=28800 on the mysql.
I'll notice the result as soon as possible.

Thanks in advance.

@herlon214, @dxdc tells me when set to 28800 the problem does not arise!!

Hey,

I have resolved reconnect issues in our project. Here are points I found.

_Don't catch Sequelize connection error_

Previously we catch all db error and return corresponding errors to our client like 503, service down,please try later. It works properly except db back to normal. I doubted that Sequelize could not reconnect automatically because it did not detect connect hidden errors.

So, currently we don't catch connection errors. And it can reconnect as expected.

if(error instanceof Sequelize.ConnectionError){
  throw error ;
}

_But_, if we don't catch it ourselves, the control is passed to Sequelize itself and the response returned very slow. Is there any way to inform Sequelize that connect error happened, and responds by ourselves?

Thanks in advance.

@wangpin34 Hmm, you catching the error later on should actually not matter in terms of our error handling.

@mickhansen In fact we did not catch connection errors so that Sequelize could recovery from the connection lost. But, it make the app responds very slow. And, we strong want to know if there is way to make Sequelize auto recovery from connection lost. Thanks.

Hi, did this issue ever get resolved?

I found for me that the issue was a sessions module.. changed from connect-session-mysql to connect-session-sequelize and it's working great now.

Any updates on this? What's the currently recommended way to handle reconnects, even if that way isn't native to sequelize?

Thanks!

I getting:

SequelizeConnectionError: Connection lost: The server closed the connection.
File "/home/ubuntu/x/node_modules/sequelize/lib/dialects/mysql/connection-manager.js", line 94, in Handshake.

{
parent: {
code: PROTOCOL_CONNECTION_LOST,
fatal: True
}
}

And as @wangpin34 everything starts to move slow.

@wangpin34 How did you solve this?

Looks like this is related not only to MySQL but also in PostgreSQL.
We are using RDS and today near 5AM, amazon decided to restart instance. Application didn't reconnected until I manually restarted it. Is there any way to handle such situation?

+1 for this new feature !

@jontelm Till now no solution from Sequelized side. May be we can have a timeout control to make it back to user quickly.

Just experienced a restarted Postgres instance due to this (something no developer wants to see) but isnt the end of the world... my db came back up fine

2017-05-06 03:47:43 UTC [11320-3] xxx@xxx HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2017-05-06 03:47:43 UTC [28769-2] WARNING:  terminating connection because of crash of another server process
2017-05-06 03:47:43 UTC [28769-3] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-05-06 03:47:43 UTC [28769-4] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2017-05-06 03:47:43 UTC [3594-25] LOG:  all server processes terminated; reinitializing
2017-05-06 03:47:43 UTC [12817-1] LOG:  database system was interrupted; last known up at 2017-05-06 03:45:24 UTC
2017-05-06 03:47:43 UTC [12817-2] LOG:  database system was not properly shut down; automatic recovery in progress
2017-05-06 03:47:43 UTC [12817-3] LOG:  redo starts at 9/98717168
2017-05-06 03:47:43 UTC [12817-4] LOG:  invalid record length at 9/98F875D8: wanted 24, got 0
2017-05-06 03:47:43 UTC [12817-5] LOG:  redo done at 9/98F875B0
2017-05-06 03:47:43 UTC [12817-6] LOG:  last completed transaction was at log time 2017-05-06 03:47:42.537368+00
2017-05-06 03:47:44 UTC [12817-7] LOG:  MultiXact member wraparound protections are now enabled
2017-05-06 03:47:44 UTC [3594-26] LOG:  database system is ready to accept connections
2017-05-06 03:47:44 UTC [12821-1] LOG:  autovacuum launcher started

the end of the world is when your web servers don't reconnect...

here is the stack. Can sequelize reconnect or do we have to fix this at the pg driver level?
(pg version v6.1.2)

03:47:45 05-06-17 { Error: This socket has been ended by the other party
    at Socket.writeAfterFIN [as write] (net.js:291:12)
    at Connection.query (/root//node_modules/pg/lib/connection.js:204:15)
    at Query.submit (/root//node_modules/pg/lib/query.js:138:16)
    at Client._pulseQueryQueue (/root//node_modules/pg/lib/client.js:307:24)
    at Client.query (/root//node_modules/pg/lib/client.js:335:8)
    at Query.run (/root//node_modules/sequelize/lib/dialects/postgres/query.js:62:106)
    at /root//node_modules/sequelize/lib/sequelize.js:849:20
    at /root//node_modules/retry-as-promised/index.js:40:21
    at Promise._execute (/root//node_modules/bluebird/js/release/debuggability.js:300:9)
    at Promise._resolveFromExecutor (/root/node_modules/bluebird/js/release/promise.js:481:18)

+1

Anyone has informations about this feature ? The issue has been created the 4 Aug 2014 and we are the 15 June 2017.

@janmeier I assume this issue is now so stale no one will be working on it. Can this be closed as a wontfix so that instead of never being looked at ever again, people who still run into this can file a new issue to be tackled by whoever now works on sequelize?

+1

+1

Hi,

I have been confronted to the problem so I worked on this and found a way to make sequelize reconnect.
The request isn't replayed,the reconnection is done when a new request arrives.

My problem is : how can I test this ? For my own, I killed my db server and relaunch it; but for unit testing, how can I do that ?

EDIT : What do you think of using this (https://github.com/chrisblossom/toxiproxy-node-client) for killing the connection ?

+1 on this

@mickhansen @janmeier Any update ?

I verified the recent one (4.11.1) seems to have fixed this issue.

Hi everybody,

Fyi, you have to update your sequelize package to the latest stable version if you want to have this feature ;)

Regards,

@rimiti @rameshm1
thanks for the update.Do we need to pass any parameter as a part of config to make it working?

@ashish277d No. If you have the above-mentioned version (4.11.1), the reconnection behavior is default.

I couldn't see this feature/fix on the changelog, can anybody reference the appropriate commit for this? Thanks.

@ashish277d

@ashish277d No. If you have the above-mentioned version (4.11.1), the reconnection behavior is default.
馃憤

I did the following testing:

  • Setup a connection pool with at least 5 connections, at most 100 connections, and maximum idle time is 10 minutes.
  • Start the application and the connections are being established.
  • Shutdown the database (PostgreSQL 9.6 in my case). All connections in the connection pool are disconnected.
  • Start the database again.
  • Keep trying to send queries to the database.

According to my testing, it takes around 1 minute before a query can be sent to the database successfully.

Is it possible to configure how quickly are those connections being re-established?

it is still not working for me. I have latest version 4.13.8.
events.js:160
throw er; // Unhandled 'error' event
^
Error: Connection lost: The server closed the connection.
at Protocol.end (/home/default/node_modules/mysql/lib/protocol/Protocol.js:113:13)
at Socket. (/home/default/node_modules/mysql/lib/Connection.js:109:28)
at emitNone (events.js:91:20)
at Socket.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)
npm info lifecycle @qes/[email protected]~start: Failed to exec start script

@ashish-dehariya it is still not working for me. I have latest version 4.13.8.

I think Sequelize reconnects after a certain period of time (perhaps controlled by a timeout value?) instead of reconnect immediately, but I'm not sure about this.

did anyone manage to get this working? it doesnt work for me either.

@kostyay did anyone manage to get this working? it doesnt work for me either.

See my comment above.

Here's my thinking:

  • sequelize already has retry logic for queries
  • sequelize already manages the connection pool, and removes closed/errored connections

From what I understand the only missing piece (and the root of this issue) is that the retry around the queries will reuse the same connection, even if it died. I'll try to dig more into this and send a PR, but I think this might end up being a really simple fix - when retrying a query, yield the old connection and ask for a new one.

This configuration will already retry the queries on connection errors, but it will fail forever because it reuses the old connection:

// this goes to retry-as-promised for queries
retry: {
  match: [
    /SequelizeConnectionError/,
    /SequelizeConnectionRefusedError/,
    /SequelizeHostNotFoundError/,
    /SequelizeHostNotReachableError/,
    /SequelizeInvalidConnectionError/,
    /SequelizeConnectionTimedOutError/
  ],
  name: 'query',
  backoffBase: 100,
  backoffExponent: 1.1,
  timeout: 60000,
  max: Infinity
}

When a query fails due to a connection error, retry with exponential backoff with a max time of 60 seconds. This seems like a really robust way to manage reconnects/retries, just needs that fix for the connections and this should close a bunch of tickets around this.

Edit: PR submitted - https://github.com/sequelize/sequelize/pull/8961

im still unable to get this working...

according to the documentation (http://docs.sequelizejs.com/class/lib/sequelize.js~Sequelize.html) it states that there are defaults set for all the pool options..

my repro steps using mariadb 10.2:

1) start my api server that is using sequelize v4.37.10
2) issue a query (works fine)
3) stop the database
4) issue a query (fails due to db being down)
5) start the database
6) issue a query (continues to fail)

even after a few minutes it seems as though the connections arent reconnecting and im unable to query the db.

my config looks like so...

{
    dialect: 'mysql',
    dialectOptions: {
      multipleStatements: true,
      //this is to force mysql2 lib to coerce decimal numbers into actual numbers instead of strings https://github.com/sequelize/sequelize/issues/8019
      decimalNumbers: true
    },
    replication: {
      write: {
        host: process.env.MYSQL_MASTER_HOST,
        port: process.env.MYSQL_MASTER_PORT,
        username: process.env.MYSQL_MASTER_USERNAME,
        password: process.env.MYSQL_MASTER_PASSWORD
      },
      read: [{
        host: process.env.MYSQL_SLAVE_HOST,
        port: process.env.MYSQL_SLAVE_PORT,
        username: process.env.MYSQL_SLAVE_USERNAME,
        password: process.env.MYSQL_SLAVE_PASSWORD
      }]
    },
    pool: {
      max: process.env.MYSQL_MASTER_CONNECTION_LIMIT,
      min: 1,
      idle: 10000
    },
    define: {
      timestamps: false
    }
}

this recently affected us because there was an unexpected DB crash with our AWS RDS instance.

@stringbeans
Did you ever figure this out? Seeing this on production right now, Aurora Postgres failover and Sequelize doesn't reconnect (for over 40 minutes). Before when we didn't have replication I think it worked OK. But after adding replication config it won't reconnect on failover.

I ended up listening for a SequelizeConnectionRefusedError or SequelizeConnectionError in my middleware and if it detects that it restarts the process. A bit hacky but it works.

After a lot testing, I can confirm that this issue still exists under replication mode.
A quick & dirty mitigation is increasing pool.acquire to a huge value, and configure retry like @contra mentioned above.

~I am trying to locate the bug and maybe make a patch for it.~ (not anymore as we moved to TypeORM 馃槩 )

My connection to MySQL is lost and never recovers.

Sequelize: 4.41.1
Node: 8.12.0

@kingjerod would you mind posting a code snippet for your middleware hack?

@davelandry

My processes are managed with PM2 so exiting will cause PM2 to restart the process. There is another middleware before this that catches errors and sets the ctx.body.error message. There is some typescript in here that can be easily stripped out.

let exiting: boolean = false;
async function failoverCheck(ctx: Koa.Context, next) {
  await next();
  if (exiting) {
    return;
  }

  if (ctx.body && ctx.body.error && typeof ctx.body.error === "string") {
    const error: string = ctx.body.error.toLowerCase();
    const sequelizeErrors = [
      "SequelizeConnectionRefusedError",
      "SequelizeConnectionError"
    ];
    for (const sequelizeError of sequelizeErrors) {
      if (error.includes(sequelizeError.toLowerCase())) {
        exiting = true;
        // We wait 10 seconds so that when it reconnects hopefully the failover is finished
        setTimeout(() => {
          process.emit("SIGTERM");
        }, 10000);
      }
    }
  }
}

Here's my thinking:

* sequelize already has retry logic for queries

* sequelize already manages the connection pool, and removes closed/errored connections

From what I understand the only missing piece (and the root of this issue) is that the retry around the queries will reuse the same connection, even if it died. I'll try to dig more into this and send a PR, but I think this might end up being a really simple fix - when retrying a query, yield the old connection and ask for a new one.

This configuration will already retry the queries on connection errors, but it will fail forever because it reuses the old connection:

// this goes to retry-as-promised for queries
retry: {
  match: [
    /SequelizeConnectionError/,
    /SequelizeConnectionRefusedError/,
    /SequelizeHostNotFoundError/,
    /SequelizeHostNotReachableError/,
    /SequelizeInvalidConnectionError/,
    /SequelizeConnectionTimedOutError/
  ],
  name: 'query',
  backoffBase: 100,
  backoffExponent: 1.1,
  timeout: 60000,
  max: Infinity
}

When a query fails due to a connection error, retry with exponential backoff with a max time of 60 seconds. This seems like a really robust way to manage reconnects/retries, just needs that fix for the connections and this should close a bunch of tickets around this.

Edit: PR submitted - #8961

@contra Quick question: I see that the documentation mentions the retry.match and retry.max properties, but none of the other ones. Are those properties undocumented? Thanks.

@gwilakers Yeah, they are undocumented.

BTW for retrying queries - this bug is currently breaking some cases: https://github.com/sequelize/sequelize/issues/10453

@contra Okay, thanks for confirming.

Ironically, I just subscribed to that issue this morning haha. Thanks for the update.

This doesn't seem to work in one specific case:
1) first start sequelize with sequelize.sync()
2) second start mysql db

The connection to the database is established fine and sequelize is operational, however model sync has been skipped and is never retried.

Any ideas if I'm doing something wrong?

Was this page helpful?
0 / 5 - 0 ratings