Loopback: Automigrate Memory Leak

Created on 6 Mar 2015 · 24Comments · Source: strongloop/loopback

(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at DataSource.addListener (events.js:179:15)
    at DataSource.once (events.js:204:8)
    at MySQL.query (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/lib/mysql.js:133:28)
    at MySQL.SqlConnector.command (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:45:15)
    at MySQL.SqlConnector.dropTable (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:411:8)
    at /Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:389:10
    at /Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:125:13
    at Array.forEach (native)
    at _each (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:46:24)
    at Object.async.each (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:124:9)

Loopback version 2.8.0

triaging

Source

brunocascio

Most helpful comment

LoopBack uses new DataSource(...) to create data source synchronously. It takes some time to create a connection pool to the database server. The constructor returns before the connection pool is ready. SQL statements will be deferred using once('connected'). To avoid the warnings, you can run automigrate upon the connected event. For example,

var ds = app.dataSources.mysql;
if(ds.connected) {
  ds.automigrate(...);
} else {
  ds.once('connected', function() {
    ds.automigrate(...);
  });
}

raymondfeng on 15 Jun 2015

👍16 ❤1 🎉1

All 24 comments

LoopBack try to defer a CRUD method invocation if the data source has not been connected to the DB. The scheduled method will be called once the connected event is fired and the listener will be removed. Can you check what's going on with your DB connection?

raymondfeng on 6 Mar 2015

I'm also getting this warning. Everything continues to run fine, but it's annoying none the less. Using a local DB connection, nothing seems to be wrong there.

NJordan72 on 8 Apr 2015

I also get this warning when running against a local MySQL instance. The update process seems to run fine.

thovden on 11 Jun 2015

violet-day on 15 Jun 2015

var ds = app.dataSources.mysql;
if(ds.connected) {
  ds.automigrate(...);
} else {
  ds.once('connected', function() {
    ds.automigrate(...);
  });
}

raymondfeng on 15 Jun 2015

👍16 ❤1 🎉1

@raymondfeng Is this good to close?

superkhau on 7 Jul 2015

I have the same problem with postgres. I tried the solution suggested by @raymondfeng but I still get the same error message.

pbraun on 11 Aug 2015

(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at DataSource.addListener (events.js:179:15)
    at DataSource.once (events.js:204:8)
    at stillConnecting (/Users/peter/Projects/Project1/node_modules/loopback-connector-postgresql/lib/postgresql.js:147:14)
    at PostgreSQL.query (/Users/peter/Projects/Project1/node_modules/loopback-connector-postgresql/lib/postgresql.js:166:7)
    at dropTableFun (/Users/peter/Projects/Project1/node_modules/loopback-connector-postgresql/lib/postgresql.js:1333:10)
    at PostgreSQL.dropTable (/Users/peter/Projects/Project1/node_modules/loopback-connector-postgresql/lib/postgresql.js:1338:3)
    at /Users/peter/Projects/Project1/node_modules/loopback-connector-postgresql/node_modules/loopback-connector/lib/sql.js:389:10
    at /Users/peter/Projects/Project1/node_modules/async/lib/async.js:125:13
    at Array.forEach (native)
    at _each (/Users/peter/Projects/Project1/node_modules/async/lib/async.js:46:24)

pbraun on 11 Aug 2015

Never mind. The suggested solution works fine.

pbraun on 11 Aug 2015

I'm getting the same issue but with Mongodb upon server startup.

(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at DataSource.addListener (events.js:239:17)
at DataSource.once (events.js:265:8)
at DataSource.ready (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/datasource.js:2105:8)
at stillConnecting (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/dao.js:365:18)
at Function.DataAccessObject.create (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/dao.js:160:27)
at Object.eval as forward
at Function.eval as create
at Function.Note.createFakeData (/Users/petergiannopoulos/Projects/otolane/occ/common/models/note.js:6:17)
at Object.module.exports as func
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:287:9
at iterate (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:146:13)
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:157:25
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:289:7
at iterate (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:146:13)
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:157:25
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:289:7
(node) warning: possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at DataSource.addListener (events.js:239:17)
at DataSource.once (events.js:265:8)
at DataSource.ready (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/datasource.js:2106:8)
at stillConnecting (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/dao.js:365:18)
at Function.DataAccessObject.create (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-datasource-juggler/lib/dao.js:160:27)
at Object.eval as forward
at Function.eval as create
at Function.Note.createFakeData (/Users/petergiannopoulos/Projects/otolane/occ/common/models/note.js:6:17)
at Object.module.exports as func
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:287:9
at iterate (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:146:13)
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:157:25
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:289:7
at iterate (/Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:146:13)
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/node_modules/async/lib/async.js:157:25
at /Users/petergiannopoulos/Projects/otolane/occ/node_modules/loopback-boot/lib/executor.js:289:7

petergi on 16 Oct 2015

@petergi Did you try the solution provided by @raymondfeng above?

superkhau on 17 Oct 2015

Yeah no dice. But I tracked it down to using faker on a before hook. It's what seems to trigger the warning. I was using faker to generate fake data to work with.

Thanks!

Sent from my iPhone

On Oct 16, 2015, at 9:46 PM, Simon Ho [email protected] wrote:

@petergi Did you try the solution provided by @raymondfeng above?

—
Reply to this email directly or view it on GitHub.

petergi on 17 Oct 2015

@petergi Can you provide a link to a test project on GitHub? See https://github.com/strongloop/loopback/wiki/Reporting-issues#bug-report

superkhau on 22 Oct 2015

I wrote a simple data-fixture script:

var server = require('./server');
var ds = server.dataSources.postgre;
var fs = require("fs");

var fixtures = [
  'some',
  'model',
  'names'
];

function migrate(modelName, data) {
  var model = server.models[modelName];
  var count = data.length;

  ds.automigrate(modelName, function(er){
    if (er) throw er;
    data.forEach(function(item) {
      model.create(item, function(er, result) {
        if (er) {
          throw er;
          console.log(er);
          return;
        }
        console.log('Record created:', result);
        count--;
        if(count === 0) {
          console.log('done');
          ds.disconnect();
        }
      });
    });    
  });  
}

function readData(modelName) {
  var content = fs.readFileSync('data-fixtures/' + modelName + '.json');
  // Define to JSON type
  return JSON.parse(content);  
}

for (var i = 0; i < fixtures.length; i++) {
  var data = readData(fixtures[i]);
  migrate(fixtures[i], data);
}

the script runs fine, and all looks fine in the db, but I am getting

_(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at DataSource.addListener (events.js:179:15)
at DataSource.once (events.js:204:8)
at migrate (/home/.../strongloop-test/server/data-fixture.js:152:8)
at Object. (/home/.../strongloop-test/server/data-fixture.js:184:3)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)_

The solution https://github.com/strongloop/loopback/issues/1186#issuecomment-111934355 by @raymondfeng does not solve the problem.

// it is ugly, just for testing
function migrate(modelName, data) {
  var model = server.models[modelName];
  var count = data.length;
  if(ds.connected) {
      ds.automigrate(modelName, function(er){
        if (er) throw er;
        data.forEach(function(item) {
          model.create(item, function(er, result) {
            if (er) {
              throw er;
              console.log(er);
              return;
            }
            console.log('Record created:', result);
            count--;
            if(count === 0) {
              console.log('done');
              ds.disconnect();
            }
          });
        });    
      });
  } else {
    ds.once('connected', function() {
      ds.automigrate(modelName, function(er){
        if (er) throw er;
        data.forEach(function(item) {
          model.create(item, function(er, result) {
            if (er) {
              throw er;
              console.log(er);
              return;
            }
            console.log('Record created:', result);
            count--;
            if(count === 0) {
              console.log('done');
              ds.disconnect();
            }
          });
        });    
      });
    });
  } 
}

_(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at DataSource.addListener (events.js:179:15)
at DataSource.once (events.js:204:8)
at migrate (/home/.../strongloop-test/server/data-fixture.js:151:8)
at Object. (/home/.../strongloop-test/server/data-fixture.js:182:3)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)_

BenjaminHorn on 12 Nov 2015

I'm having basically the same problem. Trying to load static lookup data in a boot script, and I guess all the callbacks attachments are stacking up before the connection succeeds in the background?

warriorpostman on 29 Dec 2015

aldegoeij on 9 Mar 2016

:+1:
I am also facing the same issue after upgrading my node to v4.2.x
Earlier I was using v0.10.25 (legacy) and I didn't face this error, and I am sure reverting back to that version will solve the problem.
Do we have any better fix?

yashmurty on 21 Mar 2016

@aldegoeij @yashmurty @warriorpostman @BenjaminHorn and everybody who is still having the same problem, I cannot reporduce the error with mysql, however I can reproduce with postgres connector, so the way I got it to work, is the following:
in your app, go to : node_modules\loopback-connector\lib\sql.js and line 395, after this line

if (!this.dataSource.connected) {

please add:

this.dataSource.setMaxListeners(0);

let me if this works for all of you, and I will update the code. Thanks.

loay on 7 Apr 2016

👍5 👎3

@loay thanks for investigating this. It works for me!

BenjaminHorn on 7 Apr 2016

@BenjaminHorn Thanks.
I am closing the issue. If the problem is still persisting, please feel free to leave a comment and I will re-open. Thanks.

loay on 7 Apr 2016

@loay I have same problem yet. I need help please, i have this versions:

    "loopback": "^3.22.0",
    "loopback-boot": "^2.6.5",
    "loopback-component-explorer": "^6.2.0",
    "loopback-connector-postgresql": "^3.4.0",

nternouski on 23 Nov 2018

@nternouski Sorry, I haven't worked on this project for a while. You probably want to mention one of the current maintainers to help you.
Also, I suggest that you try one of the above mentioned solutions and if that does not work, you probably want to ask on Loopback google groups or open another issue since this issue has been closed for over 2 years !

loay on 23 Nov 2018

Thanks. I try the solutions but did not work. On other hand, modify node_modules\loopback-connector\lib\sql.js is not a good practice because I use docker and not recommended.

@bajtos @fabien @clarkorz @ebarault @zbarbuto @nitro404 The problem is similar but using postgresSQL ..

(node) warning: possible EventEmitter memory leak detected. 11 connected listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at DataSource.addListener (events.js:179:15)
    at DataSource.once (events.js:204:8)
    at MySQL.query (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/lib/mysql.js:133:28)
    at MySQL.SqlConnector.command (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:45:15)
    at MySQL.SqlConnector.dropTable (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:411:8)
    at /Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:389:10
    at /Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:125:13
    at Array.forEach (native)
    at _each (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:46:24)
    at Object.async.each (/Users/brunocascio/Documents/Projects/Backend/NodeJS/loopback/WTE/node_modules/loopback-connector-mysql/node_modules/async/lib/async.js:124:9)

Loopback version 2.8.0

when i run:

var server = require("./server");
var ds = server.dataSources.accountDS; // ds is a postgresSQL database
var lbTables = []; // Array of 12 elements
ds.setMaxListeners(0);
ds.automigrate(lbTables, function (er) {
    ds.disconnect();
    if (er) throw er;
    console.log("Loopback tables [" + lbTables + "] created in ", ds.adapter.name);
});

And use that versions:

    "loopback": "^3.22.0",
    "loopback-boot": "^2.6.5",
    "loopback-component-explorer": "^6.2.0",
    "loopback-connector-postgresql": "^3.4.0",

nternouski on 23 Nov 2018

possible EventEmitter memory leak detected.

Key word here is "possible". The issue is that if you run automigrate on a bunch of tables before the datasource is fully attached, loopback will setup a bunch of once listeners to run the migration sql after the datasource _is_ attached:

  if (!this.dataSource.connected) {
    return this.dataSource.once('connected', function() {
      self.execute(sql, params, options, callback);
    });
  }

It's a relatively harmless node warning - but if you want to avoid it you can try setting up the attached listener yourself before running your migrations:

ds.once('attached', () => {
  // Now you can run your migrations and should not get the warning.
});

zbarbuto on 25 Nov 2018

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Patch Attributes needs to be updated on default user model

devotox · 4Comments

401 unauthorized instead of 403 forbidden while authenticated

daankets · 4Comments

Add info on how to "skin" or brand API Explorer

crandmck · 3Comments

Updating related model breaks sanitization

Overdrivr · 3Comments

Async boot scripts doc clarification

Overdrivr · 4Comments