Loopback: Performance comparison Loopback and Express/Mongoose

[dnovice]

I'm getting started in node.js and evaluating alternatives for a new project, and have a performance concern.

The use case is to support a very simple HTTP POST of a JSON document of about 300 bytes, validate the fields, and persist that to a MongoDb collection.
I wrote the app twice: Once using Loopback with loopback-connector-mongodb, and once using simple express with mongoose. To write the loopback app, I pretty much just did: ( slc loopback; npm install loopback-connector-mongodb; slc loopback:datasource; slc loopback:model; slc loopback:boot-script) and the app was ready to run, except for some tuning tweaks I'll describe for connection pool and server queue.

Then I ran some load tests using JMeter to submit concurrent POST requests to confirm that I could support some load. I tweaked the OS (Linux) kernel to support multiple connections, set the http server to allow a backlog (app.listen(port,hostname,backlog), and set the mongo connection pool up to 500 connections. In JMeter I ran 1000 threads for about 10 minutes, with some time delay between each post.

Now finally to my concern: The mongoose version ran fine, with no connection errors, using relatively low CPU and having relatively quick (2ms) average response time. However the Loopback version performed worse: It had socket timeout connection errors (.25%), used 3 times the amount of CPU, and much longer average response times (80ms). Nothing else changed between runs, and this was repeatable, and independent of mongo file creation issues.

My questions are: Is this expected? Is there something I can do to easily determine the loopback bottleneck, or boost performance?

[robotiko]

+1

[raymondfeng]

Did you configure the mongo driver connection pool size inside the data source settings?

[dnovice]

I think so. Do you mean this?
$ more datasources.json
{
"signage": {
"name": "signage",
"connector": "mongodb",
"url": "mongodb://127.0.0.1/signage?maxPoolSize=100"
}
}

[raymondfeng]

Yes, per http://docs.mongodb.org/manual/reference/connection-string/#connection-pool-options

[raymondfeng]

Do you use the same connection string for mongoose too?

[dnovice]

I intended to. In fact in the end there was a slight difference here. In mongoose version I have in db/mongoDB.js:
mongoose.connect('mongodb://localhost:27017/signage?maxPoolSize=1000&minPoolSize=500');
but if I attempted this in datasources.json I have two issues: a) minPoolSize is not supported and b) Upon app startup, the app crashes with
"Connection fails: { [MongoError: connection 630 to 127.0.0.1:27017 closed]
name: 'MongoError',
message: 'connection 630 to 127.0.0.1:27017 closed' }
"
So I reduced the size of the pool in Loopback version. Actually, I ran the Loopback version with 500 rather than 100 as max connections, with essentially the same results as with 100.

[raymondfeng]

1. Can you check the version of mongodb driver that mongoose use? We need to make sure the same version is used for mongoose and loopback-connector-mongodb.
2. Can you make sure the connection pool settings are the same for mongoose and loopback?
3. What's your ulimit -a?

[dnovice]

Thank you so much for assisting me in this issue.
MongoDB shell version: 3.0.3
Loopback:
$ npm ls | grep mongodb
├─┬ [email protected]
│ └─┬ [email protected]
│ ├─┬ [email protected]

Mongoose:
$ npm ls | grep mongodb
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]

Please let me know if you think the differences in the above versions are significant.

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 106496
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 15000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I changed the mongoose connection string so that it too only had maxPoolSize=500. It had essentially the same results as the mongoose test from before. However, it brings up two additional questions I have with the Loopback configurations:

1. Why is it that it does not support minPoolSize?
2. Why is it that I cannot set maxPoolSize to 1000 when I can with the mongoose connection. Is the value interpreted differently?

Thanks again for your help.

[rus0000]

+1

[raymondfeng]

The connection string is passed to mongodb driver directly. If minPoolSize is supported by the driver, I don't see why it's not effective.

[rus0000]

@dnovice Could you share your test-case.

[dnovice]

Rus: I don't know how to share binary files in this part of github. Please see https://groups.google.com/forum/#!topic/loopbackjs/QWTKVG814ws
for the tar file.

[superkhau]

@raymondfeng Assigning to you as you are already working on this.

[dnovice]

I would appreciate a response to this. Is there an expectation of a fix or is this an architectural issue?

[raymondfeng]

@dnovice Sorry I was side-tracked. Will look into your sample project.

[raymondfeng]

I found out that mongoose use poolSize instead of 'maxPoolSize` to control the connection pool. See http://mongoosejs.com/docs/connections.html. As a result, your mongoose setup actually uses a connection pool of 5 connections. Can you adjust that?

[raymondfeng]

The mongoose test has the following index set up -

asrunLogSchema.index({accountId:1, site:-1});

Do you want to do so with LoopBack too? See an example at https://github.com/strongloop/loopback-connector-mongodb/blob/master/test/mongodb.test.js#L17-L22. You should run automigrate or autoupdate to create the indexes or manually create it.

[raymondfeng]

I'm seeing avg response time for loopback on my Mac is 3-4 ms.

[dnovice]

Is that under load or just one-off?

Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

I'm seeing avg response time for loopback on my Mac is 3-4 ms.

‹
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-122972291
.

[dnovice]

Sorry, Raymond.
I¹m a newbie on Loopback and I don¹t know what you are talking about. Could
you show me where in my code I submitted where I would put that, if not in
the connection string in server/datasources.json? If in the connection
string, I¹ve tried that with no change.

-David

From: Raymond Feng [email protected]
Reply-To: strongloop/loopback
[email protected]>
Date: Monday, July 20, 2015 at 1:57 PM
To: strongloop/loopback [email protected]
Cc: David Novice [email protected]
Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

I found out that mongoose use poolSize instead of 'maxPoolSize` to control
the connection pool. See http://mongoosejs.com/docs/connections.html. As a
result, your mongoose setup actually uses a connection pool of 5
connections. Can you adjust that?

‹
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-122967954
.

[raymondfeng]

1. I'm referring to:

mongoose.connect('mongodb://localhost:27017/signage?maxPoolSize=1000&minPoolSize=500');

It should be:

mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');

1. I use your jmeter test plan to run the tests.

[raymondfeng]

[dnovice]

So you are working on the mongoose version, not the loopback version, of my
code.
The mongoose version does indeed return low-millisecond response times.
I¹m currently interested to know if the loopback version can be made to work
as quickly as the mongoose version.

Thanks,

Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

1. I'm referring to:
   mongoose.connect('mongodb://localhost:27017/signage?maxPoolSize=1000&minPool
   Size=500');
   It should be:
   mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');
2. I use your jmeter test plan to run the tests.
   ‹
   Reply to this email directly or view it on GitHub
   https://github.com/strongloop/loopback/issues/1469#issuecomment-123006164
   .

[raymondfeng]

No, the test I ran is the LoopBack version

Sent from my iPhone 6 Plus

On Jul 20, 2015, at 12:58 PM, dnovice [email protected] wrote:

So you are working on the mongoose version, not the loopback version, of my
code.
The mongoose version does indeed return low-millisecond response times.
I¹m currently interested to know if the loopback version can be made to work
as quickly as the mongoose version.

Thanks,

From: Raymond Feng [email protected]
Reply-To: strongloop/loopback
[email protected]>
Date: Monday, July 20, 2015 at 3:49 PM
To: strongloop/loopback [email protected]
Cc: David Novice [email protected]
Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

1. I'm referring to:
   mongoose.connect('mongodb://localhost:27017/signage?maxPoolSize=1000&minPool
   Size=500');
   It should be:
   mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');
2. I use your jmeter test plan to run the tests.
   ‹
   Reply to this email directly or view it on GitHub
   https://github.com/strongloop/loopback/issues/1469#issuecomment-123006164
   .

—
Reply to this email directly or view it on GitHub.

[dnovice]

OK, but then I still don’t know where to change the code in the loopback
version to properly set the connection pool size.

Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

No, the test I ran is the LoopBack version

Sent from my iPhone 6 Plus

On Jul 20, 2015, at 12:58 PM, dnovice [email protected] wrote:

So you are working on the mongoose version, not the loopback version, of my
code.
The mongoose version does indeed return low-millisecond response times.
I¹m currently interested to know if the loopback version can be made to work
as quickly as the mongoose version.

Thanks,

Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

1. I'm referring to:
   mongoose.connect('mongodb://localhost:27017/signage?maxPoolSize=1000&minPool
   Size=500');
   It should be:
   mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');
2. I use your jmeter test plan to run the tests.
   ‹
   Reply to this email directly or view it on GitHub
   https://github.com/strongloop/loopback/issues/1469#issuecomment-123006164
   .

―
Reply to this email directly or view it on GitHub.

―
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-123013937
.

[raymondfeng]

• For mongoose, change the poolSize at mongoose/db/mongoDB.js:

mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');

• For loopback, change the poolSize at loopback/server/datasources.json:

"signage": {
    "name": "signage",
    "connector": "mongodb",
    "url": "mongodb://127.0.0.1/signage?maxPoolSize=100"
  }

In my tests, the pool size doesn't matter so much for your case.

[dnovice]

Thanks for the specifics. I have in fact tried that change in
datasources.json, as I indicated before. I agree that in this test, pool
size doesn¹t matter.
I still have easily-reproducible significant performance differences between
the mongoose and the loopback versions.
Where do you suggest I go from here?

David

Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

• For mongoose, change the poolSize at mongoose/db/mongoDB.js:
  mongoose.connect('mongodb://localhost:27017/signage?poolSize=100');
• For loopback, change the poolSize at loopback/server/datasources.json:
  "signage": {
  "name": "signage",
  "connector": "mongodb",
  "url": "mongodb://127.0.0.1/signage?maxPoolSize=100"
  }
  In my tests, the pool size doesn't matter so much for your case.

‹
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-123019684
.

[JoshWilliams92]

Loopback comes with an SDK so it's a perfect solution for my mobile app. Is it possible to use loopback as an alternative to mongoose?

[dnovice]

Loopback wraps the mongoose functionality and uses it underneath the covers.

From: JoshWilliams92 [email protected]
Reply-To: strongloop/loopback
[email protected]>
Date: Wednesday, November 11, 2015 at 3:39 PM
To: strongloop/loopback [email protected]
Cc: David Novice [email protected]
Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

Is it possible to use loopback as an alternative to mongoose?

‹
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-155903415
.

[JoshWilliams92]

I have done some research on IBM and concluded that loopback allows you to create .json files and .js files which is your data model and object model. Works perfectly for me, except I need to know how I can use it with gridfs instead of mongodb.

[mojo5000]

Did this thread ever get some resolution? Are there updated performance comparisons available?

[dnovice]

Others who tried to reproduce my issue stated that they did not see an
issue. I was consistently able to reproduce the issue, but that¹s as far as
it got.
-David

From: Steve Motola [email protected]
Reply-To: strongloop/loopback
[email protected]>
Date: Wednesday, December 2, 2015 at 1:41 PM
To: strongloop/loopback [email protected]
Cc: David Novice [email protected]
Subject: Re: [loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

Did this thread ever get some resolution? Are there updated performance
comparisons available?

‹
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-161393569
.

[tongfa]

dnovice,

Early on, you mentioned you tweaked something in the linux kernel. What did you do exactly? I'm wondering if that impacted your test results, but no one else did this.

[dnovice]

I did this just to try to get it to work better, after it was clear to me
that Loopback was much slower than Mongoose by itself.

sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.core.somaxconn = 65536

From: Chris [email protected]
Reply-To: strongloop/loopback
[email protected]>
Date: Thursday, April 14, 2016 at 10:11 PM
To: strongloop/loopback [email protected]
Cc: David Novice [email protected]
Subject: Re: [strongloop/loopback] Performance comparison Loopback and
Express/Mongoose (#1469)

dnovice,

Early on, you mentioned you tweaked something in the linux kernel. What did
you do exactly? I'm wondering if that impacted your test results, but no
one else did this.

‹
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/strongloop/loopback/issues/1469#issuecomment-210252425

[m0onspell]

We are currently going to try some language/framework combinations to see if they can improve our PHP/Yii2 API performance under high load.
I was looking at loopback as possible option to try, but thankfully, I found this thread.
I'll just go with express, since maintainers didn't provide any reasonable advice.
Thanks for saving my time.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.