Docker-alpine: Performance of Node.js is quite bad compared to other images

[paulrutter]

Hi,

We've been running Alpine containers in production for over a half year now, but performance of our applications is starting to become a real issue.
We're running Node.js (4.x) on the latest Alpine release (3.3).
We are quite happy with Alpine, because of its minimalistic approach, but a little less happy with its performance.

When comparing performance of an Alpine container against another "flavour", like CentOS or Ubuntu, we were quite shocked; the performance (of this particular script) on Alpine is three times as slow as on other images!

We _think_ this is caused by the fundamental choice of Alpine using "musl libc", but we're not sure yet.
Can you guys shed some light on this?
I haven't found any other issues about performance being bad when using Alpine, so either we're the first that encountered this issue, or we're doing something wrong ;-)

The following script we've used for testing performance:

   var instance = {};
    instance.a = 'asdfasdasdf';
    instance.b = 'asdfasdasdf';
    instance.c = 'asdfasdasdf';
    instance.d = 'asdfasdasdf';
    instance.e = 'asdfasdasdf';

    for (var i = 0; i < 5; i++) {
      var instance2 = {};
      instance2.a = instance;
      instance2.b = instance;
      instance2.c = instance;
      instance2.d = instance;
      instance2.e = instance;
      instance2.f = instance;
      instance = instance2;
    }

    var json = JSON.stringify(instance2);

    var test;
    for (var i = 0; i < 100; i++) {
       JSON.parse(json);
      for (var j = 0; j < 10000; j++) {
        test = test + 'a';
        test = test.substring(1, test.length - 1);
      }

    }

When running "time node app.js" command on an Alpine based image:

real 0m 4.99s
user 0m 4.90s
sys 0m 0.10s

When running "time node app.js" command on an CentOS based image:

real 0m1.682s
user 0m1.596s
sys 0m0.092s

Thanks in advance!

[rofl0r]

runs in 1.76 secs on sabotage linux, so musl is probably not the problem (node v0.12.4)

[ncopa]

@rofl0r and node v0.12.4 built with glibc on same hardware?

i can reproduce the issue here:
debian (official node:4.3): ~0.98s
alpine, node 4.3: ~3.75s

alpine node is approx 4x slower

[ncopa]

I tested with void linux, glibc and musl. They should be built with same flags and have same version. Both variants run at ~1.0 sec. So I believe this is not musl libc issue. It has probably more to do with how alpine linux build the package.

[paulrutter]

Thanks for your swift response, both @rofl0r and @ncopa.
I hope you find the culprit soon!

[paulrutter]

Any progress made on analyzing this issue?

[rofl0r]

yeah it's the specific node version shipped by alpine

[ncopa]

We noticed that node 5.10.x is significantly faster than node 4.4.x. I have not yet figured out what or why 4.4 is so slow.

[paulrutter]

Closing this issue, as it's not clear if the root cause is Alpine.

[shaunwarman]

Any other analysis around this? Just interested.

[corporatepiyush]

@paulrutter Have you tried Node v6.x LTS with Alpine v3.6 ?

[paulrutter]

Nope, i switched to a more common base image which did not have these issues. Ubunte and centos for example.

[nalingarg2]

I am running flask server on alpine and centos. I am seeing performance degradation of 2times in alpine!

[guikubivan]

Looks like this is not an issue anymore? Just tried it with node:10.15.1-alpine on Mac and got the following on first run (real and sys time was slightly faster on 2nd run):

real    0m 0.91s
user    0m 0.72s
sys     0m 0.17s

$ docker --version
Docker version 18.09.2, build 6247962

$ uname -vv
Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64

[thomasjungblut]

@guikubivan glad that you open this discussion again somewhat. On Friday we tried to update from carbon (full debian) to dubnium-alpine. We didn't have a good time so far, it was somewhat unstable (not sure why just yet) and saw 2x-4x slowdowns on some workloads.

We're going to switch to dubnium-debian-slim tomorrow and see how that will work out.

[paulrutter]

Alpine uses musl where other images use the more common glibc, so nodejs for alpine is compiled against musl. Not sure if the performance hit is caused by this compile time difference, but it looks to me that it is.
We now run nodejs from a centos based image (glibc) that has far better nodejs performance.
Not to bash on alpine, but for our usecases it doesn't work out well.