Immer: Drop of performances on node 9

[frinyvonnick]

Hi,

I tried the performance benchmark on different versions of node and i experienced a weird drop of performances.

Here are my computer specifications:

CPU: 2,7 GHz Intel Core i5
RAM: 8 Go 1867 MHz DDR3

On 8.4.0:

  performance
    ✓ just mutate (2ms)
    ✓ deepclone, then mutate (475ms)
    ✓ handcrafted reducer (23ms)
    ✓ immutableJS (66ms)
    ✓ immer - with autofreeze (371ms)
    ✓ immer - without autofreeze (181ms)

On 9.3.0:

  performance
    ✓ just mutate (5ms)
    ✓ deepclone, then mutate (324ms)
    ✓ handcrafted reducer (19ms)
    ✓ immutableJS (69ms)
    ✓ immer - with autofreeze (510ms)
    ✓ immer - without autofreeze (1378ms)

I didn't found out the cause of this.

[mweststrate]

That is remarkable! Did you try it a few times? You could try to put the immer tests first, as I didn't figure out how to expose the garbage collector api with jest yet, so the performance tests are not really isolated; the garbage collector might kick in at the wrong moment.

[frinyvonnick]

Yes i tried multiples times. To see if garbage collector impact tests i forced it before each test and skip other tests so we only test immer and each test is "isolated".

test script:

"test": "node --expose-gc node_modules/.bin/jest",

before each:

beforeEach(() => {
  global.gc()
})

Here are the results:

On 8.4.0:

performance
  ✓ immer - with autofreeze (466ms)
  ✓ immer - without autofreeze (326ms)
  ○ skipped 4 tests

On 9.3.0:

performance
  ✓ immer - with autofreeze (670ms)
  ✓ immer - without autofreeze (748ms)
  ○ skipped 4 tests

So garbage collector and other tests had an impact on my first results but there is still a drop of performances.

[mweststrate]

Just improved the test setup to leverage the garbage collector.

@frinyvonnick could you check again if the differences are still as big?

[frinyvonnick]

Here are the results with your last update:

On 8.4.0:

  performance
    ✓ just mutate (3ms)
    ✓ deepclone, then mutate (422ms)
    ✓ handcrafted reducer (19ms)
    ✓ immutableJS (74ms)
    ✓ immer (proxy) - with autofreeze (331ms)
    ✓ immer (proxy) - without autofreeze (165ms)
    ✓ immer (es5) - with autofreeze (445ms)
    ✓ immer (es5) - without autofreeze (377ms)

On 9.3.0:

  performance
    ✓ just mutate (5ms)
    ✓ deepclone, then mutate (246ms)
    ✓ handcrafted reducer (18ms)
    ✓ immutableJS (55ms)
    ✓ immer (proxy) - with autofreeze (509ms)
    ✓ immer (proxy) - without autofreeze (1296ms)
    ✓ immer (es5) - with autofreeze (392ms)
    ✓ immer (es5) - without autofreeze (614ms)

The main problem seems to be related to proxies despite of the overall augmentation

[nlepage]

Hi :wave:

I wrote a modified version at immutadot which runs each test in a loop and computes an average execution time.
This way garbage collector time should not mess too badly with execution time...

Node 8.9.4 :

lodash deep cloning, then mutation: ~457.67ms/op on 66ops
ES2015 destructuring: ~9.51ms/op on 1000ops
immutable (w/o conversion to plain JS objects): ~24.63ms/op on 1000ops
immutable (w/ conversion to plain JS objects): ~1041.59ms/op on 29ops
immer (proxy implementation w/o autofreeze): ~139.86ms/op on 215ops
immer (ES5 implementation w/o autofreeze): ~435.78ms/op on 69ops
immutad●t: ~43.82ms/op on 688ops

Node 9.2.0 :

lodash deep cloning, then mutation: ~7386.50ms/op on 10ops
ES2015 destructuring: ~9.25ms/op on 1000ops
immutable (w/o conversion to plain JS objects): ~22.95ms/op on 1000ops
immutable (w/ conversion to plain JS objects): ~978.23ms/op on 31ops
immer (proxy implementation w/o autofreeze): ~1693.50ms/op on 18ops
immer (ES5 implementation w/o autofreeze): ~424.45ms/op on 71ops
immutad●t: ~44.93ms/op on 670ops

Results are staying close except for lodash which nearly stops (might be the gc taking time to clean up clones), and immer with proxies which node 9 definitely messes up...

Maybe try V8 options listed with node --v8-options ? Tried --trace-gc but not sure how to read this...

Feel free to reuse it if you need.

[mweststrate]

@frinyvonnick optimizations are in progress in the feature/optimization branch, my results are now as follows, completely removing the drop on node 9, and improving the general performance 4-6 times

8.4:

    ✓ just mutate (3ms)
    ✓ deepclone, then mutate (417ms)
    ✓ handcrafted reducer (20ms)
    ✓ immutableJS (71ms)
    ✓ immer (proxy) - with autofreeze (118ms)
    ✓ immer (proxy) - without autofreeze (39ms)
    ✓ immer (es5) - with autofreeze (421ms)
    ✓ immer (es5) - without autofreeze (401ms)

9.3:

    ✓ just mutate (4ms)
    ✓ deepclone, then mutate (252ms)
    ✓ handcrafted reducer (17ms)
    ✓ immutableJS (59ms)
    ✓ immer (proxy) - with autofreeze (107ms)
    ✓ immer (proxy) - without autofreeze (34ms)
    ✓ immer (es5) - with autofreeze (380ms)
    ✓ immer (es5) - without autofreeze (362ms)

[frinyvonnick]

Nice job 👍

[mweststrate]

Should be solved by 0.3.0