Jest: memory leak on deep_cyclic_copy jest 23

[sibelius]

🐛 Bug Report

The heap usage keeps increasing when using mongoose

To Reproduce

https://github.com/entria/graphql-dataloader-boilerplate has a simple config that has this issue, but it only crashes when there are a lot of tests (as the memory keeps increasing).

this happens with and without --runInBand option

Expected behavior

the memory heap decreases after a jest

Link to repl or repo (highly encouraged)

https://github.com/entria/graphql-dataloader-boilerplate

Run npx envinfo --preset jest

Paste the results here:

System:
    OS: macOS High Sierra 10.13.6
    CPU: x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  Binaries:
    Node: 10.7.0 - ~/.nvm/versions/node/v10.7.0/bin/node
    Yarn: 1.9.2 - /usr/local/bin/yarn
    npm: 6.1.0 - ~/.nvm/versions/node/v10.7.0/bin/npm
  npmPackages:
    jest: 23.4.1 => 23.4.1 

Below is a print of some tests in a private repo:

And here the crash when the memory heap is not enough anymore:

<--- Last few GCs --->

[73921:0x102801000]    94783 ms: Scavenge 1355.7 (1423.3) -> 1354.8 (1423.8) MB, 5.3 / 0.5 ms  (average mu = 0.114, current mu = 0.054) allocation failure
[73921:0x102801000]    94794 ms: Scavenge 1355.7 (1423.8) -> 1354.8 (1424.3) MB, 5.8 / 0.5 ms  (average mu = 0.114, current mu = 0.054) allocation failure
[73921:0x102801000]    94803 ms: Scavenge 1355.8 (1424.3) -> 1354.9 (1424.8) MB, 4.7 / 0.5 ms  (average mu = 0.114, current mu = 0.054) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1d2dc829e6c9 <JSObject>
    0: builtin exit frame: getOwnPropertyDescriptors(this=0x1d2dc8284519 <JSFunction Object (sfi = 0x1d2df3d902d9)>,0x1d2dab7cbad9 <Object map = 0x1d2d689b9831>,0x1d2dc8284519 <JSFunction Object (sfi = 0x1d2df3d902d9)>)

    1: deepCyclicCopyObject(aka deepCyclicCopyObject) [0x1d2d88467351] [/Users/sibelius/Dev/entria/app/app-server/node_modules/jest-util/build/deep_cyclic_copy.js:~54]...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1000389cc node::Abort() [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 2: 0x100038ba8 node::FatalTryCatch::~FatalTryCatch() [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 3: 0x1001a9d5a v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 4: 0x100578772 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 5: 0x100577729 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 6: 0x1005753b8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 7: 0x1005818fc v8::internal::Heap::AllocateRawWithRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 8: 0x10055003d v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::internal::PretenureFlag, v8::internal::Handle<v8::internal::AllocationSite>) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
 9: 0x100555626 v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::PretenureFlag, v8::internal::Handle<v8::internal::AllocationSite>) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
10: 0x10075f7dc v8::internal::PropertyDescriptor::ToObject(v8::internal::Isolate*) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
11: 0x10026ced6 v8::internal::Builtin_Impl_ObjectGetOwnPropertyDescriptors(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/sibelius/.nvm/versions/node/v10.7.0/bin/node]
12: 0x39704e60a1dd
13: 0x39704f2ca3ec
error Command failed with signal "SIGABRT".

I think this is related to https://github.com/facebook/jest/issues/6738

[rickhanlonii]

@sibelius thanks for reporting - why do you think deepCyclicCopy is leaking? I see it in the stack trace but isn't that just where it runs out of memory, not necessarily what is leaking?

[sibelius]

it makes sense, maybe the leaking is in another part of the code

I've ran in this repo as well https://github.com/graphql-compose/graphql-compose-mongoose

and the heap is increasing there as well

[rickhanlonii]

Can you run with --detectLeaks?

[sibelius]

28 tests failed with leaks

on https://github.com/graphql-compose/graphql-compose-mongoose

is there any way to get more information about it?

[sibelius]

I've found this about memory leak on mongoose https://github.com/Automattic/mongoose/issues/2874#issuecomment-388588452:

mongoose.connections.forEach(connection => {
  const modelNames = Object.keys(connection.models)

  modelNames.forEach(modelName => {
    delete connection.models[modelName]
  })

  const collectionNames = Object.keys(connection.collections)
  collectionNames.forEach(collectionName => {
    delete connection.collections[collectionName]
  })
})

const modelSchemaNames = Object.keys(mongoose.modelSchemas)
modelSchemaNames.forEach(modelSchemaName => {
  delete mongoose.modelSchemas[modelSchemaName]
})

I've added this to afterAll jest hook, but it did not worked well

[sibelius]

it could be related to this comment https://github.com/facebook/jest/issues/1456#issuecomment-395686090

[jgcmarins]

I am facing the same problema and what I am seeing here is the same message:

    1: deepCyclicCopyObject(aka deepCyclicCopyObject) [0x12794b44b231] [/path/to/app/app-server/node_modules/jest-util/build/deep_cyclic_copy.js:~54]...

[sibelius]

we added this https://github.com/facebook/jest/issues/6787#issuecomment-409297750 to afterAll hook, it solved most part of memory leak from mongoose

and we solved the other leak part mocking graphql-redis-subscriptions

[vkarpov15]

Mongoose maintainer here. This is not a memory leak, its global state that you need to clean up in your tests. In general, using global state in tests is an anti-pattern. If you want to create a new model for each test, use mongoose.createConnection() to create a new connection for each test.