Cypress: Docker crashes when renderer process eats up too much memory

[brian-mann]

Related to #348 and #349.

When running headlessly on very long and memory intense applications we are seeing renderer crashes with Docker.

[brian-mann]

This is actually not indicative of a memory leak inside of Cypress (nor your application). This has to do with the way that browsers work under the hood.

Luckily, we have a found a simple one line fix for this problem - simply add the flag --ipc=host.

This option is documented here.

The Slack app and Atom has also been documented to crash here, here, and here for the exact same reasons.

If you are using docker run

docker run --ipc=host

If you are using docker compose

version: '2'
services:
  foobarbazservice:
    ipc: host ## this line right here

In the future we are working on a more permanent fix such as described in #349 - either to automatically recover from these crashes, or mostly prevent them up from by nuking the renderer process in between spec files.

[jheijkoop]

I seem to be getting this problem too, because Chrome is running out of shared memory (/dev/shm). By default docker starts images with a 64M /dev/shm (try running df -h in you instance). To change this you can supply docker with an extra argument: docker run --shm-size 512M my-image. Because we are working with mesos/marathon I had to do this via the "docker": { "parameters": [{"key": "shm-size", "value": "512M"}], ...} (https://mesosphere.github.io/marathon/docs/native-docker.html) in the json configuration when creating the app/instances.

[einomi]

I had the same issue with Codeship CI. Just needed to change config in my codeship-services.yml file, according to documentation https://documentation.codeship.com/pro/continuous-integration/browser-testing/#chrome-crashing

[jennifer-shehane]

I wonder if this is related to our other issue with a Codeship CI run failing.https://github.com/cypress-io/cypress/issues/328 Unfortunately, we don't have a pro account, and the codeship-services.yml is only available on pro.

[egucciar]

any advice on how to solve for this in CircleCI? I do not have a services section in my config.yml, only a jobs.

[mitar]

I think that since this issue has been made there is now a better fix for the problem by asking Chrome not to use /dev/shm. I opened #3633 for more details about this.

[wralitera]

hello @brian-mann . How do you set up the --ipc=host into the gitlab-ci.yml?

[ccorcos]

Anyone know how to add --ipc=host to CircleCI? It looks like they call docker run somewhere out of our control...

[mitar]

You cannot. And this is why this workaoround does not really work.

[ccorcos]

@maximilianschmitt seems to think you can: https://gitter.im/cypress-io/cypress/archives/2018/10/17

[ccorcos]

Actually, a build just finished and it crashed pretty quickly

[ccorcos]

Looks like there's plenty of shared memory.

df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              30G  8.0K   30G   1% /dev/shm

[alk831]

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

[rwralitera]

I will test this image: cypress/browsers:node11.13.0-chrome73 right now. Thanks @alk831

[rwralitera]

I tested and it failed:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 2: 0x7fa824226887 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 3: 0x7fa823d95a57 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 4: 0x7fa823d959d5 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 5: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 6: 
<--- Last few GCs --->

[425:0x2659cbff8000]  1853805 ms: Mark-sweep 2057.7 (2176.7) -> 2057.7 (2154.2) MB, 2096.2 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2096 ms) last resort 
[425:0x2659cbff8000]  1856036 ms: Mark-sweep 2057.7 (2154.2) -> 2057.7 (2154.2) MB, 2230.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x162eb08ad681 <JSObject>
    1: bindContainer(aka bindContainer) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~22229] [pc=0x2ef0ae593de3](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <NodeObject map = 0x3516afdfd7d1>,containerFlags=45)
    2: bind(aka bind) [/builds/wynd-qa/BO/node_modules/typescript/lib/typescript.js:~23556] [pc=0x2ef0ae55cdac](this=0x2ae892502311 <undefined>,node=0x7235b368b9 <No...

v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 7: 0x7fa823b407a1 [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 8: 0x7fa823b4347b [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
 9: 0x7fa823b431bc [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
10: 0x7fa823b47b5f [/root/.cache/Cypress/3.2.0/Cypress/libnode.so]
11: 0x2ef0add043fd

Timed out waiting for the browser to connect. Retrying...

The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.

[alk831]

@bentrole It crashed only once for me with this image and now it works properly. I would try to split large tests into smaller ones and run without recording until there is no fix for this.

[rwralitera]

@alk831 I already splited my tests because of that but this bug is there since 2016, If there is no fixes until now, I think that they will not correct that! ;-(

[ccorcos]

@brian-mann can we please reopen this issue? If we cannot scale out our testing library, I don't think we can use this service :/

[alk831]

@bentrole one more thing that you can try is to run test in parallel so memory usage is smaller, however
as @brian-mann mentioned this is not strictly related with cypress and maybe your CI runner is just not capable of running chromium. If you are using gitlab CI then you could try to create your own runner.

[wralitera]

My tests are structured like that: 1 spec file = 1 scenario. So when I run all tests using the --spec option, each spec file should run one by one and should not make an out of memory.

Running in parallele works but I have not my own runner and if I use too many runner and the test take time, I ll block other developers.

I will ask to our admin system (of gitlab runner) to try to custom the runner and activate the --ipc=host suggested by @brian-mann

[Ameobea]

Bumping to cypress/browsers:node11.13.0-chrome73 as my base Docker image fixed this for me as well! 🎊

[r3dDoX]

For people struggling to set ipc=host on their respective environment it should help to mount a volume to /dev/shm to allow shared memory for POSIX objects.
In Kubernetes/OpenShift for example I mounted an "EmptyDir" volume to /dev/shm and everything is working smoothly now.

[jbinto]

This seems to have gotten way worse in 3.3.0. We've been running Cypress on CircleCI for almost 2 years, and these crashes have become way more frequent recently.

[jbinto]

We're seeing Cypress crashes multiple times daily now in CircleCI after a couple of years of solid reliability. None of the workarounds here have helped:

• Cannot set --ipc=host on Circle
• Using cypress/browsers:node11.13.0-chrome73 also had crashes
• --disable-dev-shm-usage is only available since Chrome 65, and Electron is only 61 (we use Electron on CI for video recording)
  
  
  
  • Edit: Pursuing this route futher - passing --disable-dev-shm-usage using the args method seems to hang Cypress on CircleCI, but not locally. Eventually we get the automation client disconnected. No tests ever execute.
  
  
  • Edit 2: This was due to not using the correct Docker image - cypress/included does not include browsers.
  
  

We've never seen a "sad face" crash since about 1 month ago. I'll look back at our CI logs and see when it started.

[jbinto]

It looks like crashes started 8 days ago since we upgraded to 3.3.1. I don't see any crashes associated with 3.3.0 but it was only out for 5 days in between.

[idanRiski]

Any news with this issue?
I've already tried all your suggestions to use the flages of "ipc=host", "cpus=2" or to increase the shm/dev memory.. also to downgrade the cypress to 3.2.0 and to add the flages below to the cypress.json file: "numTestsKeptInMemory": 0, "restartBrowserBetweenSpecFiles": true, "videos: false".. but none of them has solved this issue.
I got this error message (crashed becasue a sad face.." in every execution in time it using "cy.visit".
this issue happens only by docker and didn't recreat by locally execution .

please your assistance, thanks.

[jbinto]

We have switched to --browser chrome using the cypress/browsers:node10.2.1-chrome74 Docker image, and we disable /dev/shm usage as so:

// cypress/plugins/index.js

module.exports = (on, config) => {
  // ref: https://docs.cypress.io/api/plugins/browser-launch-api.html#Usage
  on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })
}

And crashes have stopped completely. I suspect something changed in between Electron 59 and Electron 61 that makes it more prone to crashes, but for anyone hitting this on CircleCI this is the only actual fix (after weeks of tinkering).

edit: The obvious drawback is we lose video recording, until #1767 is fixed. The tradeoff is unfortunately worth it here - CI being reliable is more important than being easy to debug.

[Ameobea]

@jbinto this looks really promising; I don't have access to the K8s cluster on which our build node runs, so I can't tweak anything on that end. I'll give this a shot and see if it solves our crashes as well!

[amkoehler]

@jbinto I had about perfect timing with opening this issue and seeing your solution. This is working well for us so far using the circleci/node:8-browsers image

[yuanyuanlimaggie]

Anyone has a good solution for Gitlab? I updated to cypress/browsers:node11.13.0-chrome73 in Gitlab. It still crashed. I am using Cypress 3.3.2.

[uvesten]

Please fix this. Using Cypress 3.4.1, have started seeing crashes more and more often.

[jennifer-shehane]

Going to reopen this since people are still having issues with crashes in Docker.

Our plan of action now is to investigate passing in the --disable-dev-shm-usage automatically. Related to https://github.com/cypress-io/cypress/issues/3633

Workaround Today

Follow these instructions for passing the --disable-dev-shm-usage flag: https://github.com/cypress-io/cypress/issues/350#issuecomment-503231128

cc @flotwig

[srinu-kodi]

Hi @brian-mann @jennifer-shehane
Adding the --ipc=host for docker run is also not helping out and specs are failing continuously.

We have integrated the tests in CI and specs are failing. Is there any solution in near future as it is giving hard times now...

Thanks

[you1anna]

This is now blocking us using Cypress in CI, have tried adding --disable-dev-shm-usage as @jennifer-shehane suggested but we're seeing the 'Chromium renderer process crashed' message each time in GoCD, after Cypress has run only two tests.

[srinu-kodi]

This is now blocking us using Cypress in CI, have tried adding --disable-dev-shm-usage as @jennifer-shehane suggested but we're seeing the 'Chromium renderer process crashed' message each time in GoCD, after Cypress has run only two tests.

Hi @you1anna I also faced this problem, today finally I could fix this by adding shm_size: '2gb'
in my docker-compose.yaml file and it is working consistently.
Basically this flag will allocate the 2gb memory to the docker container and such that there will be no memory related issue...

Please add this and it should be fine. BTW it doesn't have any link with GoCD :)

Let me know if you face any problem or need help.

[you1anna]

Thanks @srinu-kodi. It makes sense to use more shm, but in our tests we are using a separate container with following volume share:
volumes: - "/dev/shm:/dev/shm"
which means that all of shared memory is used by the container. Currently we are using agents with 8GB ram, so our next step is to try with 16GB. Outside of that I don't know what else we can do.

[mvandebunt]

have this problem since using 3.4.1
running dockers in a wercker pipeline

[cmcnicholas]

recently upgraded to 3.4.1 and we are also exhibiting the same issues here pretty much every test run now (because we have parallel tests) results in 2-3 machines failing whilst the other 4 pass.

[fulvio-m]

For Gitlab users the fix requires editing of the runner configuration by adding a new volume to the volumes configuration of your docker runner:

[[runners]]
  ...
  executor = "docker"
  ...
  [runners.docker]
   ...
    volumes = ["/dev/shm:/dev/shm", ...other volumes...]
  ...
  [runners.cache]

Basically this instructs the docker instance to mount the underlying /dev/shm volume.
After editing runner needs to be restarted with gitlab-runner restart.
Credits for the fix to https://lcx.wien/blog/cypress-gitlab-ci/

[mohansgithub]

Any idea how to use --ipc==host on the dynamic docker based jenkins agents ( which will be deployed on Kubernetes) ? in below configuration we need specify the cypress image and we need to use --ipc==host ?

[Bsmalhi]

@mohansgithub use Environment Variables add "ipc" as key and "host" as value that will do it for your docker container. Let me know if you figured it out.

[karthikkota]

Thank you @fulvio-m.

[JimmyKuruvilla]

--ipc=host works locally for me, but as an env var in the jenkins kubernetes plugin it did not work. However mounting it as a volume did work:

[hannahhaken]

We're running Cypress 3.8.1 in Electron 78 in Docker across multiple apps and have been seeing this issue for a couple of weeks in one particular app. The rest of the apps were fine. We tried the suggested IPC host workaround but this didn't help....

What _has_ fixed this issue is removing multiple uses of describe(). So it appears this particular test suite had multiple instances of describe() nested inside a single describe(), all in a single TypeScript file. I've now replaced these nested describe() with it(). The memory leak issue has subsided and this test suite now finishes without any problems.

---

Update: For the remaining apps which had memory leaks, it appears running the tests in Chrome rather than Electron does the trick.

[9odzilla]

@hannahhaken My tests are already written with top level describe() and it() within them. I still see this issue using electron.

[bigbitbus]

We have the same issue with Google Cloudbuild - its not possible to specify ipc=host within the Google Cloudbuild specification file.

[zeel-swiggy]

This is not happening for 3.6.0, but happened for 3.7.0 and 3.8.0

[must-git-good]

Joining the various voices that are pointing out that this error seems to be cropping up for us for the first time over the last few weeks. Didn't happen on all tests, but did consistently happen on some (that shouldn't have actual memory issues). Losing nested describe blocks helped, but errors remained (all this despite running ipc=host in Docker, etc.). It seems like swapping to Chrome for now may help, which means Electron and Cypress may have some interesting optimization/interactions going on under the hood.

No other input, will update if I find out more, but want to make sure this gets looked at, as it's been a significant frustration for us.

[lsdkjflasjflkdasj]

Same thing happening here.

[tommueller]

We are also constantly running into this issue and by now this is creating a lot of frustration here.
We are running the Cypress tests in a Drone-CI docker setup. All the tests run perfectly fine on my local machine. It's especially confusing too me, as our application is really not that memory hungry (about 80mb max memory usage says Chrome Task Manager) and tests are very short too short (mostly between 5-15 loc).

We already tried these recommendations:

• increased the size of dev/shm
• upgraded to latest versions of drone, drone-agent and cypress
• decreased numTestsKeptInMemory setting
  without any success.

I have not yet tried these options, as they are not possible with our current setup or seem to be too unstable ...

• add --ipc=host on the docker container, which is currently not possible because drone does not support it for now, there is also an open ticket for this: https://discourse.drone.io/t/does-drone-support-ipc-host-option/1049
• try using the --disable-dev-shm-usage-flag on Chrome https://github.com/cypress-io/cypress/issues/5336

Any updates on this would be a great help for us!

[davidzambrana]

We fixed this problem after implementing a few recommendations we saw here:

• Updated to Cypress 3.8.0
• Moved to this image cypress/browsers:node10.16.0-chrome77
• Started using -headless -b chrome in the run command
• Updated our plugins/index file with

on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })

The crashes stopped in all the test collections we have (~15)

[tommueller]

I can confirm that the changes @davidzambrana proposed also worked for us so far.
We now run into the issue described here https://github.com/cypress-io/cypress/issues/1297 but this is probably more an issue of how we wrote the tests.
I will try to fix the tests and keep an eye on this. Thanks @davidzambrana for now!

[mulholio]

To anyone wondering what the --disable-dev-shm-usage flag does, this page is a useful explainer https://developers.google.com/web/tools/puppeteer/troubleshooting#tips

[shlima]

https://github.com/cypress-io/cypress/issues/350#issuecomment-574072211 can confirm the solution, but I've changed cypress/browsers:node10.16.0-chrome77 image to cypress/included:4.0.2 as it currently includes cypress and chrome.

My specs started to run smoothly on Gitlab CI with cypress in docker-in-docker

[adinizs]

We resolved this problem using the image cypress/browsers:node10.16.0-chrome on dockerfile and running chrome headless.

[radDunin]

Hi,
i can also confirmed @davidzambrana solution, what is important is that the image should be this one: cypress/browsers:node10.16.0-chrome77 it works even in headed chrome. For now thats doesn't works for me on chrome 80 and chrome 69

[otalandim]

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

For me it doesnot works

[otalandim]

We fixed this problem after implementing a few recommendations we saw here:

• Updated to Cypress 3.8.0
• Moved to this image cypress/browsers:node10.16.0-chrome77
• Started using -headless -b chrome in the run command
• Updated our plugins/index file with

on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })

The crashes stopped in all the test collections we have (~15)

For me it doesnot works

Another problem happened...can you help me please

  CypressError: cy.visit() failed trying to load:
 We attempted to make an http request to this URL but the request failed without a response.
 We received this error at the network level:
   > Error: ESOCKETTIMEDOUT
 Common situations why this would fail:
   - you don't have internet access
   - you forgot to run / boot your web server
   - your web server isn't accessible
   - you have weird network configuration settings on your computer
 The stack trace for this error is:
 Error: ESOCKETTIMEDOUT
     at ClientRequest.<anonymous> (/root/.cache/Cypress/4.2.0/Cypress/resources/app/packages/server/node_modules/request/request.js:816:19)
     at Object.onceWrapper (events.js:299:28)
     at ClientRequest.emit (events.js:210:5)
     at TLSSocket.emitRequestTimeout (_http_client.js:690:9)
     at Object.onceWrapper (events.js:299:28)
     at TLSSocket.emit (events.js:210:5)
     at TLSSocket.Socket._onTimeout (net.js:468:8)
     at listOnTimeout (internal/timers.js:531:17)
     at processTimers (internal/timers.js:475:7)
 Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite:

[przemuh]

Switching to Chrome with --disable-dev-shm-usage as @davidzambrana suggested - works for me!

Thanks a lot! 🎉

@otalandim Please remember that since Cypress 4.0.0 there is a change in passing arguments to the browser (https://docs.cypress.io/guides/references/migration-guide.html#Plugin-Event-before-browser-launch)

I use GitlabCI. Here are my changes:

GitlabCI config

-  image: cypress/base:12.16.2
+  image: cypress/browsers:node12.16.2-chrome81-ff75
````

cypress/plugins/index.js
```js
on("before:browser:launch", (browser = {}, launchOptions) => {
  if (browser.name === "chrome") {
    launchOptions.args.push("--disable-dev-shm-usage");
    return launchOptions;
  }
}

run command:

cypress run --browser chrome --headless

[ttomaszewski]

Any idea when this will get resolved for Electron?

[digitalkaoz]

for anyone getting here and want to know how to fix it on gitlab-ci kubernetes executor:

$ kubectl edit cm gitlab-runner-gitlab-runner

config part to add right before # Start the runner in entrypoint
see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2578#note_186055981

    cat >> /home/gitlab-runner/.gitlab-runner/config.toml << EOF
          [[runners.kubernetes.volumes.empty_dir]]
              name = "shm"
              mount_path = "/dev/shm"
              medium = "Memory"
    EOF

[sabarishnarain]

This is still a problem in Azure CI with electron (cypress 4.11) when using the base docker images. The workaround is to use chrome by following @przemuh instructions.

[keul]

For posterity: I've this same issue in Gitlab CI.

Unluckily every gitlab-ci.yml workaround above require the change of the base docker image, while I'm not (and I can't) using any of them: I am manually installing cypress on a Centos (is an internal gitlab installation using a private runner server).

In my case this gitlab comment to issue solved the problem.

[CoryDanielson]

I saw a reduction of 30-50% of memory usage by enabling window.gc() and calling it globally in an afterEach callback.

You can enabled window.gc with browser launch options, or electron command line switches.

// For Chrome, add this flag in plugins
launchOptions.args.push('--js-flags=--expose-gc');

```console

For Electon, run cypress with this environment variable set

ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc

plugins/index.js
```js
module.exports = (on, config) => {
    on('before:browser:launch', (browser, launchOptions) => { 
        if (browser.name === 'chrome') {
            launchOptions.args.push('--js-flags=--expose-gc');
        }
        return launchOptions;
    });
};

support/index.js

afterEach(() => {
    cy.window().then(win => {
        if (typeof win.gc === 'function') {
            // calling this more often seems to trigger major GC event more reliably
            win.gc(); win.gc(); win.gc(); win.gc(); win.gc();
        }
    });
});

cypress open peak memory usage - ~50% reduction in peak memory

# --js-flags=--expose-gc is set in plugins/index.js
cypress open

Max memory in MB (recorded by watching Chrome memory tab)
| no win.gc | win.gc once | calling win.gc 5 times |
| --- | --- | --- |
| 535mb | 299 | 267 |
| 585mb | 406 | 369 |
| 598mb | 372 | 278 |
| 570mb | 445 | 298 |
| --- | averages | --- |
| 572mb | 380 (34% less) | 303 (47% less) |

cypress run peak memory usage - ~30% reduction in peak memory for chrome headless

# --js-flags=--expose-gc is set in plugins/index.js
cypress run --headless -b chrome

(recorded by watching Activity Monitor while tests were running)
| Chrome headless, no win.gc()
Chrome Rendered Memory | Chrome headless calling win.gc() 5 times
Chrome Rendered Memory |
| --- | --- |
| 890 | 580 (~30% less) |
| 850 | 560 (~30% less) |

cypress run peak memory usage = ~50% reduction in peak memory for electron

ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc cypress run

(recorded by watching Activity Monitor while tests were running)
| Electron, no win.gc()
Cypress Helper Renderer Memory | Electron, calling win.gc() 5 times |
| --- | --- |
| 850mb | 410 (~50% less) |
| 750mb | 450 (~50% less) |

Calling window.gc() multiple times was more effective (based on my imperfect tests), I suspect because it possibly caused chrome to trigger a major GC event rather than a minor one.

---

We switched our CI docker run script to use --ipc=host, --disable-dev-shm-usage, chrome headless, and use window.gc() and it's running really well now with no out-of-memory crashes

[phillyc]

For those of you who are trying to solve this issue in GitLab, on the free tier, I found a simple solution.

I was trying to create some smoke tests for my team's application, and noticed that the GitLab CICD runner would typically get through the first 2-3 tests of each test file before crashing with the above out of memory symptoms.

Condense all of the actions and test assertions into 1-2 it('can do foo') test statements.

There appears to be a pretty big memory overhead in the setup/teardown of each of these cases. I was able to take around 60 test cases in my locally passing integration test stack and condense them down into 10 test.js files with 1 test case each.

In summary, try putting all your tests into one test case.