Cypress: CI jobs freezing in Gitlab but running perfectly on local Mac

Created on 26 Mar 2020  ·  26Comments  ·  Source: cypress-io/cypress

Current behavior:

I have some Cypress scripts that run to completion every time on my local Mac environment in both headed and headless Chrome. Cypress is working perfectly as expected locally. The problem I am having is adding this to our CI process in Gitlab. I have quoted the .gitlab-ci.yml file below.

The process just hangs once it reaches the testing phase and times out after the 60 minute limit. We are getting the desired behaviour of the 3 tests scripts running in parallel on separate instances but all of them hang and timeout. Even as a single thread on once instance we see the same hanging behaviour.

I suspect there are some missing dependencies for running Cypress on this OS? The build itself runs without errors (with quiet flags removed) and the processes freezes as shown in the screenshot.

Screenshot 2020-03-26 at 14 35 30

Desired behavior:

Tests should run to completion in our Gitlab CI step and should complete without an error code to indicate that the push is acceptable.

Test code to reproduce

image: node:10.4.1-stretch

stages:
  - test

before_script:
  - echo "deb http://mirrors.kernel.org/ubuntu/ xenial main" | tee -a /etc/apt/sources.list
  - echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
  - apt-get -qq update && apt-get install -y --no-install-recommends apt-utils > /dev/null
  - apt-get -qq install -y --allow-unauthenticated libpng12-0 > /dev/null
  - apt-get -qq install -y python-pip python-dev > /dev/null
  - pip install -q --upgrade awscli
  - npm install --quiet

cache:
  untracked: true
  key: global
  paths:
    - node_modules
    - .npm

.cypress_build_script: &cypress_build_script |
  apt-get -qq install -y libgtk2.0-0 libnotify4 libgtk-3-0 libnotify-dev libgconf2-4 libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb > /dev/null
  wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb 
  apt -qq install -y --allow-unauthenticated ./google-chrome-stable_current_amd64.deb > /dev/null
  npm install -g --quiet wait-on
  npx cypress cache path
  npx cypress cache list
  PROXY=true npm run setup:development
  npm run build:dll
  npm start & wait-on http://localhost:4000

cypress_test_mdlu:
  stage: test
  script:  
    - *cypress_build_script
    - npm run cypress:run -- --spec \"cypress/integration/Smoke Tests/MDLU/*.spec.js\" --headless --browser chrome

cypress_test_dlu:
  stage: test
  script:  
    - *cypress_build_script
    - npm run cypress:run -- --spec \"cypress/integration/Smoke Tests/DLU/*.spec.js\" --headless --browser chrome

cypress_test_slu:
  stage: test
  script:  
    - *cypress_build_script
    - npm run cypress:run -- --spec \"cypress/integration/Smoke Tests/SLU/*.spec.js\" --headless --browser chrome

Versions

Cypress: 4.2.0
Browser: Chrome 80 (headless)
OS: Debian 9

gitlab
Source
picture abryant710
👍1

Most helpful comment

Thanks @0xIslamTaha

I can confirm that this particular article resolved the problem:

https://stackoverflow.com/questions/57334452/can-i-increase-shared-memory-after-launching-a-docker-session

I add these two lines to the start of my build script:

mount -o remount,size=2G /dev/shm
df -h /dev/shm

This issue can be closed.

picture abryant710 on 24 Aug 2020
👍2

All 26 comments

  • Does it timeout on the same test each time?
  • Does it timeout reliably every time?
  • Could you run Cypress in debug mode mode and print the entire set of logs here?
jennifer-shehane picture jennifer-shehane on 27 Mar 2020

Hi Jennifer,

  • Does it timeout on the same test each time?

Yes it does seem to stall on the first or second test in the first spec file for each of the three concurrently running instances.

  • Does it timeout reliably every time?

Yes always the same behavior.

  • Could you run Cypress in debug mode mode and print the entire set of logs here?

I have added this into each spec file.

Cypress.on('fail', (error, runnable) => {
  debugger;

  // we now have access to the err instance
  // and the mocha runnable this failed on

  throw error; // throw error to have test still fail
});

Here is the log of the tests running to completion on my local environment. You can see that they all pass fine:

cypress_headless_20200327105409.log

Here is an excerpt from one of the CI instances:

cypress-ci-log.log

Here are some of the config params in cypress.json:

  "baseUrl": "http://localhost:4000",
  "watchForFileChanges": false,
  "viewportWidth": 1000,
  "viewportHeight": 660,
  "defaultCommandTimeout": 20000,
  "requestTimeout": 20000,
  "responseTimeout": 20000,
  "ignoreTestFiles": "*_template.spec.js",
  "video": false,
  "numTestsKeptInMemory": 0,

Thanks

abryant710 picture abryant710 on 27 Mar 2020

Hi @abryant710, have you considered using official cypress docker images and run your tests against them ? I don't mean to dismiss your issue entirely but from past experiences, making chrome/chromium work inside docker containers can be a bit of a pain and using pre-built images saved me lots of unwanted debugging.

I have no significant experience with Gitlab CI, only Jenkins and Github actions but looking at the YAML file you supplied it seems that you can run your workflow using a base docker image right ? What do you think ?

Here is the link to the repo for Cypress docker images: https://github.com/cypress-io/cypress-docker-images.
Note: the default values for the images is a few versions back, but if you specify it explicitely you can have the latest one (4.2.0 at the moment)

Niceplace picture Niceplace on 27 Mar 2020

@Niceplace thanks for the advice. I have tried using cypress/browsers:node13.6.0-chrome80-ff72 but the testing script doesn't move much further on and it still grinds to a halt. 

stages:
  - test

before_script:
  - wget http://se.archive.ubuntu.com/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
  - dpkg -i libpng12-0_1.2.54-1ubuntu1_amd64.deb
  - apt-get -qq install -y python-pip python-dev > /dev/null
  - pip install -q --upgrade awscli

cache:
  untracked: true
  key: global
  paths:
    - node_modules
    - .npm

.cypress_build_script: &cypress_build_script |
  npm install -g n wait-on
  n 10.4.1
  node -v
  npm install --quiet
  npx cypress cache path
  npx cypress cache list
  PROXY=true npm run setup:development
  npm run build:dll
  npm start & wait-on http://localhost:4000

cypress_test_mdlu:
  stage: test
  image: cypress/browsers:node13.6.0-chrome80-ff72
  script:  
    - *cypress_build_script
    - npm run cypress:smoke_tests:headless:mdlu

cypress_test_dlu:
  stage: test
  image: cypress/browsers:node13.6.0-chrome80-ff72
  script:  
    - *cypress_build_script
    - npm run cypress:smoke_tests:headless:dlu

cypress_test_slu:
  stage: test
  image: cypress/browsers:node13.6.0-chrome80-ff72
  script:  
    - *cypress_build_script
    - npm run cypress:smoke_tests:headless:slu
abryant710 picture abryant710 on 27 Mar 2020

@jennifer-shehane This looks like a common issue a lot of people are facing.
https://github.com/cypress-io/cypress/issues/6449

vivek12345 picture vivek12345 on 30 Mar 2020

Here is some system info right before the Cypress tests run and then hang:

$ free --giga -t
              total        used        free      shared  buff/cache   available
Mem:             65           3           4           0          57          61
Swap:            33           0          33
Total:           99           4          38
$ df -ha
Filesystem      Size  Used Avail Use% Mounted on
none            427G  388G   17G  96% /
proc               0     0     0    - /proc
tmpfs            32G     0   32G   0% /dev
devpts             0     0     0    - /dev/pts
sysfs              0     0     0    - /sys
tmpfs            32G     0   32G   0% /sys/fs/cgroup
cgroup             0     0     0    - /sys/fs/cgroup/cpuset
cgroup             0     0     0    - /sys/fs/cgroup/cpu
cgroup             0     0     0    - /sys/fs/cgroup/cpuacct
cgroup             0     0     0    - /sys/fs/cgroup/blkio
cgroup             0     0     0    - /sys/fs/cgroup/memory
cgroup             0     0     0    - /sys/fs/cgroup/devices
cgroup             0     0     0    - /sys/fs/cgroup/freezer
cgroup             0     0     0    - /sys/fs/cgroup/net_cls
cgroup             0     0     0    - /sys/fs/cgroup/perf_event
cgroup             0     0     0    - /sys/fs/cgroup/net_prio
cgroup             0     0     0    - /sys/fs/cgroup/hugetlb
systemd            0     0     0    - /sys/fs/cgroup/systemd
mqueue             0     0     0    - /dev/mqueue
/dev/md2        427G  388G   17G  96% /cache
/dev/md2        427G  388G   17G  96% /builds/first
/dev/md2        427G  388G   17G  96% /etc/resolv.conf
/dev/md2        427G  388G   17G  96% /etc/hostname
/dev/md2        427G  388G   17G  96% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/md2        427G  388G   17G  96% /root/.docker/config.json
tmpfs           6.3G  1.1M  6.3G   1% /run/docker.sock
abryant710 picture abryant710 on 31 Mar 2020

I'm seeing a similar issue but using bitbucket pipelines. Interestingly enough it also happens locally on my mac though. For us it's the 17th test file that hangs both in CI and locally. I'll paste the tail of the log below - it stops after printing the context+describe for that spec and hangs there for a couple hours until the CI build times out. Locally it never finishes. Cypress version 4.1.0, it ends up using a little over 1.89 GB of ram for the cypress renderer before the CI build fails... The weirdest part is it had no issues until just yesterday - no changes to the build process or tests and yet it stalled out starting yesterday morning.

(Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        9                                                                                │
  │ Passing:      9                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     1 minute, 2 seconds                                                              │
  │ Spec Ran:     integration/association-table.spec.js                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  integration/bulk-action-enrollment.spec.js                                    (17 of 86)


  dashboard actions
    bulk people enroll
apmenard1993 picture apmenard1993 on 2 Apr 2020

Same here

edarchimbaud picture edarchimbaud on 7 Apr 2020
👍1

I'm having the same. Issue. Tests run fine for ~15 minutes in GitLab CI, but once it gets to a certain test suite, it freezes, despite having resources on the runner. I tried skipping the suite it would originally get stuck on, but then it just freezes on the next suite.

This is 100% reliably reproducible for us. It never makes it through our whole set of tests. We were not having this issue before when we were on cypress v3.8.3

We are using the cypress/browsers:node12.14.0-chrome79-ff71 image as our base image for docker.

vanakema picture vanakema on 17 Apr 2020

I also faced the same issue after upgrading Cypress from 3.8.3 to 4.4.0. CI job freezes after 20 min for both Chrome and Firefox. Setting --disable-dev-shm-usage for Chrome solved the issue for me.

plugins\index.js

 module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.family === 'chrome') {
      console.log('Adding --disable-dev-shm-usage...')
      launchOptions.args.push('--disable-dev-shm-usage')
    }

    return launchOptions
  })
}
michalziolek picture michalziolek on 24 Apr 2020

We have --disable-dev-shm-usage enabled but still the same problem.

on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.name === "chrome") {
      launchOptions.args.push("--disable-dev-shm-usage");
      return launchOptions;
    }
    return launchOptions;
  });
vivek12345 picture vivek12345 on 24 Apr 2020

Thank you for Cypress, what a great automation tool. Same problem where Cypress 4.5.0 runs fine on my Mac but freezes in a Gitlab pipeline. Things I tried:

  1. 👎🏼: tried above fix with Chrome --disable-dev-shm-usage.
  2. 👎🏼: removing first hanging test, it just hangs on the next one.
  3. 👎🏼: downgrading to Cypress 4.4.1
mrjjwright picture mrjjwright on 29 Apr 2020

So big user error, I feel bad about the above comment but it was totally a poorly written test. Thanks again for Cypress!

mrjjwright picture mrjjwright on 1 May 2020

So big user error, I feel bad about the above comment but it was totally a poorly written test. Thanks again for Cypress!

What was it about your test that was causing the issue?

vanakema picture vanakema on 1 May 2020

Yes we want to understand your issue, maybe we have the same!

sebfie picture sebfie on 4 May 2020

Sorry about the delay guys. In my case the test modifications were completely specific to my situation but the root issue was that there was an error that was only happening in our Gitlab pipeline and not locally. Also when it ran inside Docker/Gitlab, it was failing silently in the browser and nothing was logged in Cypress. Once I ran the tests in a docker image locally it showed up. So I guess the one thing to look out for is completely silent failures. That is unlikely to affect most of you all but perhaps beginners. And then to test your docker container locally as well as in your pipeline. Sorry I can't help more.

mrjjwright picture mrjjwright on 4 May 2020

we faced the same nasty bug and we managed to solve it by adding shm_size: 2GB to the cypress service inside the docker composer. If u are using docker run ... instead of docker-composer then u can do something like docker run --shm_size=2GB .....

0xIslamTaha picture 0xIslamTaha on 12 May 2020
👍1

@0xIslamTaha
It works to me.

Before add --shm-size=2GB, sometimes run cypress will freeze by docker run between case1 finish or case 2 finish(there are 3 big cases in a spec).

After add --shm-size=2GB, it can't reproduce.

JFRabbit picture JFRabbit on 16 May 2020
🎉1

Just an update on this one.

I am now using the following and still seeing it freeze without Cypress timing out and failing in the expected fashion.

  • Cypress: 4.8.0
  • Browser: Chrome 81 (headless)

I am using this Docker image now, which should contain all of the required dependencies (presumably):

cypress/browsers:node13.8.0-chrome81-ff75

My script is:

script: - touch ~/.bash_profile - wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash - . ~/.nvm/nvm.sh - nvm install 10.4.1 - npm i -g --quiet wait-on - npm ci - npx cypress cache path - npx cypress cache list - npm run cy:verify - npm run cy:info - npm start & wait-on http://localhost:4000 - npm run cy:smoke_tests:headless:all

I don't see why this is freezing. Like I said, it works fine locally. Is there something I am missing when using Docker and gitlab-runner to execute the CI process?

abryant710 picture abryant710 on 19 Jun 2020

I was using Cypress 4.8.0,and faced the issue. Now upgraded to 4.11.0, still facing the same. Is it still an open issue??

Pallabee94 picture Pallabee94 on 7 Aug 2020

@Pallabee94 @abryant710
did u try this solution?

0xIslamTaha picture 0xIslamTaha on 7 Aug 2020

@0xIslamTaha we are not giving any docker commands explicitly, but we are using gitlab-ci.yml , mentioning all the configurations in Gitlab runner. Can the command be included there?

Pallabee94 picture Pallabee94 on 7 Aug 2020

Still exists in latest cypress 4.12.1.

We are even try with docker image from cypress repo: cypress/browsers:node12.18.0-chrome83-ff77

wojtek1150 picture wojtek1150 on 10 Aug 2020

I can confirm that this issue is specific to version 4.8.0. I am trying different versions to see where it was fixed...

abryant710 picture abryant710 on 21 Aug 2020

I can confirm that this issue is specific to version 4.8.0. I am trying different versions to see where it was fixed...

I believe there is no cypress version with a fix for this ticket because actually it happened because of there is no enough memory for chrome.

So to fix this, u have to overwrite the default shared memory btn the docker and the host

0xIslamTaha picture 0xIslamTaha on 21 Aug 2020

Thanks @0xIslamTaha

I can confirm that this particular article resolved the problem:

https://stackoverflow.com/questions/57334452/can-i-increase-shared-memory-after-launching-a-docker-session

I add these two lines to the start of my build script:

mount -o remount,size=2G /dev/shm
df -h /dev/shm

This issue can be closed.

abryant710 picture abryant710 on 24 Aug 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

weskor picture weskor  ·  3Comments
Francismb picture Francismb  ·  3Comments
jennifer-shehane picture jennifer-shehane  ·  3Comments
rbung picture rbung  ·  3Comments
tahayk picture tahayk  ·  3Comments