Cypress: Cypress UI locks up during cypress run in CI (recognizable in video recording, the UI is less responsive and/or completely frozen)

[jennifer-shehane]

Current behavior:

As per the following comments, the UI appears to freeze when reviewing video taken during cypress run. There is only activity recorded for the first few seconds.

• https://github.com/cypress-io/cypress/issues/2370#issuecomment-491351832
• https://github.com/cypress-io/cypress/issues/2370#issuecomment-493160496

Example videos https://52855-7910045-gh.circle-artifacts.com/0/root/project/cypress/videos/main-page.spec.js.mp4, https://53812-7910045-gh.circle-artifacts.com/0/root/project/cypress/videos/main-page.spec.js.mp4. But I think that these "frozen" videos are a separate problem. Original problem described in this issue is about incomplete videos.

Desired behavior:

The UI/video recording should not freeze and capture the test run like usual when run during cypress open

Steps to reproduce: (app code and test code)

??? I'd love if anyone could respond with the Versions below and ideally steps to reproduce this. 🙏

Versions

Cypress: <= 3.3.1
Browser: Electron 61
OS: Linux, Debian 8-1 + ??
CI: Jenkins + ??

[jennifer-shehane]

I would appreciate if everyone with this issue would update to our latest version of Cypress - we made several improvements to the performance of the Test Runner that may have helped with the freezing. Please post any results here.

cc @platan @pisti2010 @maryhipp

[cooleiwhistles]

Hi, I am on latest version of Cypress (3.4.0) and running tests through Jenkins CI on a Linux machine. I am seeing issues with the test recording freezing, and then jumping ahead toward the end. When testing locally the recordings seem fine. Once we added the dockerfile and split the tests in parallel, we began seeing the issue.

[maximkoshelenko]

Have the same issue. Cypress 3.4.0, Gitlab Pipeline runner, cypress/browsers:node12.6.0-chrome75 image, npx cypress run script. Image record only a few seconds and freeze after.

[platan]

After updating from 3.3.2 to 3.4.0 videos are OK (https://61762-7910045-gh.circle-artifacts.com/0/root/project/cypress/videos/main-page.spec.js.mp4). Videos were also OK with 3.3.2 (https://61665-7910045-gh.circle-artifacts.com/0/root/project/cypress/videos/main-page.spec.js.mp4). We are using cypress/base:8 docker image and CircleCI (https://github.com/badges/shields/blob/master/.circleci/config.yml#L307).

[platan]

In another example from cypress 3.4.0 video is frozen: https://61786-7910045-gh.circle-artifacts.com/0/root/project/cypress/videos/main-page.spec.js.mp4.

[bahmutov]

I have done some experiments, trying to recreate a frozen video

• using https://github.com/bahmutov/demo-docker-cypress-included I have added --memory and --cpu restrictions to the Docker container as explained in https://docs.docker.com/config/containers/resource_constraints/
• I can monitor CPU and memory usage via docker stats <container id>
• when restricting memory, the test runner either runs or not, and the ffmpeg either works no gets killed when it hits the memory limit. Still, it runs reliably under 370 megabytes.
  NOTE: the 370MB worked for this _small_ web site, that did not load much during tests. If the web application were larger, and the test were longer, I am sure the browser and Cypress would need more memory to finish and not crash.
• when restricting CPU share, the entire test becomes slower, the video becomes longer. Finally, for the following parameter on my Mac Book Pro I could see a video with freezes. There was part of the middle missing and entire second half

--cpus=0.2

Note that I have enabled full memory, so the CPU is the deciding factor

Other debugging notes:

• you can see the amount of memory each second by using CYPRESS_MEMORY=1 environment variable

So I think the real culprit is the CPU resources available on CI during test runs. Remember - it is not just Cypress test runner, there might be other services running too: the server, the bundler, databases, etc.

Recommendation:

try running CI tests on a more powerful box. If the box is already powerful, try running fewer concurrent jobs.

[WillsB3]

Recommendation: try running CI tests on a more powerful box. If the box is already powerful, try running fewer concurrent jobs.

As far as I'm aware we don't have any control over the assigned hardware resources with GitLab CI, so I don't think I can personally test that suggestion unfortunately.

Additionally, with our CI pipeline, the E2E stage is the only job running by that point in the pipeline, so reducing concurrent jobs also isn't something I can try.

How do I tell Cypress to record videos when running locally? I tried the --record flag but it wants a key for uploading the video to the Dashboard however I just want the recording stored locally?

[bahmutov]

• are you using shared GitLab CI runners? Because they are probably underpowered, I would suggest running your own private runners that you do have control over, or even trying CircleCI or SemaphoreCI - we never had problems with videos there, and we are not even using anything but the default boxes.
• Cypress test runner records video locally by default when you do cypress run. Unless someone has set video: false via configuration options https://on.cypress.io/configuration

[WillsB3]

are you using shared GitLab CI runners? Because they are probably underpowered, I would suggest running your own private runners that you do have control over, or even trying CircleCI or SemaphoreCI - we never had problems with videos there, and we are not even using anything but the default boxes.

Yeah - we're using the shared runners - moving to a different CI platform is not an option for us.

Cypress test runner records video locally by default when you do cypress run. Unless someone has set video: false via configuration options https://on.cypress.io/configuration

Ah - ok, thanks. I'll try running some tests locally too

[dotexe0]

I'm experiencing a similar issue on "cypress": "3.4.1". We use docker-compose in Google Cloud Build to orchestrate e2e testing with some micro-services. I can trigger the e2e tests locally with docker-compose up and the videos get uploaded to the dashboard with no missing frames, but when I pass it through the CI/CD pipeline, the videos in the dashboard freeze after about 3 seconds. The tests always pass and take about the right amount of time to complete even when frozen.

Cypress Dockerfile:

FROM cypress/base:10
WORKDIR /app

COPY package.json .
COPY package-lock.json .

ENV CI=1
RUN npm ci
RUN npx cypress verify

Cypress package.json:

{
  "name": "e2e",
  "version": "1.0.0",
  "description": "Cypress tests",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "license": "ISC",
  "private": true,
  "devDependencies": {
    "cypress": "3.4.1",
    "faker": "^4.1.0"
  }
}

docker-compose:

version: '3'
services:
  web:
    build:
      context: .
      dockerfile: ./Dockerfile 
    ports:
      - 3000
    env_file:
      - .env.development

###--- //  ....microservices.... // ---###

  e2e:
    image: cypress
    build: ./e2e
    container_name: cypress
    environment:
      - CYPRESS_baseUrl=http://web:3000
    command: npx cypress run --group integrations --env ENV=production --record --key #### --parallel --ci-build-id ${BUILD_ID}
    volumes:
      - ./e2e/cypress:/app/cypress
      - ./e2e/cypress.json:/app/cypress.json

cypress.json:

{
  "projectId": "####",
  "baseUrl": "http://localhost:3000",
  "chromeWebSecurity": false,
  "hosts": {
    "https://dev-####.oktapreview.com/api/v1/authn": "127.0.0.1"
  },
  ...
}

[bahmutov]

@dotexe0, unfortunately, this is a problem with underpowered CI container that prevents the video from being captured without gaps or freezing. Try switching to a more powerful container if possible or even to a different CI provider. For example we never have frozen video on CircleCI machines, yet I see freezing video quite often on GitLab CI.

[bahmutov]

We will close this issue since this is due to the underpowered CPU available to Cypress in the CI container. I will think how to detect the frozen video happening and give a warning, have opened #5061

[archfz]

I am also experiencing this. v3.8.1; browser electron 78. Isn't there any way for the ui to wait for the recording ?

[tnedev]

I'm having the same issue in CircleCI with large machine of theirs.