Cypress: Not able to detect ciBuildId for Jenkins or run in --parallel

[drewbrend]

This issue stemmed out of a conversion with @bahmutov on gitter

Current behavior:

Running my specs in Jenkins I get this error:

12:10:32 cypress-tests  | You passed the --group or --parallel flag but we could not automatically determine or generate a ciBuildId.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | The --parallel flag you passed was: true
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | In order to use either of these features a ciBuildId must be determined.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | The ciBuildId is automatically detected if you are running Cypress in any of the these CI providers:
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | - appveyor
12:10:32 cypress-tests  | - bamboo
12:10:32 cypress-tests  | - buildkite
12:10:32 cypress-tests  | - circle
12:10:32 cypress-tests  | - codeship
12:10:32 cypress-tests  | - drone
12:10:32 cypress-tests  | - gitlab
12:10:32 cypress-tests  | - jenkins
12:10:32 cypress-tests  | - semaphore
12:10:32 cypress-tests  | - shippable
12:10:32 cypress-tests  | - travis
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | Because the ciBuildId could not be auto-detected you must pass the --ci-build-id flag manually.
12:10:32 cypress-tests  | 
12:10:32 cypress-tests  | https://on.cypress.io/indeterminate-ci-build-id

If I add --ci-build-id ${BUILD_NUMBER} to the run command my test will run and report to the dashboard, however the values for Brach/Author/Commit/Message/CI are not populated in the dashboard, they show "unknown", and I am only running on one executor - not in parallel.

I am running inside a docker container, I think this may be part of the issues I am having and I may have to explicitly expose the environment variables that Cypress uses to my container. However, I do not know what environment variables Cypress needs to figure this all out.

Desired behavior:

Cypress should detect Jenkins and all environment variables automatically

Steps to reproduce:

Run Cypress specs in a docker container in Jenkins with the --parallel flag specified. I am on v1.642.4

Versions

Cypress 3.1.0
Jenkins 1.642.4
Electron 59

[JPHamlett1993]

I have the exact same issue.

I'm thinking the issue is more in Docker

When I do

$(npm bin)/cypress run --parallel --record --key ci-key --ci-build-id test2

And have some docker images running

docker run -itd --env BUILD_NO=test2 harbor.bluestembrands.com/fulton/cypressio:3.1.0 tail -f /dev/null

I get no parallelization occuring

[drewbrend]

For the comment about the Branch/Author/Commit/Message/CI not being populated in the dashboard, this looks like the reason for author and message: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L241

[drewbrend]

Some more info.

From https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L20 it looks like if I pass the JENKINS_URL env var into my docker container it should trigger isJekins to be true. I have done this and removed --ci-build-id ${BUILD_NUMBER} from my run command. It seems like it attempts to work with Jenkins but something goes wrong. Here is the error that's spit out

14:34:15 cypress-tests  | We encountered an unexpected error talking to our servers.
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | There is likely something wrong with the request.
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | The --parallel flag you passed was: true
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | The server's response was:
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | StatusCodeError: 422
14:34:15 cypress-tests  | 
14:34:15 cypress-tests  | {
14:34:15 cypress-tests  |   "code": "INDETERMINATE_CI_BUILD_ID",
14:34:15 cypress-tests  |   "message": "CI Build ID could not be generated and was not specified."
14:34:15 cypress-tests  | }

[jennifer-shehane]

@drewbrend Yeah - this is the error when we cannot get the CI Build id from the CI provider essentially. It looks like we are looking for these env vars from Jenkins to determine this ourselves: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee#L129

[drewbrend]

Ahhh thank you! I passed those into my container via the environment: section of my docker-compose file and it detects Jenkins and runs without me specifying the ci build id. The CI field is populated in the dashboard and the link works.

Some docs around what env vars Cypress uses would be very useful so we know which ones to provide to a docker container.

I still don't have Author, Commit, or Message populated in the dashboard, so that needs work.

And still need to work with someone to get parallelization working.

[drewbrend]

I spent a good deal of time on this but have parallelization working across multiple nodes in Jenkins, running in docker containers. Here's what I did for anyone who stumbles on this in the future. This assumes you already have cypress running successfully in a Jenkins job.

Sign up for the cypress dashboard, this is required for parallelization. Follow cypress docs on how to get your project reporting to the dashboard, it's easy, I won't explain it.

In Jenkins I created a cypress dispatcher job, which uses the Jenkins build-flow plugin to determine how many jobs to kick off in parallel and kicks off a cypress runner job for each of those inside of a parallel() block. The dispatcher waits for all of these to complete, then returns pass if they all pass, or a fail if any of those fail.

The runner job does basically what you already have set up to run serially, except it also passes these flags to the cypress run command: --parallel --ci-build-id ${CI_BUILD_ID}. The CI build ID is how cypress knows that all the jobs are supposed to run together, then does a bunch of magic behind the scenes to distribute specs and group results.

The dispatcher job passes the same CI_BUILD_ID parameter to the runner jobs, this needs to be a key that is unique to each run of the dispatcher, so I used the env variable BUILD_NUMBER.

[drewbrend]

So now that the parallelization is resolved, the only things outstanding for this ticket are the git details not being reported to the dashboard, which will require changes in Cypress:

-Author
-Commit
-Message

[bahmutov]

@drewbrend thanks so much for this! We are putting a fix for commit information here https://github.com/cypress-io/cypress/pull/2345 so that hopefully fills the missing fields

[bahmutov]

also @drewbrend can you list all environment variables from a few CI boxes that run in parallel (hiding or removing sensitive keys) - maybe there is something in common that Jenkins already puts for you?

[drewbrend]

Sure thing, these is the dump from the printenv command in the cypress runner job, with a liberal amount of stuff changed. Some of these will be params passed into the job, some are jenkins env variables. Any questions just ask.

Box#1:

10:36:53 CYPRESS_browser=electron
10:36:53 PARALLEL_LETTER=E
10:36:53 BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/62/
10:36:53 HUDSON_SERVER_COOKIE=****
10:36:53 SHELL=/bin/bash
10:36:53 JENKINS_API_KEY=****
10:36:53 SSH_CLIENT=23.23.182.73 42456 22
10:36:53 BUILD_TAG=jenkins-cypress-runner-62
10:36:53 CI_BUILD_ID=40
10:36:53 GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:53 ROOT_BUILD_CAUSE=MANUALTRIGGER
10:36:53 WORKSPACE=/var/lib/jenkins/workspace/cypress-runner
10:36:53 JOB_URL=http://ci.OurCompany.com/job/cypress-runner/
10:36:53 branch=QA-311
10:36:53 RUN_CHANGES_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/62/display/redirect?page=changes
10:36:53 USER=jenkins
10:36:53 CYPRESS_baseUrl=https://OurCompany-blue.OurDomain.net
10:36:53 GITHUB_STATUS_OAUTH=****
10:36:53 BUILD_CAUSE_UPSTREAMTRIGGER=true
10:36:53 SSH_AUTH_SOCK=/tmp/jenkins4683669874741720002.jnr
10:36:53 OurCompany_BUILD_TOOLS_PATH=/opt/OurCompany/build-tools
10:36:53 GIT_AUTHOR_NAME=FirstName LastName
10:36:53 CYPRESS_API=prod
10:36:53 browser=electron
10:36:53 spec=/**/*.js
10:36:53 GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
10:36:53 JENKINS_HOME=/var/lib/jenkins
10:36:53 MAIL=/var/mail/jenkins
10:36:53 PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
10:36:53 RUN_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/62/display/redirect
10:36:53 GITHUB_OAUTH_TOKEN=****
10:36:53 CYPRESS_spec=/**/*.js
10:36:53 PWD=/var/lib/jenkins/workspace/cypress-runner
10:36:53 HUDSON_URL=http://ci.OurCompany.com/
10:36:53 LANG=en_US.UTF-8
10:36:53 JOB_NAME=cypress-runner
10:36:53 BUILD_DISPLAY_NAME=#62 - origin/QA-311 - E
10:36:53 BUILD_CAUSE=UPSTREAMTRIGGER
10:36:53 BUILD_ID=62
10:36:53 JENKINS_URL=http://ci.OurCompany.com/
10:36:53 OurCompany_DOCKER_REPO=****.dkr.ecr.us-east-1.amazonaws.com
10:36:53 GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:53 HOME=/var/lib/jenkins
10:36:53 SHLVL=2
10:36:53 PG_CYPRESS_BRANCH=QA-311
10:36:53 GIT_BRANCH=origin/QA-311
10:36:53 EXECUTOR_NUMBER=0
10:36:53 JENKINS_SERVER_COOKIE=**********
10:36:53 [email protected]:OurCompany/OurProduct
10:36:53 NODE_LABELS=scaled-dev scaled-slave-2
10:36:53 LOGNAME=jenkins
10:36:53 SSH_CONNECTION=23.23.182.73 42456 172.31.64.52 22
10:36:53 HUDSON_HOME=/var/lib/jenkins
10:36:53 NODE_NAME=scaled-slave-2
10:36:53 BUILD_NUMBER=62
10:36:53 JOB_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/display/redirect
10:36:53 ROOT_BUILD_CAUSE_MANUALTRIGGER=true
10:36:53 HUDSON_COOKIE=****
10:36:53 [email protected]
10:36:53 NEW_RELIC_INSIGHTS_KEY=****

Box#2:

10:36:48 CYPRESS_browser=electron
10:36:48 PARALLEL_LETTER=A
10:36:48 BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/61/
10:36:48 HUDSON_SERVER_COOKIE=****
10:36:48 SHELL=/bin/bash
10:36:48 JENKINS_API_KEY=****
10:36:48 SSH_CLIENT=23.23.182.73 37475 22
10:36:48 BUILD_TAG=jenkins-cypress-runner-61
10:36:48 CI_BUILD_ID=40
10:36:48 GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:48 ROOT_BUILD_CAUSE=MANUALTRIGGER
10:36:48 WORKSPACE=/var/lib/jenkins/workspace/cypress-runner
10:36:48 JOB_URL=http://ci.OurCompany.com/job/cypress-runner/
10:36:48 branch=QA-311
10:36:48 RUN_CHANGES_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/61/display/redirect?page=changes
10:36:48 USER=jenkins
10:36:48 CYPRESS_baseUrl=https://OurCompany-blue.OurDomain.net
10:36:48 GITHUB_STATUS_OAUTH=****
10:36:48 BUILD_CAUSE_UPSTREAMTRIGGER=true
10:36:48 SSH_AUTH_SOCK=/tmp/jenkins3675280086874796853.jnr
10:36:48 OurCompany_BUILD_TOOLS_PATH=/opt/OurCompany/build-tools
10:36:48 GIT_AUTHOR_NAME=FirstName LastName
10:36:48 CYPRESS_API=prod
10:36:48 browser=electron
10:36:48 spec=/**/*.js
10:36:48 GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
10:36:48 JENKINS_HOME=/var/lib/jenkins
10:36:48 MAIL=/var/mail/jenkins
10:36:48 PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
10:36:48 RUN_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/61/display/redirect
10:36:48 GITHUB_OAUTH_TOKEN=****
10:36:48 CYPRESS_spec=/**/*.js
10:36:48 PWD=/var/lib/jenkins/workspace/cypress-runner
10:36:48 HUDSON_URL=http://ci.OurCompany.com/
10:36:48 LANG=en_US.UTF-8
10:36:48 JOB_NAME=cypress-runner
10:36:48 BUILD_DISPLAY_NAME=#61 - origin/QA-311 - A
10:36:48 BUILD_CAUSE=UPSTREAMTRIGGER
10:36:48 BUILD_ID=61
10:36:48 JENKINS_URL=http://ci.OurCompany.com/
10:36:48 OurCompany_DOCKER_REPO=****.dkr.ecr.us-east-1.amazonaws.com
10:36:48 GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
10:36:48 HOME=/var/lib/jenkins
10:36:48 SHLVL=2
10:36:48 PG_CYPRESS_BRANCH=QA-311
10:36:48 GIT_BRANCH=origin/QA-311
10:36:48 EXECUTOR_NUMBER=0
10:36:48 JENKINS_SERVER_COOKIE=****
10:36:48 [email protected]:OurCompany/OurProduct
10:36:48 NODE_LABELS=scaled-dev scaled-slave-8
10:36:48 LOGNAME=jenkins
10:36:48 SSH_CONNECTION=23.23.182.73 37475 172.31.65.176 22
10:36:48 HUDSON_HOME=/var/lib/jenkins
10:36:48 NODE_NAME=scaled-slave-8
10:36:48 BUILD_NUMBER=61
10:36:48 JOB_DISPLAY_URL=http://ci.OurCompany.com/job/cypress-runner/display/redirect
10:36:48 ROOT_BUILD_CAUSE_MANUALTRIGGER=true
10:36:48 HUDSON_COOKIE=****
10:36:48 [email protected]
10:36:48 NEW_RELIC_INSIGHTS_KEY=****

[bahmutov]

@drewbrend to get the git info, do you think this would work

• authorEmail: GIT_AUTHOR_EMAIL
• authorName: GIT_AUTHOR_NAME
• sha: GIT_COMMIT
• branch: GIT_BRANCH

The only one missing is message

from these vars

[email protected]
GIT_AUTHOR_NAME=FirstName LastName
GIT_BRANCH=origin/QA-311
GIT_COMMIT=b34641f414894587b27e528dba8202c3cfb4f9b2
GIT_PREVIOUS_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
GIT_PREVIOUS_SUCCESSFUL_COMMIT=a5590be2335010bd9e9a45955641b001c20f7889
[email protected]:OurCompany/OurProduct

[bahmutov]

I see how CI_BUILD_ID stays the same 40 while BUILD_ID=61 and BUILD_ID=62 is different, and we can link to the individual builds using BUILD_URL=http://ci.OurCompany.com/job/cypress-runner/62/ but not tothe common CI build id 40.

[drewbrend]

@bahmutov The GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL env vars are created during the run of my cypress-runner job to try to get around the issues I was having when I created this ticket. GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL did not get picked up by cypress like I thought they would, so they are currently not doing anything. CI_BUILD_ID is passed from the cypress-dispatcher job to the cypress-runner job and then passed into the cypress run command via the --ci-build-id flag. So all of that is manual.

If you can do something similar to what I have below in cypress before you pass the infomation to the dashboard, I think that could work for the Git info. I imagine you could do the same to get message as well. GIT_COMMIT and GIT_BRANCH exist already, so those are an easy win.

This shell is in my cypress-runner job to create those variables

export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an' $GIT_COMMIT)
export GIT_AUTHOR_EMAIL=$(git --no-pager show -s --format='%ae' $GIT_COMMIT)

if [ -z "$CI_BUILD_ID" ]; then
  export CI_BUILD_ID=$BUILD_NUMBER
fi

[bahmutov]

hmm, we do use https://github.com/cypress-io/commit-info to fetch Git things https://github.com/cypress-io/commit-info/blob/418002428b653ed75ccd82bff6966529c9009e24/src/git-api.js#L9 but we do NOT pass GIT_COMMIT, maybe that's why it fails?!!!

would git --no-pager show -s --format='%an' not work inside the job?

[drewbrend]

If you're not passing GIT_COMMIT, that sounds like that could be the issue.

I am running git --no-pager show -s --format='%an' inside the cypress-runner job. sorry, I will update my previous message - it said it was in the cypress-dispatcher job

[bahmutov]

could you add just git --no-pager show -s --format='%an' command inside each job just to see if it works without commit id, if this is the issue, we can update commit-info to be explicit if GIT_COMMIT is present

[drewbrend]

@bahmutov I changed export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an' $GIT_COMMIT) to export GIT_AUTHOR_NAME=$(git --no-pager show -s --format='%an') and it still works, the GIT_AUTHOR_NAME env var is set.

[bahmutov]

Hmm, ok @drewbrend if you can have another env variable DEBUG=commit-info to see output from each individual git command from https://github.com/cypress-io/commit-info module. Then we could see what was going on when it tries to fetch commit information.

[drewbrend]

@bahmutov I set that, and the printenv command shows DEBUG=commit-info in the list of env vars, however I don't see any additional debug output. Just the usual cypress output about the tests that have ran.

[bahmutov]

@drewbrend ughh, we are hiding DEBUG output from stderr, that's unfortunate. I will need to think of the way to debug this

[drewbrend]

@bahmutov I'm happy to help, let me know what you need. I'll have time this evening to run a couple tests if you think of anything to try.

[bahmutov]

Great @drewbrend so one thing you might do is to look at the code in https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.coffee and enable debug logs for it to see the commit information in https://github.com/cypress-io/cypress/blob/3b804cd09ccea3100c9783def5ada6327e3e7562/packages/server/lib/util/ci_provider.coffee#L310 which I believe should be printed if DEBUG=cypress:server is set

commitDefaults = (existingInfo) ->
  commitParamsObj = commitParams() or {}
  debug("commit params object")
  debug(commitParamsObj)

  ## based on the existingInfo properties
  ## merge in the commitParams if null or undefined
  ## defaulting back to null if all fails
  _.transform existingInfo, (memo, value, key) ->
    memo[key] = _.defaultTo(value ? commitParamsObj[key], null)

and here is where it is called https://github.com/cypress-io/cypress/blob/8f0a31aaa32c4ce5fe6bdf096c6f20224cb438dd/packages/server/lib/modes/record.coffee#L266

which is logged using DEBUG=cypress:server:record. So we can enable both logs using DEBUG=cypress:server,cypress:server:record

if we can see what each part does we could determine where things go wrong ...

[drewbrend]

@bahmutov I have DEBUG=cypress:server,cypress:server:record set and am not seeing any additional output to the console in Jenkins.

[bahmutov]

Hmm, we might have added more debug log output after 3.1.0 to actually debug this problem better

[drewbrend]

Okay no problem. It isn't a blocker for me, but would be nice to have. We have parallel working and with our setup we can pretty easily figure out the author and commit from the branch name if we need to.

[bahmutov]

Great, yeah, if you got it working - that is nice, and if this is not a blocker, than perfect, and next release hopefully will have all the information either fixed or able to debug. I have installed local Jenkins in Docker setup https://hub.docker.com/r/jenkinsci/blueocean/ --- and could not get Node plugin to actually add Node there :( https://medium.com/wearetheledger/jenkins-your-butler-for-continuous-integration-blue-ocean-docker-image-solution-to-node-not-ebf9ebfbfa78 Plus we really want a public Jenkins instance that can show to everyone the build results and pipeline for our cypress-example-kitchensink, so Jenkins CI demo is still work in progress.

[jennifer-shehane]

Closing as resolved. Please comment if you are still having this issue and we will consider reopening.