Jest: Fresh jenkins installation crash when running jest.

[kopax]

Bug request

Bug

Current behavior

Using the official docker image of jenkins/ v2.46.2.

I have a webpack project that use [email protected] that contain more than 500 hundreds of enzyme test mainly.

Reproduction

1. Run jenkins (MUST have port 8080 free or change the host port)

   docker run -p 8080:8080 -v /var/jenkins_home jenkins
   

2. Print the secret key to be inserted on install

   docker exec -i $(docker ps | grep jenkins | awk -F ' ' '{print $1}') cat /var/jenkins_home/secret.key
   

3. Past the key on http://localhost:8080 and complete the jenkins installation

4. Load from a CVS a project that use jest-cli 19.0.2 and contains around 500 enzyme (mount/shallow) unit test.

5. Configure a new project on Jenkins and import project from CVS

6. Create a build terminal step with npm install and npm test

With a fresh installation and one project, jenkins crash during the build project.

This is my hardware configuration, I don't think it is the issue here

CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64
node version 7.9.0

Expected behavior

jenkins should perform the jest command to the end.

[thymikee]

There's no such version as 19.2.1 for jest-cli.
What does it mean that "jenkins crash during the build project". How is build connected to running tests? What's the error message?
Also, I think you forgot to provide a repro which you're describing. Would you mind providing that? :)

[kopax]

I have updated with the correct version number.

What does it mean that "jenkins crash during the build project".

Jenkins fail silently without throwing any log, even with the highest log level.

How is build connected to running tests?

In fact, it's not a build, it's a npm test job.

What's the error message?

There is no error message at all, I can see hole in job numbers, because it is planting. The end of the task just look like :

Also, I think you forgot to provide a repro which you're describing. Would you mind providing that? :)

We are using jest for a library and it's documentation. Both aren't release on GitHub yet but we will. If you wan't to have a reproduction, I can prepare one for your team. I rather keep the code private until the official release is pronounced. Is it ok for you?

[thymikee]

No need for me to see your private code, just a minimal case so we can repro, although I don't have experience with Jenkins so it would be problematic to setup whole thing on my machine.

What you can try (just to test it) is to run npm test -- --forceExit or you can split your tests and run them under separate commands, maybe there's something in your code which causes problems.

[kopax]

No need for me to see your private code, just a minimal case so we can repro,

I am pretty sure I can do that. It's not private, we are going to release it, it is just released because of details like this issue. (you might like it and it's well documented:)

although I don't have experience with Jenkins so it would be problematic to setup whole thing on my machine.

Do you have experience with Docker ?
It's better if you do, otherwise you can just execute the command I give you in order it won't be much.
For the jenkins part, you don't have to click everywhere, just follow the instructions it look as follow:

Start a container/create a projet/step pull/step npm install/step npm test.

I could prepare a tutorial with every step described and screenshots. Count approx 20 min to setup. You will also need a host running on Debian 8 and Docker 1.13.1. I will detailled everything properly if you say yes.

What you can try (just to test it) is to run npm test -- --forceExit or you can split your tests and run them under separate commands, maybe there's something in your code which causes problems.

I have also tried with --runInBand and it's not changing much. It might be because of the code, but we have 99% code coverage and we were 3 working on it without any issues. Also, this issue happen with all my jest project that use enzyme and jest for testing. I have tried jest v18 and v19, tried a bunch of version of node. I thing I might need some help to understand what's really happening.

[thymikee]

So maybe that's something with the version of enzyme and e.g. enzyme-to-json you use. Hard to tell without an example.

[kopax]

I am not using enzyme-to-json in my project. I have upgraded enzyme from 2.6.0 to 2.8.2 and it still fails. I am preparing a reproduction now.

Edit I increased my swap partition after seeing jenkins+jest together drastically increase it's usage.
I thought this would correct the issue but it keep crashing even with lot's of resources available.

[kopax]

I apologize for the tutorial, and would like to thanks in advance the person who will achieve it.

I am __not confident__ that test will crash in your environment if you are not running your container on mesos/marathon.

In case you just wan't to see the crash , I have recorded it with my smartphone, it's not a professional footage but you can see it :

https://drive.google.com/file/d/0B6drRJ3_BeQSMVh2RWE4cnc2NUU/view

Prerequesite

In order to be exactly with the same configuration, you will need Host 32H offer at OVH.

CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64

1. Install Debian Jessie. (can be skipped but potentially you won't get the error)
2. Install Mesos/Marathon. (can be skipped but potentially you won't get the error)
3. Install docker-engine 1.13.1.

During my test, I use the following configuration with Debian Jessie 8.0:

Jenkins installation

1. Download and run jenkins official image. From a separate bash: (you must have port 8080 available)

   docker run -p 8080:8080 jenkins
   

2. You will see in your terminal password. Copy it.

   Jenkins initial setup is required. An admin user has been created and a password generated.
   Please use the following password to proceed to installation:
   
   39ae474fe5544c00881167cb8764cb14
   
   This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
   

Or this will print the content of /var/jenkins_home/secrets/initialAdminPassword:

    docker exec -ti $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') cat /var/jenkins_home/secrets/initialAdminPassword

1. Now open chrome http://localhost:8080/ and past the code

1. Click on Select plugin to install

1. Go to manage jenkins > manage plugin and complete the installation by checking for installation the following plugins :

• NodeJS

• Git plugin

Validate the change

1. Wait for the download and installation of the jenkins plugins :

1. Configure the administrator account as follow

1. You will arrive on the jenkins dashboard

1. Click on "Manage Jenkins"

1. Now click on "Global Tool Configuration"

1. Configure a NodeJS version 7.9.0

1. Click on "New Item"

1. Name it and choose "Freestyle project"

1. Under "Source Code Management", select the radio "git" and add the following repository url : https://github.com/kopax/jest-crashing-jenkins

1. Under "Build Environment" check "provide Node & npm bin/ folder to path" and select the version "7.9.0" of NodeJS.

1. Under "Build", click on "Execute shell"

1. Enter "npm install" and "npm test", then click "Save"

1. Now click on Build now

1. There will be a new job added. You can click on it and then click on "Console Output"

1. While running the build, you can use htop to view your memory usage (sudo apt-get update && sudo apt-get install htop -y)

1. Also you should note the CONTAINER_ID so you will see when it crash

   docker ps | grep jenkins
   abf10c67f072        jenkins                 "/bin/tini -- /usr/lo"   About an hour ago   Up About an hour    0.0.0.0:8080->8080/tcp, 50000/tcp   reverent_saha
   

I wasn't able to reproduce the crash it except in my production environment.

This is how my test fail in production :

Notice the Jobs id 48, because when Jenkins will come back, it will miss that ID and jenkins will do like nothing happened before.

_Tips 1: If you wish to have a more verbose log level when doing logs of your container you can do the following withing the containers, I have already checked and there is nothing interesting into:_

 docker exec -it $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') cat > /var/jenkins_home/log.properties <<EOF
handlers=java.util.logging.ConsoleHandler
jenkins.level=FINEST
java.util.logging.ConsoleHandler.level=FINEST
EOF

And then restart your container with :

 docker stop $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}')
 docker start $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties"

_Tips 2: If you want to upgrade jenkins to the latest version (see the warning), you can activate a hidden button "Upgrade jenkins automatically button". For that, you need to give the good permissions to /usr/share/jenkins within the container, this will make the button appear in the "Jenkins Configuration" page and you will then be able to upgrade Jenkins_

    docker exec -it --user root $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}') chown jenkins:jenkins -R /usr/share/jenkins

Note that I am running my Jenkins container on Mesos/Marathon and I only have the issue here. As attachement this is the marathon deployement json I use for deploying my jenkins container:

{
  "id": "developers/jenkins",
  "cmd": null,
  "mem": 4096,
  "cpus": 2,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "jenkins:latest",
      "forcePullImage": true,
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 0,
          "protocol": "tcp"
        }
      ]
    },
    "volumes": [
      {
        "containerPath": "/var/jenkins_home",
        "hostPath": "/srv/developers/jenkins/var/jenkins_home",
        "mode": "RW"
      },
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      },
      {
        "containerPath": "/usr/bin/docker",
        "hostPath": "/usr/bin/docker",
        "mode": "RW"
      },
      {
        "containerPath": "/lib/x86_64-linux-gnu/libapparmor.so.1",
        "hostPath": "/usr/lib/x86_64-linux-gnu/libapparmor.so.1",
        "mode": "RW"
      }
    ]
  },
  "healthChecks": [
    {
      "portIndex": 0,
      "protocol": "TCP",
      "gracePeriodSeconds": 30,
      "intervalSeconds": 10,
      "timeoutSeconds": 30,
      "maxConsecutiveFailures": 3
    }
  ],
  "upgradeStrategy": {
      "minimumHealthCapacity": 1,
      "maximumOverCapacity": 0.1
  }
}

It's only in mesos where the container get killed with an exit code 137 :

processname:marathon groupname:marathon pid:19 channel:stdout
[2017-04-28 20:21:48,630] INFO Received status update for task kopaxgroup_developers_jenkins-2.dc2b0461-2c4d-11e7-9190-92077ff7c44b: TASK_FAILED (Docker container run error: Container exited on error: exited with status 137) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$a504fd7e:Thread-1442)

[skovhus]

Are you running the tests with the Jest --runInBand cli option? Else can try that.

[skovhus]

Sorry. Just saw you already tried this. : )

[kopax]

Yes I did. I am desperate. Apparently it is a OOM error. Task is killed because the jvm resources aren't properly configured for docker. This is a memory leak in java 8 and I don't know why exactly this is jest task that crash my Jenkins.

I am moving all of my test integration that use jest into gitlab-ci using gitlab-runner.

[kopax]

Could we please change the label of the issue from Needs more info to Need testers ?

[kopax]

After 15 days trying to bypass this issue, it appear it's resolution became critical for us.

We cannot do any javascript build. All our JS application use jest for unit testing, I tried to delegate the test part to gitlab-runner but we still need to produce the docker image which require the build which require the test to be able to run.
We definitively need a support for Jenkins in a mesos environment for deploying to our production our docker images.

This require the test to be able to run in that environment. Still no jest member available for proceeding to deeper tests ?

[cpojer]

Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions. Thank you :)

[paulmoyag]

Software engineer, DevOps & Cloud computing