Node: CI is unusable

Created on 25 Feb 2020  ·  16Comments  ·  Source: nodejs/node

Latest build history:

ci-history

CI / flaky test build
Source
picture tniessen

Most helpful comment

As of right now, the highest failure rate is on node-test-commit-linux-containered.

picture Trott on 29 Mar 2020
👍2

All 16 comments

cc @Trott @nodejs/build

tniessen picture tniessen on 25 Feb 2020

Browsing through some of these I'm not seeing a pattern that pertains to our build infra.

Unless you can hone in on a particular infra issue, it's on you all to give our infra better code so we don't have so many flaky failures.

rvagg picture rvagg on 26 Feb 2020
👍1

@rvagg @tniessen One of the known_issues tests is failing (that is, passing) pretty much constantly on Raspberry Pi devices. I don't know why that started but it's very recent, probably in the last 24 hours I'd say.

Trott picture Trott on 26 Feb 2020
👍1

I am going through some of the failures and triaging those.

gireeshpunathil picture gireeshpunathil on 26 Feb 2020
👍1

@gireeshpunathil I would suggest not going through build failures in node-test-pull-request, failures are expected on PRs (code fails), so its hard to detangle expected vs unexpected failures.

I suggest you look at https://ci.nodejs.org/job/node-daily-master/ -- those builds _all_ come from master, therefore every failure there is either a failure in infrastructure, or a flaky test.

https://github.com/nodejs/node-core-utils/pull/379 can be helpful in analysis.

output

core/node-core-utils (add-daily-master $% u=) % ./bin/ncu-ci daily
✔  Done
--------------------------------------------------------------------------------
[1/8] Running health
--------------------------------------------------------------------------------
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-02-26 15:02 | 0       | 1       | 4        | 0       | 7       | 8.33%      |

--------------------------------------------------------------------------------
[2/8] Running https://ci.nodejs.org/job/node-daily-master/1847/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35963/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [940325042bef] test: add secp224k1 check in crypto-dh-stateless
Date       2020-02-25 06:14:32 +0100
Author     Daniel Bevenius <[email protected]>
------------------------ RUN_SUBSET=0,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4539/console
Type       JS_TEST_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Reason
  not ok 650 known_issues/test-vm-timeout-escape-queuemicrotask
    ---
    duration_ms: 1.243
    severity: fail
    stack: |-
    ...

------------------------ RUN_SUBSET=0,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4539/console
Type       JS_TEST_FAILURE
Built On   test-requireio_securogroup-debian10-arm64_pi3-2
Reason
  not ok 660 parallel/test-worker-stdio
    ---
    duration_ms: 251.448
    severity: fail
    exitcode: -15
    stack: |-
      timeout
    ...

--------------------------------- Other builds ---------------------------------
--------------------------------------------------------------------------------
[3/8] Running https://ci.nodejs.org/job/node-daily-master/1846/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-commit/35927/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [9c702922cdcf] async_hooks: introduce async-context API
Date       2020-02-23 20:24:02 +0100
Author     Vladimir de Turckheim <[email protected]>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13338/
--------------------------------------------------------------------------------
[4/8] Running https://ci.nodejs.org/job/node-daily-master/1844/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35887/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [2f23918ca509] doc: update stream.pipeline() signature
Date       2020-02-22 08:18:43 +0100
Author     Luigi Pinca <[email protected]>
------------------------------- node-test-linter -------------------------------
URL        https://ci.nodejs.org/job/node-test-linter/32923/console
Type       BUILD_FAILURE
Built On   test-packetnet-ubuntu1604-x64-1
Reason
  Error: No module named enum
  Makefile:1347: recipe for target 'lint-py' failed
  make: *** [lint-py] Error 1
  + cat test-eslint.tap
  + grep -v '^ok\|^TAP version 13\|^1\.\.'
  + sed '/^\s*$/d'
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13298/
--------------------------------------------------------------------------------
[5/8] Running https://ci.nodejs.org/job/node-daily-master/1841/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-commit/35839/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [b8e41774d428] fs: add fs/promises alias module
Date       2020-02-18 22:15:50 -0800
Author     Gus Caplan <[email protected]>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13255/
--------------------------------------------------------------------------------
[6/8] Running https://ci.nodejs.org/job/node-daily-master/1839/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35767/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [4c746a6cfda9] doc: move gireeshpunathil to TSC emeritus
Date       2020-02-17 10:33:54 +0530
Author     Gireesh Punathil <[email protected]>
--------------- RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017 ----------------
URL        https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017/1820/console
Type       JENKINS_FAILURE
Built On   test-rackspace-win2008r2_vs2017-x64-3
Reason
  Build timed out (after 10 minutes). Marking the build as failed.
------------------------------ centos7-arm64-gcc6 ------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29460/console
Type       JS_TEST_FAILURE
Built On   test-packetnet-centos7-arm64-2
Reason
  not ok 2899 sequential/test-worker-prof
    ---
    duration_ms: 0.814
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: child exited with signal: {
        status: null,
        signal: 'SIGSEGV',
        output: [ null, '', '' ],
        pid: 98106,
        stdout: '',
        stderr: ''
      }
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-prof.js:57:10)
          at Module._compile (internal/modules/cjs/loader.js:1206:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1226:10)
          at Module.load (internal/modules/cjs/loader.js:1055:32)
          at Function.Module._load (internal/modules/cjs/loader.js:950:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'SIGSEGV',
        expected: null,
        operator: 'strictEqual'
      }
    ...

------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4399/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=3,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=3,label=pi3-docker/4399/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------- Other builds ---------------------------------
--------------------------------------------------------------------------------
[7/8] Running https://ci.nodejs.org/job/node-daily-master/1838/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35752/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [a7c523e26df4] src: prefer 3-argument Array::New()
Date       2020-02-15 21:39:32 +0100
Author     Anna Henningsen <[email protected]>
------------------------------ centos7-arm64-gcc6 ------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29448/console
Type       JS_TEST_FAILURE
Built On   test-packetnet-centos7-arm64-2
Reason
  not ok 2897 sequential/test-worker-prof
    ---
    duration_ms: 1.716
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: child exited with signal: {
        status: null,
        signal: 'SIGSEGV',
        output: [ null, '', '' ],
        pid: 91489,
        stdout: '',
        stderr: ''
      }
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-prof.js:57:10)
          at Module._compile (internal/modules/cjs/loader.js:1206:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1226:10)
          at Module.load (internal/modules/cjs/loader.js:1055:32)
          at Function.Module._load (internal/modules/cjs/loader.js:950:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'SIGSEGV',
        expected: null,
        operator: 'strictEqual'
      }
    ...

------------------------ RUN_SUBSET=1,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4387/console
Type       JENKINS_FAILURE
Built On   test-requireio_louiscntr-debian10-armv7l_pi2-1
Reason
  java.io.IOException: Backing channel 'JNLP4-connect connection from 203.217.18.114/203.217.18.114:56254' is disconnected.
        at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy90.isAlive(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150)
        at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142)
------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4387/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=3,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=3,label=pi3-docker/4387/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-windows-fanned/33992/
--------------------------------------------------------------------------------
[8/8] Running https://ci.nodejs.org/job/node-daily-master/1837/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35734/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [79296dc2d02c] doc: guide - using valgrind to debug memory leaks
Date       2020-02-14 18:19:16 -0500
Author     Michael Dawson <[email protected]>
------------------------ RUN_SUBSET=0,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_rvagg-debian10-armv7l_pi2-1
Reason
  warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
------------------------ RUN_SUBSET=0,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_williamkapke-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
------------------------ RUN_SUBSET=1,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-2
Reason
  warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=2,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Reason
  warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
------------------------ RUN_SUBSET=2,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_pivotalagency-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-windows-fanned/33978/

------------------------------------ Stats -------------------------------------

------------------------------- JENKINS_FAILURE --------------------------------
Reason     Backing channel 'JNLP4-connect connection from ... is disconnected.
Type       JENKINS_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_louiscntr-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35752/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4387/console

java.io.IOException: Backing channel 'JNLP4-connect connection from 203.217.18.114/203.217.18.114:56254' is disconnected.
        at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy90.isAlive(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150)
        at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142)
--------------------------------------------------------------------------------
Reason     Build timed out (after 10 minutes). Marking the build as failed.
Type       JENKINS_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-rackspace-win2008r2_vs2017-x64-3
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017/1820/console

Build timed out (after 10 minutes). Marking the build as failed.
--------------------------------------------------------------------------------
-------------------------------- BUILD_FAILURE ---------------------------------
Reason     Error: No module named enum
Type       BUILD_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-packetnet-ubuntu1604-x64-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35887/

Example: https://ci.nodejs.org/job/node-test-linter/32923/console

Error: No module named enum
Makefile:1347: recipe for target 'lint-py' failed
make: *** [lint-py] Error 1
+ cat test-eslint.tap
+ grep -v '^ok\|^TAP version 13\|^1\.\.'
+ sed '/^\s*$/d'
--------------------------------------------------------------------------------
------------------------------- JS_TEST_FAILURE --------------------------------
Reason     known_issues/test-vm-timeout-escape-queuemicrotask
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35963/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4539/console

not ok 650 known_issues/test-vm-timeout-escape-queuemicrotask
  ---
  duration_ms: 1.243
  severity: fail
  stack: |-
  ...

--------------------------------------------------------------------------------
Reason     parallel/test-worker-stdio
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_securogroup-debian10-arm64_pi3-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35963/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4539/console

not ok 660 parallel/test-worker-stdio
  ---
  duration_ms: 251.448
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

--------------------------------------------------------------------------------
Reason     sequential/test-worker-prof
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-packetnet-centos7-arm64-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29460/console

not ok 2899 sequential/test-worker-prof
  ---
  duration_ms: 0.814
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:102
      throw new AssertionError(obj);
      ^

    AssertionError [ERR_ASSERTION]: child exited with signal: {
      status: null,
      signal: 'SIGSEGV',
      output: [ null, '', '' ],
      pid: 98106,
      stdout: '',
      stderr: ''
    }
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-pr...
--------------------------------------------------------------------------------
--------------------------------- GIT_FAILURE ----------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_williamkapke-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4370/console

warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4399/console

warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_pivotalagency-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi3-docker/4370/console

warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_rvagg-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
--------------------------------------------------------------------------------
core/node-core-utils (add-daily-master $% u=) %

sam-github picture sam-github on 26 Feb 2020
👍1

apologies all, half of my triages went erroneous. Everyday something new to learn, so!

gireeshpunathil picture gireeshpunathil on 27 Feb 2020
🎉1

We can land PRs due to CI being yellow-ish (which means "acceptable failure"), but it is still not looking great:

ci-history

tniessen picture tniessen on 13 Mar 2020
👍2

We can land PRs due to CI being yellow-ish (which means "acceptable failure"), but it is still not looking great:

Perma-yellow at this time is to due to node-test-commit-osx, specifically the new osx1015. Given that we have lots of macOS users as Collaborators, I'm hopeful that someone can dig deep on those failures and get that working reliably. @nodejs/platform-macos

Trott picture Trott on 13 Mar 2020

@Trott we are waiting on Apple to fix their regression to lift us out of perma yellow - https://github.com/nodejs/node/issues/30030#issuecomment-594840385

AshCripps picture AshCripps on 13 Mar 2020
👍1

I still think that case should be skipped, perhaps actual js code (since it doesn't need to be skipped for all OS X versions). Our CI shouldn't be yellow if Apple, at their discretion, no longer ship a mostly unused network stack feature (kq notification of zero-length udp packets). It should be yellow if _we_ have a bug.

sam-github picture sam-github on 13 Mar 2020
👍1

I still think that case should be skipped, perhaps actual js code (since it doesn't need to be skipped for all OS X versions). Our CI shouldn't be yellow if Apple, at their discretion, no longer ship a mostly unused network stack feature (kq notification of zero-length udp packets). It should be yellow if _we_ have a bug.

There's precedence for this too. We definitely used to have code in a test that skipped if it detected it was on macOS at a certain version. We took it out when Apple fixed the bug.

I agree with Sam that an OS bug that causes a test to fail all the time should be skipped. (Even better if a second nearly-identical test is added to known_issues so that we can detect when the issue is fixed. But that's a bonus.) Tests should be marked flaky if they are unreliable, not if they are completely broken.

Trott picture Trott on 13 Mar 2020

Which test(s) should be skipped on osx1015?

mmarchini picture mmarchini on 13 Mar 2020

The ones marked flaky here - https://github.com/nodejs/node/pull/31936/files

would need backporting too to at least 12 (some are missing on 10 so its not as affected IIRC)

AshCripps picture AshCripps on 13 Mar 2020

FWIW the last few PRs I landed I had to run CI a few times before getting a green/yellow, so the problem is not limited to the OS X 10.15 issue.

mmarchini picture mmarchini on 17 Mar 2020

As of right now, the highest failure rate is on node-test-commit-linux-containered.

Trott picture Trott on 29 Mar 2020
👍2

Given that we have the flaky test project board and there really hasn't been updates on this in a while, closing. Can reopen if necessary.

jasnell picture jasnell on 26 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

stevenvachon picture stevenvachon  ·  3Comments
srl295 picture srl295  ·  3Comments
cong88 picture cong88  ·  3Comments
mcollina picture mcollina  ·  3Comments
danielstaleiny picture danielstaleiny  ·  3Comments