Tape: Running tests concurrently

[tjconcept]

Why did Tape choose to run tests serially? Is concurrent tests something we might see in the future?

[ljharb]

JS is exclusively single-threaded - I'm not sure what other option you're proposing.

[tjconcept]

I was under the impression that one test block wouldn't run before the last block with a plan finished, but that assumption might be bogus?

If not, what I'm suggesting is calling all test blocks as soon as possible.

[wavded]

@tjconcept currently/parallel/async tests are easy to get wrong and have weird edge cases in my experience and i'm glad tape doesn't do that.. although arguably there are cases for it, but other libs are built for that like https://github.com/vowsjs/vows

[tjconcept]

Okay, fair.
I would suggest those weird edge cases are actually bugs in my software I would like to know about and fix - or are you talking about something in the platform?

[wavded]

no i'm talking about the platform (not your code), now you have one more thing to manage, synchronizing your asynchronous tests to they run in the right order :( if you need to simulate a race condition or something you can still use t.plan and setTimeout in tape or if you use promises (in blue-tape)

[tjconcept]

But that's my point; I would not want there to be any "wrong order". Tests should be independent thus being able to run concurrently without breaking each other.

[ljharb]

They should, of course! However, JS doesn't make it easy (or practically speaking, possible) to enforce that, and it's much harder to track down this class of bugs when test ordering is nondeterministic.

[tjconcept]

Sure, I've tracked some down and it's absolutely horrible.
Still I'd rather know than simply ignore it. Sequential testing could be an optional thing to enable to rule out race conditions.

[substack]

When an error occurs, it's harder to isolate where when tests are running in parallel. Serial execution also make tests more deterministic, which is helpful when tracking down race conditions.

[tjconcept]

When an error occurs, it's harder to isolate where when tests are running in parallel. Serial execution also make tests more deterministic, which is helpful when tracking down race conditions.

Isn't that like saying: "write fewer tests, then your suite will pass more easily and you don't have to maintain as much"? What am I missing?
I don't think the errors I'm catching now (with serial execution) would get any harder to track down. I might have some race conditions I don't even know of though (due to serial execution), and yes, _those_ will be hard to track down, but right now I'm just not seeing them.

[tjconcept]

I can see how the output might pose a challenge. Deterministic TAP output is a nice trait and mocha's approach will make it a bit less transparent. For those situations it would nice to have a way of forcing serial execution.

[substack]

At any rate, here are some options for parallel execution:

• a runner that executes tap-compatible files in parallel and prints a results summary like the tap command
• a tape wrapper library for parallel execution

[Marak]

@substack -

Would be nice to have a tape wrapper library that enables parallel execution of tests.

[wavded]

You can use the prove command to execute tests parallel (http://perldoc.perl.org/prove.html). Although you don't get granular control over which tests get executed in parallel. I think it would be helpful to have a t.parallel() way to mark a test as being able to run in parallel with others like t.skip (e.g. https://golang.org/pkg/testing/#T.Parallel) and leave it to test runners on how to execute that perhaps.

[Marak]

@wavded -

What I've been considering is spawning a new process per test file. So the syntax for writing tests remains the same, the only thing that changes is how the runner runs the tests.

I'm trying prove now, do you know the syntax for making it work with tape tests?

[wavded]

@Marak your comment reminded me of prove but now that I'm diving into it, I can't remember if I got the darn thing to work. I was struggling with the -j flag and getting errors in my output. Looking back in the code base we did this in, it ended up being a Node script to run npm test in each directory/module (each had a package.json) in parallel and outputting a tap file for each and then the configuring the CI to bundle them into one report.

[tjconcept]

What I've been considering is spawning a new process per test file. So the syntax for writing tests remains the same, the only thing that changes is how the runner runs the tests.

Why do you think the syntax has to change to be executed in parallel?

[Marak]

I said the syntax would remain the same. Not change.

[tjconcept]

I said the syntax would remain the same. Not change.

Ah, sorry, I understood your comment as: "to keep the syntax the same, we need to spawn processes".

[Marak]

@wavded - I'll probably be doing something similar.

I'll put something together but I'm not sure if I will be able to open-source it.

Hopefully someone releases a library that wraps tape for concurrent tests.

[Raynos]

I think running tests in parallel is something that should be done with processes.

Someone can write a tape test runner that takes a glob pattern and then runs every file in its own processes and linearly reports the results of the test suite.

This means an entire file runs in very simple to debug run to completion and you can parallelize across multiple files.

[Marak]

@Raynos - Are there any open-source solutions available for doing this?

I'm working with a test suite with is going to require parrelization. I've already spec'd out a small tool for the job, but I don't think I will be able to open-source the project.

Would much rather collaborate on an already existing project.

[Raynos]

@Marak Not that I know of.

Maybe try contributing to isaacs/tap since that supports running tap tests as sequential child processes; seems like a good feature for parallel child processes.

[ORESoftware]

hey all, I am writing a library called Suman that will compete with tape, ava etc and I need help finishing it. looking at this thread pains me because it's so obvious that test runners like tape and mocha could benefit greatly from parallelization. it runs tests in parallel or series according to the devs needs and runs tests suites in separate node.js processes. there's a whole host of other advantages as well. if you are willing to help out let me know. library is github.com/ORESoftware/suman, cheers

[ljharb]

@ORESoftware JS can't ever be parallel - it is single-threaded. the term you're looking for is "concurrent" which all of the frameworks you mentioned are. Separate processes works but precludes sharing objects across tests. You can do that just fine with tape by putting each "process"'s tests in a separate file, and running those in parallel with parallelshell, for example.

[ORESoftware]

That's correct. Suman utilizes multiple cores by running each test in a
separate process which does preclude memory sharing which is exactly what
you want with testing. Suman uses dependency injection so you can share
code across processes whilst not sharing memory, if you can follow that. As
for concurrency vs parallelism, sure we can call it concurrency and in that
case we can call Suman concurrent and Tape is not. There's a huge
difference in leverage async I/O and the concurrency that comes with it.
Tape does not leverage it. Mocha does not leverage it. Suman and AVA
leverage it.
On Feb 13, 2016 12:38 AM, "Jordan Harband" [email protected] wrote:

@ORESoftware https://github.com/ORESoftware JS can't ever be parallel -
it is single-threaded. the term you're looking for is "concurrent" which
all of the frameworks you mentioned are. Separate processes works but
precludes sharing objects across tests. You can do that just fine with tape
by putting each "process"'s tests in a separate file, and running those in
parallel with parallelshell, for example.

—
Reply to this email directly or view it on GitHub
https://github.com/substack/tape/issues/215#issuecomment-183625280.

[ORESoftware]

But you are wrong about this Justin - not all frameworks are concurrent.
Mocha, Tape are not concurrent, AVA and Suman are. There's a big
difference. It's all about leveraging async I/O. Mocha and tape run tests
serially which means one test has to complete before the next test case can
even start. In AVA and Suman even if you have one test suite in a single
node processe, test cases can run at the same time, if you so choose. You
can choose to run test cases serially or concurrently. I will literally
walk you through a demonstration if you want.
On Feb 13, 2016 11:05 AM, "Alexander Mills" [email protected] wrote:

That's correct. Suman utilizes multiple cores by running each test in a
separate process which does preclude memory sharing which is exactly what
you want with testing. Suman uses dependency injection so you can share
code across processes whilst not sharing memory, if you can follow that. As
for concurrency vs parallelism, sure we can call it concurrency and in that
case we can call Suman concurrent and Tape is not. There's a huge
difference in leverage async I/O and the concurrency that comes with it.
Tape does not leverage it. Mocha does not leverage it. Suman and AVA
leverage it.
On Feb 13, 2016 12:38 AM, "Jordan Harband" [email protected]
wrote:

@ORESoftware https://github.com/ORESoftware JS can't ever be parallel

• it is single-threaded. the term you're looking for is "concurrent" which
  all of the frameworks you mentioned are. Separate processes works but
  precludes sharing objects across tests. You can do that just fine with tape
  by putting each "process"'s tests in a separate file, and running those in
  parallel with parallelshell, for example.

—
Reply to this email directly or view it on GitHub
https://github.com/substack/tape/issues/215#issuecomment-183625280.

[tjconcept]

Suman uses dependency injection so you can share code across processes whilst not sharing memory

How does that work? I have never seen dependency injection across threads, or is it more like sending some serialisable data?

Suman utilizes multiple cores by running each test in a separate process

we can call Suman concurrent and Tape is not

I'm confused. Is it concurrent or multi-threaded or both?

test cases can run at the same time

Not at the same time, but interleaving, in AVA at least ;)

[ORESoftware]

Suman is just like AVA in terms of concurrency, they are basically the same in that respect. Each suite is designed to run in a separate process out of the box. For any given suite, you can have test cases run in "parallel" if you want using async I/O or async features of the event loop such as setTimeout.

As a quick example.

Here are test cases running in serial, like Mocha and Tape would always be like:

https://gist.github.com/ORESoftware/bcab3bcaa36427ec7cce

and here are test cases running in parallel:

https://gist.github.com/ORESoftware/35305f45f6eea17de83f

See the difference? :)

this is with setTimeout, but the same behavior will happen with async I/O.

the developer is in control, s/he can decide which test cases are appropriate for running in parallel or not. Parallel in this case is an OK word to use because it's more than likely that one test will start processing before another finishes, but if you prefer "interleaving", then pretend I said that instead.

You might ask, well what about befores/afters/beforeEaches/afterEaches, can they be concurrent as well?...the answer is - befores/afters are always blocking, and beforeEaches and afterEaches are only blocking on a per test case basis, hope that makes sense. The simple way to prevent data sharing between tests is to attach data to each test, the same way you attach data to req in an Express server. Not that complicated.

If you have any trouble running the above gists, please let me know, Suman is in beta. But Suman is going to give AVA a run for its money. I can talk about dependency injection on another thread, because that would be off topic. But Suman's biggest advantage over AVA would be that.

[phaux]

I think tape should remain as it is. If concurrent tests fit my use case better I use Vows.

[keithkml]

Our test suite currently takes several minutes to run with coverage enabled. During this run, I can see 7 cores on my computer sitting idle.

I'd love it if tape spawned 7 more node processes, divided tests equally among them, and consolidated the output.

[jdrew1303]

So it is possible to run your test concurrently (if by concurrently you mean running separate suites/files in a child process). I think the is what @keithkml was getting at?

Here's my code for handling this:

const spawn = require('child_process').spawn;
const numCPUs = require('os').cpus().length;
const Promise = require('bluebird');
const glob = Promise.promisify(require("glob"));

// The basic idea here is that we spawn a child process to run each test suite in 
// our test folder up to the number of CPUs present in the system. We stream all 
// the output back to the CLI and we handle everything as a promise to make the 
// code easier to handle. This should speed up the tests significantly. NOTE there 
// is a minor issue with the output being muddled up when passed through faucet. 😟 
// This is causing two tests to fail only when passed through faucet (That I 
// haven't defined). Use tap-summary instead. It handles output perfectly. 🕵💪
// 
// Output:
// 
// not ok 7 assert out of order
// not ok 8 unexpected additional plan
// ⨯ fail  2
// 
// Other then this it all seems to be quite successful 🤓
function loadProcess(filename) {

    return new Promise(function(resolve, reject) {
        const process = spawn('node', [filename]);
        const data = [];
        process.stdout.on('data', function(d) {
            data.push(d);
        });

        process.stderr.on('data', function(err) {
            reject(err.toString());
        });

        process.on('exit', function() {
            console.log(data.join(''));
            resolve();
        });
    });
}

return glob(`${__dirname}/**/*.spec.js`)
    .map((f) => loadProcess(f), {
        concurrency: numCPUs
    })

This is the output from tap-summary:

jdrew1303:~/workspace $ node ./test/runner.js | tap-summary

------------------------------------ Tests -------------------------------------

✔ Can naively parse a function. Dont expect anything too fancy. No ES6 😬 [pass: 1, fail: 0, duration: 168ms]
✔ Can create a new container [pass: 1, fail: 0, duration: 1ms]
✔ Can accept a simple binding [pass: 1, fail: 0, duration: 15ms]
✔ Parses a function declaration. [pass: 1, fail: 0, duration: 0ms]
✔ Parses a functional expression. [pass: 1, fail: 0, duration: 0ms]
✔ Parses a generator function. [pass: 1, fail: 0, duration: 0ms]
✔ Parses an arrow function [pass: 1, fail: 0, duration: 1ms]
✖ Parses complex default params [pass: 0, fail: 1, duration: 1ms]
✖ Parses object destructuring param definitions. [pass: 0, fail: 1, duration: 1ms]
✖ Parses object destructuring param definitions. [pass: 0, fail: 1, duration: 1ms]
✔ Parses a class constructor when passed through [pass: 1, fail: 0, duration: 4ms]

----------------------------------- Summary ------------------------------------

duration: 192ms
planned: 11
assertions: 11
pass: 8
fail: 3

------------------------------------ Fails -------------------------------------

# Parses complex default params
  ✖ should be equivalent
    operator: deepEqual
    expected: |-
      { params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
    actual: |-
      { params: [ undefined, 'parses', 'me' ], type: 'FunctionDeclaration' }
    at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:65:7)

# Parses object destructuring param definitions.
  ✖ should be equivalent
    operator: deepEqual
    expected: |-
      { params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
    actual: |-
      { params: [ undefined ], type: 'FunctionDeclaration' }
    at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:76:7)
  ✖ should be equivalent
    operator: deepEqual
    expected: |-
      { params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
    actual: |-
      { params: [ undefined ], type: 'FunctionDeclaration' }
    at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:85:7)

I'm not sure how this will handle in a CI environment. I haven't tried. Any help or ideas on this would be great. I wouldnt mind if this was a bit more robust. If these issues were resolved then I might wrap it with a cli lib and publish it.

Hope this helps a bit 🙋🏻

[DiegoRBaquero]

@jdrew1303 Were you able to make it a CLI module?

EDIT: I made bogota, runs test files in parallel, outputs tap-spec: https://www.npmjs.com/package/bogota

[ORESoftware]

Use AVA, Suman, or just put your tape tests in a bash file:

tape x &
tape y &
tape z &
wait

[ljharb]

@ORESoftware i don't think that will exit nonzero if any of the subtasks do, tho.

[ORESoftware]

Good point dunno - maybe you can grab the exit codes somehow?

[ORESoftware]

http://unix.stackexchange.com/questions/344360/collect-exit-codes-of-parallel-background-processes-sub-shells

will report back upon answers received

[ORESoftware]

@ljharb et al, you can run things in parallel with a bash script like so:

http://unix.stackexchange.com/questions/344360/collect-exit-codes-of-parallel-background-processes-sub-shells/344387#344387

#!/usr/bin/env bash

arr=()  # holds PIDs

commands=(
    "echo 'hallelujah'; exit 0;"
    "echo 'marzepan'; exit 3;"
)

clen=`expr "${#commands[@]}" - 1` # get length of commands - 1

for i in `seq 0 "$clen"`
do
   # run the command via bash in subshell and push PID to array
    (echo "${commands[$i]}" | bash) & arr+=($!)  
done

len=`expr "${#arr[@]}" - 1`  # get length of arr - 1

EXIT_CODE=0;  # exit code of overall script

for i in `seq 0 "$len"`
do
    pid="${arr[$i]}";
    echo "PID => $pid"
    wait ${pid} ; CODE="$?"
    if [[ ${CODE} > 0 ]]; then
       echo "at least one test failed";
      EXIT_CODE=1;
      fi
done

echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"

throw your tape commands in the commands=() array, and you should see what you'd expect to see. bash is not pretty, and I avoid it at almost all costs. but if you want a good way to run tape tests concurrently, then this is it.

[ORESoftware]

Actually this script is better, using trap; if you use wait -n or wait <pid> you get fallthrough

#!/usr/bin/env bash

set -m # allow for job control
EXIT_CODE=0;  # exit code of overall script

function foo() {
   echo "CHLD exit code is $?"
   if [[ $? > 0 ]]; then
      echo "at least one test failed";
      EXIT_CODE=1;
   fi
}

trap 'foo' CHLD

DIRN=$(dirname "$0");

commands=(
    "tape 'x'; exit 1;"
    "tape 'y'; exit 0;"
    "tape 'z'; exit 2;"
)

clen=`expr "${#commands[@]}" - 1` # get length of commands - 1

for i in `seq 0 "$clen"`; do
    (echo "${commands[$i]}" | bash) &   # run the command via bash in subshell
    echo "$i ith command has been issued as a background job"
done

# wait for all to finish
wait;

echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"

# end

[DiegoRBaquero]

Just publish bogota version 2! It require Node 6+ but runs tests concurrently very well! Running webtorrent's tests in 1.2x - 2x speed.

It's great for tests that have timeouts/network!

[iagomelanias]

@DiegoRBaquero I have tested here, but sadly the tests never exit when injecting a server.

[DiegoRBaquero]

@iagomelanias You tested running tape's tests? tape's tests use tap too, which I don't support.

[pbouzakis]

For linux/mac users, moreutil's parallel command seems to do the trick:

parallel tape -- ./src/**/*.test.js | tap-summary

[ORESoftware]

@pbouzakis we'd have to verify that this tool captures exit codes from each subshell and then exits with 1 if any of the subshells exit with non-zero - my guess is that it does not.

[imsnif]

I hope it's okay to revive this inactive thread, but I created the aforementioned wrapper library for tape that runs tests concurrently (on the same thread). I hope this will come through for people looking here:

https://github.com/imsnif/mixed-tape

[screendriver]

With one of the latest Node.js versions it is now possible to use Worker Threads (the are like Web Workers in the Browser) directly in Node.js 😱 This should definitely help to implement parallel execution of tests in tape ☺️

[dave-dm]

I've implemented a small concurrent runner that you can add inside single files. You replace test with testConcurrent and then call runConcurrent at the end. The console output gets mixed up though. Maybe someone can improve on it. There's also a bug when there's only synchronous tests, it adds an extra plan. Also if someone knows a way to remove the need to call runConcurrent that'd be nice.

class TestConcurrent {
  constructor (tape) {
    this._tape = tape
    this._tests = []
    this.testConcurrent = this.testConcurrent.bind(this)
    this.runConcurrent = this.runConcurrent.bind(this)
  }

  testConcurrent (description, test) {
    this._tests.push({description, test})
  }

  runConcurrent () {
    this._tape('concurrent tests', async t => {
      const plan = t.plan
      let planCount = 0
      t.plan = (incrementBy) => planCount += incrementBy

      const promiseArray = []
      for (const test of this._tests) {
        const promise = new Promise(async resolve => {
          t.comment(test.description)
          await test.test(t)
          resolve()
        })
        promiseArray.push(promise)
      }

      plan(planCount)
      await Promise.all(promiseArray)
    })
  }
}

const test = require('tape')
const {testConcurrent, runConcurrent} = new TestConcurrent(test)

testConcurrent('test1', t => {
  t.plan(1)
  t.pass('done1')
})
testConcurrent('test2', async t => {
  t.plan(1)
  await new Promise(resolve => setTimeout(resolve, 4000))
  t.pass('done2')
})
testConcurrent('test3', async t => {
  t.plan(1)
  await new Promise(resolve => setTimeout(resolve, 5000))
  t.pass('done3')
})
test('regular test', t => {
  t.plan(1)
  t.pass('done')
})

runConcurrent()

[ljharb]

Using worker_threads isn't something that can be done transparently to tape test authors.

It would be possible to run each file passed to tape - only the binary, not the library - in a separate process, but that would add a lot of complexity both around spawning and managing the processes, as well as around correctly collating and displaying output.

Instead of any changes to tape here, I think that someone should invest the time to create a jest tape runner - that way, all of jest's already-solved parallelism can be applied to tape tests, pretty transparently for the test author, without adding complexity to tape.

I would be more than happy to help maintain such a runner in a different repository, and publicize/advocate for it, if someone did the initial work to create it. In the meantime, I'm going to close this.