Behat: parallel execution of tests

[agiledivider]

Hi!

I am quite interested to get results from tests quickly. So we are currently reaching a point, where we are splitting tests into groups to run them in parallel. But execution times are growing.

What about a central feature to let behat fork testrunners, so we can run tests in parallel? I seriously would love that.

Would need an "@exclusive" tag to run a test alone or a "@group a" which would not run grouped tests in parallel. But we are closing in on 30m tests, which is much tooo long for my taste. ;)

Best regards
Falk

[stof]

you can already use tags to exclude scenarios (or features as scenarios inherit the tags from the feature) from the run: http://docs.behat.org/guides/6.cli.html#gherkin-filters

[everzet]

As @stof mentioned, you could run only specific set of features (filtering by tags, titles) and group them into profiles. Multiprofiles runner is coming.

[agiledivider]

I know that. And we are already doing that, but why should i reorganize my tests all the time, if i want to run them in parallel. Adding or removing tags to hundreds of tests seems inefficient.

I would rather mark the exceptions and let behat handle the parallell testrunning. I could just tell behat to run 1 test in parallel :) or 10 with a single configuration option.

That would speed up our testing proccess immensely.

[everzet]

@cogitatio as i've said, multirunner is in works :) I'm planning to do controller around behat CLI to do multiprofile/multiserver runs.

[agiledivider]

Interesting, what exactly are you planning there?

[glenjamin]

@cogitatio I'm intrigued, if you just fork the process and run some tests in each one - how much speedup would you expect to get?

I'd expect that the only way to get a massive speed up by parallelisation would be to split across multiple boxes - especially if you're using something like Mink to make HTTP calls to a web service that has some sort of state (database for example). How do you stop the two parallel tests from clobbering each other's state?

[robocoder]

@everzet does that mean the multirunner should still work if I override behat.console.command.class?

[everzet]

dunno :)

[cgsmcmlxxv]

@everzet, I successfully ran behat tests in parallel threads using pcntl_fork without clobbering the thread sates. What is your status? Can I help?

[MPV]

Interesting topic.

I ran across parallel_tests for Cucumber the other day that seems to do something similar for the Ruby people.
Maybe you've already seen it – otherwise it could serve as some inspiration?

[fpalomo]

hello guys,

any news on this feature?

[odino]

+1

any news?

[VladSavitsky]

Currently I'm running 4 processes of behat at the same time.

I'm using drupal which allow to have multisiting configuration which means that I have 1 copy of code base. But it's not a problem to clone whole project in separate folders and run behat in each of them.

Then I created separate database for each site so there will be no conflicts between behat-processes. They could happen when one process creates data and another removes it.

Next I've created separate behat.yml file which point to desired domain. I've placed this files in the same folder with default behat.yml.

Now I could run several behat-processes using tags to separate test scenarios:
./behat --config .//behat.clean.test1.yml -f "pretty,html" --out ", ./reports/report_clean.test1_date +%Y-%m-%d--%H:%M:%S.html" --tags "~@access&&~@menu&&~@permissions"

Next process should include only scenarios excluded before:
./behat --config .//behat.clean.test2.yml -f "pretty,html" --out ", ./reports/report_clean.test2_date +%Y-%m-%d--%H:%M:%S.html" --tags "@access,@menu,@permissions"

Note! There is a difference in tags in this two commands.

If there is another way to run it I'll try because configuration takes a lot of time.

[odino]

@everzet any idea?

[neochief]

Ok, you've asked, I've got it :)
https://github.com/shvetsgroup/ParallelRunner

[fantasticjamieburns]

If your fixtures are isolated properly then this works well under linux whilst developing:

find ./features -name *.feature \
| parallel --gnu --halt-on-error=2 --keep-order \
  bin/behat --ansi --format=pretty

The "parallel" command will automatically detect the number of CPU cores available and scale accordingly. In addition, it can be configured to spread the load across a cluster of machines via SSH.

Caveat: this approach will not aggregate the total passed/skipped/failed for reporting purposes.

[rajeshtaneja]

+1
@neochief Any plans to update ParallelRunner to 3.x ?

[eddiejaoude]

My 2 cents...

Robo Task Runner is another option http://robo.li/tasks.html#parallelexec

Codeception has a good usage example: http://codeception.com/docs/12-ParallelExecution

eg.

    function parallelAll()
    {
        $this->parallelSplitTests();
        $result = $this->parallelRun();
        $this->parallelMergeResults();
        return $result;
    }

---

@fantasticjamieburns great command for pure maximum speed :+1:

[rajeshtaneja]

Thanks @eddiejaoude

I am trying this now. Nice framework :+1:

[timiTao]

@eddiejaoude first link is not working...

[eddiejaoude]

@timiTao Robo have updated their website & I can't find their equivalent docs for that page. The 2nd link has an example of how to use Robo in parallel.

It might be work logging an issue/ticket with Robo

[jwfuller]

http://robo.li/tasks/Base/#parallelexec

[timiTao]

@jwfuller thaks a lot :) :+1:

[jhenya]

Hi there
fantasticjamieburns,
Did you try it?
"Caveat: this approach will not aggregate the total passed/skipped/failed for reporting purposes." commented on 1 Jul 2013

[everzet]

I'm closing this issue as this will not happen anymore. There are easier ways of achieving parallel execution. I also reached conclusion that parallelisation of tests creates a trade-off you'd never want to have in your tests - trading stability for speed. The best way to improve speed of your tests is following a testing pyramid.

[kandelyabre]

you can try extension https://packagist.org/packages/tonicforhealth/behat-parallel-scenario

[ioleo]

Also see liuggio/fastest

[jhenya]

I implement parallel execution it with xargs and GNU Parallel and implement consolidated report based on BehatHtmlFormatterPlugin extension. Please see details here
http://parallelandvisualtestingwithbehat.blogspot.com/p/blog-page.html

[seyfer]

@jhenya will not work to run tests just in parallel since with one DB parallel changes via UI lead to other tests to fail.

@ioleo thanks, so far the most advanced tool. But for me it failed to run, I have created an issue.

@neochief tried this one also. Outdated for Behat 3. Tried to fix it, but doesn't work now.
If anyone can have a look, would be great.

Thanks all for sharing.
What worked for me so far is not paralleling, but running chrome without Selenium in headless mode. It 2-3x times faster. But without VNC.

[PurHur]

For people passing by: You can utilize docker commit to split the test run path with Database and files... But depends on docker...