Testng: Suite of suites testng.xml parallel execution

Created on 6 Feb 2018  路  26Comments  路  Source: cbeust/testng

TestNG Version

            <dependency>
                <groupId>org.testng</groupId>
                <artifactId>testng</artifactId>
                <version>6.9.10</version>
                <scope>test</scope>
            </dependency>

Expected behavior

When running a suite of suites via testing.xml I expect each class within a suite to run in parallel.

Actual behaviour

Each suite runs in parallel but classes within the suite run sequentially:
suitea testcase 1 and suite b testcase2 run in parallel
suitea testcase3 runs after testcase1 completes

Is the issue reproductible on runner?

IntelliJ

Test case sample

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
    <suite-files>
        <suite-file path="suitea.xml" />
        <suite-file path="suiteb.xml" />
    </suite-files>
</suite> <!-- Suite -->

------suitea.xml--------------

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
    <test name="FirstTest" thread-count="5" parallel="methods" >
        <parameter name="browser" value="chrome"/>
        <parameter name = "title" value="SS1 Verify the login section"/>
        <classes>
            <class name="testsuite.TestCase1"/>
        </classes>
    </test> <!-- First Test -->
    <test name="SecondTest" thread-count="5" parallel="methods" >
        <parameter name="browser" value="chrome"/>
        <parameter name = "title" value="SS2 Second Login Test"/>
        <classes>
            <class name="testsuite.TestCase3"/>
        </classes>
    </test> <!-- Second Test -->
</suite> <!-- Suite -->

------suiteb.xml--------------

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">

    <test name="SecondTest" thread-count="5" parallel="methods" >
        <parameter name="browser" value="chrome"/>
        <parameter name = "title" value="SR1 Verify item in mobile list page can be sorted by name"/>
        <classes>
            <class name="testsuite.TestCase2"/>
        </classes>
    </test> <!-- First Test -->
</suite> <!-- Suite -->
multi-suite parallel

Most helpful comment

@krmahadevan as far as i see, in your example suites are executed sequentially instead of parallel (first - suitea.xml, then suiteb.xml), but the main issue is how to run suitea and suiteb parallel

All 26 comments

@petermartin2002 - Can you please retry using the latest released version of TestNG 6.14.2 and let us know if the issue exists ? 6.9.10 is a very old version.

Upgraded and still the same issue

@petermartin2002 - Can you please help share a sample that can be used to recreate this issue ?

testsuite.zip
framework.zip

Test cases, pom.xml, and testng, and suite xml attached.

It is really straightforward to recreate from that.

Any advice here folks?

When I run the suites on their own they run in parallel but the suite of suites runs each suite and its subsequent tests serially. Quite frustrating

@petermartin2002 - The attached project contains references to some classes that aren't included. So its not possible to reproduce the issue.

I would request you to please attach a simple test project (without any page objects or selenium dependency), that can be used to reproduce the issue.

im still encountering this parallel testing problem when i run it two actual android device in v.6.14.3. any idea how can i resolve this. thanks in advance :)

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Automation Suite POC 2" parallel="tests" thread-count="2">
    <test name="Test : TOMini 121">
         <parameter name="Port" value="4723"/>
        <parameter name="UDID" value="192.168.8.121:5555"/>
        <classes>
            <class name="com.xxx.tob.settings.WatchInSettingsTest"/>
        </classes>
    </test>
    <test name="Test : TOMini 131">
        <parameter name="Port" value="4725"/>
        <parameter name="UDID" value="192.168.8.131:5555"/>
        <classes>
            <class name="com.xxx.tob.settings.WatchInSettingsTest"/>
        </classes>
    </test>
</suite>

--pom..xml

 <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.14.3</version>
            <scope>test</scope>
  </dependency>

screen shot 2018-04-20 at 10 28 34 am

@glenn04 - Can you please help create a simple standalone project (which uses nothing outside of core java) that we can use to reproduce the problem ?

Hello,
@glenn04 and @petermartin2002 Have you fix your problem ? Because I have exactly the same issue... Please here is a simple project without other dependencies. Ready to use !
issues1686.zip
To execute :
mvn clean verify -DsuiteXmlFile=myTestng.xml

Thank you :)

@albanlorillard not yet bro, no answer yet from @krmahadevan ?

@glenn04 - I will get to this as soon as possible. A bit swamped by stuff.

@albanlorillard - Thanks for providing a reproducible test. I will try using that to see what is going on.

Hello @krmahadevan ,
Please consider below sample code.

addProduct.txt
LoginTest.txt

Testng.xml










pom.xml


org.testng
testng
6.14.3
test

Also my all test run in parallel.
Out Put :
Product Added
User Login
Product Edited
Register User
Delete Product
Forgot Password

===============================================
Suite

Total tests run: 6, Failures: 0, Skips: 0

Did any one found any solution ?

if you make yours testng.xml as shown below then your test will execute in proper manner.

MyTestng.txt

Share your experience or thoughts
@VishnuGupta16

@DhavalAtri But it is not suite of suites
The question was about suite of suites parallel run

@krmahadevan Any updates on this?
I am using cucumber testng and have the same problem with suite of suites (that listed there suited runs one after another, not in parallel)
Have tried with 'cucumber-testng', version: '1.2.5' & 'cucumber-testng', version: '1.2.4'

@tarasmytlovych - No I am yet to get to this issue. Is this a problem when using TestNG 7.0.0-beta3 (latest released version as of today) ?

I have tried reproducing this issue with TestNG 7.0.0-beta3 and I can't seem to recreate it

@tarasmytlovych - Unless you have a reproducible test that can still reproduce the problem, I may have to close this issue.

Here's what I have (Its the same example that @petermartin2002 had shared when the bug was created. I just cleaned it up and added thread id printing)

package com.rationaleemotions.github.issue1686;

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestCase1 {
  @BeforeMethod
  public void beforeMethod() {
    System.err.println(getPrefix() + getClass().getName() + " triggered beforeMethod() on #" + Thread.currentThread().getId());
  }

  @Test(priority = 1, description = "SS1 Verify the login section")
  public void login() {
    System.err.println(getPrefix() + getClass().getName() + " triggered login() on #" + Thread.currentThread().getId());
  }

  @AfterMethod
  @Parameters("title")
  public void validate(String title) {
    System.err.println(getPrefix() + getClass().getName() + " triggered validate() with parameter [" + title + "] on #" + Thread.currentThread().getId());
  }

  private static String getPrefix() {
    ITestResult result = Reporter.getCurrentTestResult();
    String suiteName = result.getTestContext().getSuite().getName();
    String testTagName = result.getTestContext().getName();
    return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
  }

}
package com.rationaleemotions.github.issue1686;

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class TestCase2 {
  @BeforeMethod
  public void beforeMethod() {
    System.err.println(getPrefix() + getClass().getName() + " triggered beforeMethod() on #" + Thread.currentThread().getId());
  }

  @Test(priority = 1, description = "SR1 Verify item in mobile list page can be sorted by name")
  public void SR1() {
    System.err.println(getPrefix() + getClass().getName() + " triggered SR1() on #" + Thread.currentThread().getId());
  }

  @AfterMethod
  public void validate(){
    System.err.println(getPrefix() + getClass().getName() + " triggered validate() on #" + Thread.currentThread().getId());
  }

  private static String getPrefix() {
    ITestResult result = Reporter.getCurrentTestResult();
    String suiteName = result.getTestContext().getSuite().getName();
    String testTagName = result.getTestContext().getName();
    return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
  }

}
package com.rationaleemotions.github.issue1686;

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestCase3 {
  @BeforeMethod
  public void beforeMethod() {
    System.err.println(
        getPrefix()
            + getClass().getName()
            + " triggered beforeMethod() on #"
            + Thread.currentThread().getId());
  }

  @Test(priority = 1, description = "SS1 Verify the login section")
  public void login() {
    System.err.println(
        getPrefix()
            + getClass().getName()
            + " triggered login() on #"
            + Thread.currentThread().getId());
  }

  @AfterMethod
  @Parameters("title")
  public void validate(String title) {
    System.err.println(
        getPrefix()
            + getClass().getName()
            + " triggered validate() with parameter ["
            + title
            + "] on #"
            + Thread.currentThread().getId());
  }

  private static String getPrefix() {
    ITestResult result = Reporter.getCurrentTestResult();
    String suiteName = result.getTestContext().getSuite().getName();
    String testTagName = result.getTestContext().getName();
    return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
  }
}

Suites look like below

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
  <test name="FirstTest" thread-count="5" parallel="methods">
    <parameter name="browser" value="chrome"/>
    <parameter name="title" value="SS1 Verify the login section"/>
    <classes>
      <class name="com.rationaleemotions.github.issue1686.TestCase1"/>
    </classes>
  </test> <!-- First Test -->
  <test name="SecondTest" thread-count="5" parallel="methods">
    <parameter name="browser" value="chrome"/>
    <parameter name="title" value="SS2 Second Login Test"/>
    <classes>
      <class name="com.rationaleemotions.github.issue1686.TestCase3"/>
    </classes>
  </test> <!-- Second Test -->
</suite> <!-- Suite -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">

  <test name="SecondTest" thread-count="5" parallel="methods" >
    <parameter name="browser" value="chrome"/>
    <parameter name = "title" value="SR1 Verify item in mobile list page can be sorted by name"/>
    <classes>
      <class name="com.rationaleemotions.github.issue1686.TestCase3"/>
    </classes>
  </test> <!-- First Test -->
</suite> <!-- Suite -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests" verbose="2">
  <suite-files>
    <suite-file path="./suitea.xml" />
    <suite-file path="./suiteb.xml" />
  </suite-files>
</suite> <!-- Suite -->

Execution output

...
... TestNG 7.0.0-beta3 by C茅dric Beust ([email protected])
...
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered beforeMethod() on #13
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered beforeMethod() on #14
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered login() on #14
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered login() on #13
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered validate() with parameter [SS1 Verify the login section] on #14
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered validate() with parameter [SS2 Second Login Test] on #13

===============================================
Suite (0)
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered beforeMethod() on #16
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered login() on #16
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered validate() with parameter [SR1 Verify item in mobile list page can be sorted by name] on #16

===============================================
Suite (1)
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

===============================================
Suite
Total tests run: 3, Passes: 3, Failures: 0, Skips: 0
===============================================

Hi All:

Have anybody tried a suite of xml suites syntax with something :
<suite thread-count="3" name="Suite" parallel="xml"> ?

OR
just
parallel="xml"

Have an xml suite os xml suites and need to find a proper syntax.
Thanks in advance for the advice!

@agoldenberg02

<suite thread-count="3" name="Suite" parallel="xml"> ?

There is nothing like that. The only allowed values are

  1. methods
  2. classes
  3. tests
  4. instances

Haven't seen any response for my earlier question on this and since I am also not able to reproduce the problem. Closing this. If this is still a problem, need a reproducible sample using the latest released version of TestNG (7.0.0-beta7 as of today)

@krmahadevan as far as i see, in your example suites are executed sequentially instead of parallel (first - suitea.xml, then suiteb.xml), but the main issue is how to run suitea and suiteb parallel

Even I am waiting for same issue's solution

This is exactly what I am observing as well, even with version 7.1.0. The suite-file's in a suite-files list are executed in sequence, instead of in parallel.

Here's the suite of suite file

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="github-1686-master-suite" verbose="2">
  <suite-files>
    <suite-file path="slave-1.xml"/>
    <suite-file path="slave-2.xml"/>
  </suite-files>
</suite>

And here's how the surefire plugin looks like

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M1</version>
        <executions>
          <execution>
            <phase>test</phase>
          </execution>
        </executions>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>src/test/resources/master.xml</suiteXmlFile>
          </suiteXmlFiles>
          <properties>
            <property>
              <name>suitethreadpoolsize</name>
              <value>${threads}</value>
            </property>
          </properties>
        </configuration>
      </plugin>

Execution output

[INFO] Running TestSuite
...
... TestNG 7.1.0 by C茅dric Beust ([email protected])
...

Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 14
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 13
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.515 s - in TestSuite

The below attached project should demonstrate the parallelism in action.
github-1686.zip

If this is still a problem, please create a new issue and attach a sample project that can be used to reproduce the problem.

Was this page helpful?
0 / 5 - 0 ratings