<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.10</version>
<scope>test</scope>
</dependency>
When running a suite of suites via testing.xml I expect each class within a suite to run in parallel.
Each suite runs in parallel but classes within the suite run sequentially:
suitea testcase 1 and suite b testcase2 run in parallel
suitea testcase3 runs after testcase1 completes
IntelliJ
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
<suite-files>
<suite-file path="suitea.xml" />
<suite-file path="suiteb.xml" />
</suite-files>
</suite> <!-- Suite -->
------suitea.xml--------------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
<test name="FirstTest" thread-count="5" parallel="methods" >
<parameter name="browser" value="chrome"/>
<parameter name = "title" value="SS1 Verify the login section"/>
<classes>
<class name="testsuite.TestCase1"/>
</classes>
</test> <!-- First Test -->
<test name="SecondTest" thread-count="5" parallel="methods" >
<parameter name="browser" value="chrome"/>
<parameter name = "title" value="SS2 Second Login Test"/>
<classes>
<class name="testsuite.TestCase3"/>
</classes>
</test> <!-- Second Test -->
</suite> <!-- Suite -->
------suiteb.xml--------------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
<test name="SecondTest" thread-count="5" parallel="methods" >
<parameter name="browser" value="chrome"/>
<parameter name = "title" value="SR1 Verify item in mobile list page can be sorted by name"/>
<classes>
<class name="testsuite.TestCase2"/>
</classes>
</test> <!-- First Test -->
</suite> <!-- Suite -->
@petermartin2002 - Can you please retry using the latest released version of TestNG 6.14.2
and let us know if the issue exists ? 6.9.10
is a very old version.
Upgraded and still the same issue
@petermartin2002 - Can you please help share a sample that can be used to recreate this issue ?
Test cases, pom.xml, and testng, and suite xml attached.
It is really straightforward to recreate from that.
Any advice here folks?
When I run the suites on their own they run in parallel but the suite of suites runs each suite and its subsequent tests serially. Quite frustrating
@petermartin2002 - The attached project contains references to some classes that aren't included. So its not possible to reproduce the issue.
I would request you to please attach a simple test project (without any page objects or selenium dependency), that can be used to reproduce the issue.
im still encountering this parallel testing problem when i run it two actual android device in v.6.14.3. any idea how can i resolve this. thanks in advance :)
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Automation Suite POC 2" parallel="tests" thread-count="2">
<test name="Test : TOMini 121">
<parameter name="Port" value="4723"/>
<parameter name="UDID" value="192.168.8.121:5555"/>
<classes>
<class name="com.xxx.tob.settings.WatchInSettingsTest"/>
</classes>
</test>
<test name="Test : TOMini 131">
<parameter name="Port" value="4725"/>
<parameter name="UDID" value="192.168.8.131:5555"/>
<classes>
<class name="com.xxx.tob.settings.WatchInSettingsTest"/>
</classes>
</test>
</suite>
--pom..xml
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.14.3</version>
<scope>test</scope>
</dependency>
@glenn04 - Can you please help create a simple standalone project (which uses nothing outside of core java) that we can use to reproduce the problem ?
Hello,
@glenn04 and @petermartin2002 Have you fix your problem ? Because I have exactly the same issue... Please here is a simple project without other dependencies. Ready to use !
issues1686.zip
To execute :
mvn clean verify -DsuiteXmlFile=myTestng.xml
Thank you :)
@albanlorillard not yet bro, no answer yet from @krmahadevan ?
@glenn04 - I will get to this as soon as possible. A bit swamped by stuff.
@albanlorillard - Thanks for providing a reproducible test. I will try using that to see what is going on.
Hello @krmahadevan ,
Please consider below sample code.
Testng.xml
pom.xml
Also my all test run in parallel.
Out Put :
Product Added
User Login
Product Edited
Register User
Delete Product
Forgot Password
===============================================
Suite
Did any one found any solution ?
if you make yours testng.xml as shown below then your test will execute in proper manner.
Share your experience or thoughts
@VishnuGupta16
@DhavalAtri But it is not suite of suites
The question was about suite of suites parallel run
@krmahadevan Any updates on this?
I am using cucumber testng and have the same problem with suite of suites (that listed there suited runs one after another, not in parallel)
Have tried with 'cucumber-testng', version: '1.2.5' & 'cucumber-testng', version: '1.2.4'
@tarasmytlovych - No I am yet to get to this issue. Is this a problem when using TestNG 7.0.0-beta3
(latest released version as of today) ?
I have tried reproducing this issue with TestNG 7.0.0-beta3
and I can't seem to recreate it
@tarasmytlovych - Unless you have a reproducible test that can still reproduce the problem, I may have to close this issue.
Here's what I have (Its the same example that @petermartin2002 had shared when the bug was created. I just cleaned it up and added thread id printing)
package com.rationaleemotions.github.issue1686;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
public class TestCase1 {
@BeforeMethod
public void beforeMethod() {
System.err.println(getPrefix() + getClass().getName() + " triggered beforeMethod() on #" + Thread.currentThread().getId());
}
@Test(priority = 1, description = "SS1 Verify the login section")
public void login() {
System.err.println(getPrefix() + getClass().getName() + " triggered login() on #" + Thread.currentThread().getId());
}
@AfterMethod
@Parameters("title")
public void validate(String title) {
System.err.println(getPrefix() + getClass().getName() + " triggered validate() with parameter [" + title + "] on #" + Thread.currentThread().getId());
}
private static String getPrefix() {
ITestResult result = Reporter.getCurrentTestResult();
String suiteName = result.getTestContext().getSuite().getName();
String testTagName = result.getTestContext().getName();
return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
}
}
package com.rationaleemotions.github.issue1686;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class TestCase2 {
@BeforeMethod
public void beforeMethod() {
System.err.println(getPrefix() + getClass().getName() + " triggered beforeMethod() on #" + Thread.currentThread().getId());
}
@Test(priority = 1, description = "SR1 Verify item in mobile list page can be sorted by name")
public void SR1() {
System.err.println(getPrefix() + getClass().getName() + " triggered SR1() on #" + Thread.currentThread().getId());
}
@AfterMethod
public void validate(){
System.err.println(getPrefix() + getClass().getName() + " triggered validate() on #" + Thread.currentThread().getId());
}
private static String getPrefix() {
ITestResult result = Reporter.getCurrentTestResult();
String suiteName = result.getTestContext().getSuite().getName();
String testTagName = result.getTestContext().getName();
return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
}
}
package com.rationaleemotions.github.issue1686;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
public class TestCase3 {
@BeforeMethod
public void beforeMethod() {
System.err.println(
getPrefix()
+ getClass().getName()
+ " triggered beforeMethod() on #"
+ Thread.currentThread().getId());
}
@Test(priority = 1, description = "SS1 Verify the login section")
public void login() {
System.err.println(
getPrefix()
+ getClass().getName()
+ " triggered login() on #"
+ Thread.currentThread().getId());
}
@AfterMethod
@Parameters("title")
public void validate(String title) {
System.err.println(
getPrefix()
+ getClass().getName()
+ " triggered validate() with parameter ["
+ title
+ "] on #"
+ Thread.currentThread().getId());
}
private static String getPrefix() {
ITestResult result = Reporter.getCurrentTestResult();
String suiteName = result.getTestContext().getSuite().getName();
String testTagName = result.getTestContext().getName();
return "[Suite :<" + suiteName + ">, Test: <" + testTagName + "> ";
}
}
Suites look like below
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
<test name="FirstTest" thread-count="5" parallel="methods">
<parameter name="browser" value="chrome"/>
<parameter name="title" value="SS1 Verify the login section"/>
<classes>
<class name="com.rationaleemotions.github.issue1686.TestCase1"/>
</classes>
</test> <!-- First Test -->
<test name="SecondTest" thread-count="5" parallel="methods">
<parameter name="browser" value="chrome"/>
<parameter name="title" value="SS2 Second Login Test"/>
<classes>
<class name="com.rationaleemotions.github.issue1686.TestCase3"/>
</classes>
</test> <!-- Second Test -->
</suite> <!-- Suite -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests">
<test name="SecondTest" thread-count="5" parallel="methods" >
<parameter name="browser" value="chrome"/>
<parameter name = "title" value="SR1 Verify item in mobile list page can be sorted by name"/>
<classes>
<class name="com.rationaleemotions.github.issue1686.TestCase3"/>
</classes>
</test> <!-- First Test -->
</suite> <!-- Suite -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Suite" parallel="tests" verbose="2">
<suite-files>
<suite-file path="./suitea.xml" />
<suite-file path="./suiteb.xml" />
</suite-files>
</suite> <!-- Suite -->
Execution output
...
... TestNG 7.0.0-beta3 by C茅dric Beust ([email protected])
...
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered beforeMethod() on #13
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered beforeMethod() on #14
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered login() on #14
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered login() on #13
[Suite :<Suite (0)>, Test: <FirstTest> com.rationaleemotions.github.issue1686.TestCase1 triggered validate() with parameter [SS1 Verify the login section] on #14
[Suite :<Suite (0)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered validate() with parameter [SS2 Second Login Test] on #13
===============================================
Suite (0)
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered beforeMethod() on #16
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered login() on #16
[Suite :<Suite (1)>, Test: <SecondTest> com.rationaleemotions.github.issue1686.TestCase3 triggered validate() with parameter [SR1 Verify item in mobile list page can be sorted by name] on #16
===============================================
Suite (1)
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================
===============================================
Suite
Total tests run: 3, Passes: 3, Failures: 0, Skips: 0
===============================================
Hi All:
Have anybody tried a suite of xml suites syntax with something :
<suite thread-count="3" name="Suite" parallel="xml">
?
OR
just
parallel="xml"
Have an xml suite os xml suites and need to find a proper syntax.
Thanks in advance for the advice!
@agoldenberg02
<suite thread-count="3" name="Suite" parallel="xml"> ?
There is nothing like that. The only allowed values are
Haven't seen any response for my earlier question on this and since I am also not able to reproduce the problem. Closing this. If this is still a problem, need a reproducible sample using the latest released version of TestNG (7.0.0-beta7
as of today)
@krmahadevan as far as i see, in your example suites are executed sequentially instead of parallel (first - suitea.xml, then suiteb.xml), but the main issue is how to run suitea and suiteb parallel
Even I am waiting for same issue's solution
This is exactly what I am observing as well, even with version 7.1.0. The suite-file's in a suite-files list are executed in sequence, instead of in parallel.
Here's the suite of suite file
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="github-1686-master-suite" verbose="2">
<suite-files>
<suite-file path="slave-1.xml"/>
<suite-file path="slave-2.xml"/>
</suite-files>
</suite>
And here's how the surefire plugin looks like
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M1</version>
<executions>
<execution>
<phase>test</phase>
</execution>
</executions>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/master.xml</suiteXmlFile>
</suiteXmlFiles>
<properties>
<property>
<name>suitethreadpoolsize</name>
<value>${threads}</value>
</property>
</properties>
</configuration>
</plugin>
Execution output
[INFO] Running TestSuite
...
... TestNG 7.1.0 by C茅dric Beust ([email protected])
...
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 14
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-21>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-11>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase1.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase1.two()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase2.one()> running on Thread : 13
Suite: <github-1686-slave-suite-2>, TestName: <github-1686-test-22>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 14
Suite: <github-1686-slave-suite-1>, TestName: <github-1686-test-12>, MethodName: <com.rationaleemotions.TestCase2.two()> running on Thread : 13
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.515 s - in TestSuite
The below attached project should demonstrate the parallelism in action.
github-1686.zip
If this is still a problem, please create a new issue and attach a sample project that can be used to reproduce the problem.
Most helpful comment
@krmahadevan as far as i see, in your example suites are executed sequentially instead of parallel (first - suitea.xml, then suiteb.xml), but the main issue is how to run suitea and suiteb parallel