Testng: ITestListener's methods get called multiple times for one test, when there are multiple tests in the xml

[libertylocked]

Similar to #171, this applies to ITestListener instances.

When there are multiple and the xmls all use the same test listener, onFinish() gets called multiple times for one test, which does not seem intended.

[stardust20]

Hello,

I have a similar issue, my testng.xml looks like the following:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="suite1" verbose="1">
    <listeners>
        <listener class-name="org.testng.reporters.SuiteHTMLReporter"/>
        <listener class-name="org.testng.reporters.TestHTMLReporter"/>
    </listeners>
    <test name="package1">
        <classes>
            <class name="xxx.xxx.xxx.Account"/>
            <class name="xxx.xxx.xxx.Corporate"/>
            <class name="xxx.xxx.xxx.Department"/>
            <class name="xxx.xxx.xxx.Login"/>
            <class name="xxx.xxx.xxx.Subscriber"/>
        </classes>
    </test>
</suite>

I have a Listener which I use for all the classes above, that implements ITestListener. Every time a test is run, I see the following output in the log:

2016-01-25 15:28:22,517 [main] INFO  xxx.xxx.xxx.CustomListener - Start of execution package1
2016-01-25 15:28:22,518 [main] INFO  xxx.xxx.xxx.CustomListener - Start of execution package1
2016-01-25 15:28:22,518 [main] INFO  xxx.xxx.xxx.CustomListener - Start of execution package1
2016-01-25 15:28:22,519 [main] INFO  xxx.xxx.xxx.CustomListener - Start of execution package1
2016-01-25 15:28:22,519 [main] INFO  xxx.xxx.xxx.CustomListener - Start of execution package1
2016-01-25 15:28:26,716 [main] INFO  xxx.xxx.xxx.CustomListener - Test started: A0101_SearchAccountByName
2016-01-25 15:28:26,717 [main] INFO  xxx.xxx.xxx.CustomListener - Test started: A0101_SearchAccountByName
2016-01-25 15:28:26,717 [main] INFO  xxx.xxx.xxx.CustomListener - Test started: A0101_SearchAccountByName
2016-01-25 15:28:26,717 [main] INFO  xxx.xxx.xxx.CustomListener - Test started: A0101_SearchAccountByName
2016-01-25 15:28:26,717 [main] INFO  xxx.xxx.xxx.CustomListener - Test started: A0101_SearchAccountByName

The methods inside CustomListener look like the following:

    @Override
    public void onTestStart(ITestResult iTestResult) {
        log.info("Test started: " + iTestResult.getName());
    }

    @Override
    public void onStart(ITestContext iTestContext) {
        log.info("Start of execution " + iTestContext.getName());
    }

The problem occurs with all the methods inside the listener, i.e. _onTestSuccess_, _onTestFailure_, etc.

Any ideas?

[stevie1877]

I had this very same problem, and got round it by adding a boolean e.g. testStartListenerYetToRun to the base class and setting it to True in the @BeforeMethod (also in the base class), and adding an if condition to the CustomListener e.g.:

@Override
    public void onTestStart(ITestResult iTestResult) {
        if (((AbstractTestNGOrderGridTests)iTestResult.getInstance()).testStartListenerYetToRun) {
            ((AbstractTestNGOrderGridTests) iTestResult.getInstance()).testStartListenerYetToRun = false;
            log.info("Test started: " + iTestResult.getName());
       }
    }

You will need a second boolean for the onStart method.

The methods will of course still be invoked once per class but at least the code will only be executed once.

[krmahadevan]

@LibertyLocked - Is the below sample what you were using as well ?

@stevie1877 - Are you trying to do this on TestNG 6.10 ? If not, can you please try this without your work around and see if you are able to reproduce this ? If you are still able to recreate it, it would be great if you could please help share a sample that I can use to reproduce this issue, so that we can get this fixed.

Here's what I have as test classes and suite xml file that I tried using to recreate the problem [ using TestNG v6.10 ] I am not able to reproduce this problem using the below samples on TestNG 6.10

package org.rationale.emotions.github.issue956;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(ListenerFor956.class)
public class AccountTest {
    @Test
    public void testMethod() {
        System.err.println("accountTest()");
    }
}

package org.rationale.emotions.github.issue956;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners (ListenerFor956.class)
public class CorporateTest {
    @Test
    public void testMethod() {
        System.err.println("corporateTest()");
    }

}

package org.rationale.emotions.github.issue956;

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class ListenerFor956 implements ITestListener {
    public ListenerFor956() {
        System.err.println("New instance created ");
    }

    @Override
    public void onTestStart(ITestResult result) {
        String print = result.getTestClass().getRealClass().getName() + "." + result.getMethod()
            .getConstructorOrMethod().getMethod().getName() + "()";
        System.err.println("Test started: " + print);
    }

    @Override
    public void onTestSuccess(ITestResult result) {}

    @Override
    public void onTestFailure(ITestResult result) {}

    @Override
    public void onTestSkipped(ITestResult result) {}

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {}

    @Override
    public void onStart(ITestContext context) {
        System.err.println("Start of execution <" + context.getName() + ">");
    }

    @Override
    public void onFinish(ITestContext context) {}
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="1265_Suite" parallel="false" verbose="2">
    <test name="956">
        <classes>
            <class name="org.rationale.emotions.github.issue956.AccountTest"/>
            <class name="org.rationale.emotions.github.issue956.CorporateTest"/>
        </classes>
    </test>
</suite>

Here's the output

[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ scrappable-project ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
[XmlSuite] [WARN] 'parallel' value 'false' is deprecated, default value will be used instead: 'none'.
...
... TestNG 6.10 by Cédric Beust ([email protected])
...

New instance created
Start of execution <956>
Test started: org.rationale.emotions.github.issue956.AccountTest.testMethod()
accountTest()
Test started: org.rationale.emotions.github.issue956.CorporateTest.testMethod()
corporateTest()
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.461 sec - in TestSuite

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.752 s
[INFO] Finished at: 2016-12-09T21:48:28+05:30
[INFO] Final Memory: 22M/244M
[INFO] ------------------------------------------------------------------------

ping @juherr (Just keeping your fyi'ed)

[stevie1877]

Thanks @krmahadevan , no I can't replicate the issue using 6.10 - I have been using TestNG 6.1.1, so quite an old version. I will update to 6.10 as this clearly solves the problem. Many thanks.

[krmahadevan]

@juherr - I guess we can close this issue as well.

[juherr]

@krmahadevan Could you check if the current tests are covering this case?

[snehalizaj]

I am experiencing this issue with TestNG 6.11.0

[juherr]

@snehalizaj Could you share a project sample?

[zfolwick]

For what it's worth, I could not duplicate this machine locally on my windows box, but I was able to get a repro for our jenkins machine for testNG 6.10

[krmahadevan]

What about v6.11 or the latest v6.12 ? ( 6.12 is only on Jcenter and is yet
to make it to Maven Central).
Can you please try and let us know? Also pls help share a sample to
recreate the problem.

[manosnoam]

Upgrading TestNG from 6.9 to 6.11 solved it for me. Thanks!

[ils-808]

Hi!

I'm using latest version of TestNG 6.11, but it does not resolve this issue for me.

My xml config looks so:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test all" >
    <listeners>
        <listener class-name="com.selenium.test.testng.listeners.TestListener"/>
    </listeners>
    <test name="CheckoutDesktop">
        <classes>
            <class name="com.selenium.test.testng.tests.CheckoutTestsDesktop"/>
        </classes>
    </test>
    <test name="CheckoutMobile">
        <classes>
            <class name="com.selenium.test.testng.tests.CheckoutTestsMobile"/>
        </classes>
    </test>
</suite>

When I run test in Intellij IDEA all methods of TestListener called two times.

[krmahadevan]

@ils-808 - Do you have a test that we can use to reproduce the problem ?
Also what happens when you run your tests from a command line (or) using a build tool such as Gradle/Maven ?

[ils-808]

@krmahadevan, I found that it was my fault. Version 6.11 works fine, thank you!

[Abhijitdatta]

I am facing the same issue . In my scenario, there is only one testcase in single class. I am using Testing version 7.1.0 and I am facing the issue. I have tried testing version below 6.10 but facing the same. Kindly help.

[krmahadevan]

@Abhijitdatta pls share a sample that can be used to reproduce the problem

[seleniumsb]

TestNG version on eclipseIDE: 7.1.0
I seem to be facing the same issue. I have @beforemethod and @Test which is grouped in positive,negative and smoke tests in my java project file. When running via command prompt (mvn clean test)3 tests are run as expected, but with maven on eclipse , testng results displays 6 tests with 3 ignored using suitexml.
Attached testngreportxml and suitexml (in text file formats)
and screenshot of the report on eclipse.
Logintestsuite.txt
testngreport.txt

Can you please let me know why this seems to be the issue on the testng report.

[krmahadevan]

@seleniumsb - If the problem is only when running tests via Eclipse, then you should be filing an issue on https://github.com/cbeust/testng-eclipse/issues

If this problem can be reproduced even via running from maven command line, then please create a new bug with a full fledged example that can be used to reproduce the problem.

[Abhijitdatta]

@krmahadevan , this is following scenario.

This is the simple Testcase

    @Test
    public void getTitleTest()
    {
        String actualTitle = driver.getTitle();
        System.out.println(actualTitle);
        String expectedTitle = "Free CRM #1 cloud software for any business large or small123";
        Assert.assertEquals(actualTitle, expectedTitle);
    }

Base class scereenshot method

 public void getScreenshot(String testcasename, WebDriver driver) throws IOException
    {

        System.out.println("in screenshot method");
        TakesScreenshot ts = (TakesScreenshot)driver;
        System.out.println("in line1");
        File source = ts.getScreenshotAs(OutputType.FILE);
        System.out.println("in line2");
        String destinationFile = System.getProperty("user.dir")+"\\reports\\"+testcasename+".png";
        FileUtils.copyFile(source, new File (destinationFile));

    }

intentionally expected title is given wrong to capture screenshot.. @Test method is getting failed for two times . for 1st time, its assertion issue and for 2nd time, it's null pointer exception.

[krmahadevan]

@Abhijitdatta - Thats not a sample that can be used to reproduce the problem. Please share a simple standalone complete example (without any 3rd party dependencies such as selenium) with the listener that is getting invoked multiple times, so that we can reproduce the problem. The information shared is NOT sufficient to debug

[kubakostrzewski]

I've reproduced the issue.
Listener:

public class TestEventListener implements ITestListener {

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("Listener: test failed");
        System.out.println("Test class: " + result.getTestClass());
        System.out.println("Test method: " + result.getMethod() + "\n");
    }
}

Two test classes:

@Listeners({TestEventListener.class})
public class FirstTestClass {

    @Test
    public void firstTest(){
        System.out.println("First test");
        fail();
}

@Listeners({TestEventListener.class})
public class SecondTestClass {

    @Test
    public void secondTest(){
        System.out.println("Second test");
        fail();
    }
}

Two tests declared in testng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Selenium UI tests">
    <test name="First test">
        <classes>
            <class name="com.example.FirstTestClass"/>
        </classes>
    </test>
    <test name="Second test">
        <classes>
            <class name="com.example.SecondTestClass"/>
        </classes>
    </test>
</suite>

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>testng-listener-issue</artifactId>
    <packaging>pom</packaging>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.1.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M4</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

When I call maven clean test on the output i can see that after first test listener method was called twice and after second test it was called once:

First test                                                                                                                                                                                        
Listener: test failed                                                                                                                                                                                                                                                           
Test class: [TestClass name=class example.FirstTestClass]                                                                                                                                                                                                                       
Test method: FirstTestClass.firstTest()[pri:0, instance:example.FirstTestClass@40005471]                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
Listener: test failed                                                                                                                                                                                                                                                           
Test class: [TestClass name=class example.FirstTestClass]                                                                                                                                                                                                                       
Test method: FirstTestClass.firstTest()[pri:0, instance:example.FirstTestClass@40005471]                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
Second test                                                                                                                                                                                                                                                                     
Listener: test failed                                                                                                                                                                                                                                                           
Test class: [TestClass name=class example.SecondTestClass]                                                                                                                                                                                                                      
Test method: SecondTestClass.secondTest()[pri:0, instance:example.SecondTestClass@20d525]

When I declare only one test in testng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Selenium UI tests">
    <test name="First test">
        <classes>
            <class name="example.FirstTestClass"/>
        </classes>
    </test>
</suite>

I can see that listener method was called once:

First test                                                                                                                                                                                                                                                                      
Listener: test failed                                                                                                                                                                                                                                                           
Test class: [TestClass name=class example.FirstTestClass]                                                                                                                                                                                                                       
Test method: FirstTestClass.firstTest()[pri:0, instance:example.FirstTestClass@40005471]

[krmahadevan]

This is fixed via PR https://github.com/cbeust/testng/pull/2222 and is available in TestNG 7.3.0 (latest released version)