✕
Skip to content

[Regression] TestNG extension onStart/onFinish called too many times #3462

New issue
New issue
Closed
Bug
#3470
Closed
[Regression] TestNG extension onStart/onFinish called too many times#3462
Bug
#3470
Assignees
olamy
Milestone
3.6.1

Description

@lprimak
lprimak
opened on Sep 7, 2026

Affected version

3.6.0

Bug description

Starting with 3.6.0, TestNG extension onStart() and onFinish() methods get called too many times.
Previous plugin versions would only call those methods one time.
There is one more caveat to this. If only JUnit Jupiter exist (No TestNG tests), but TestNG is on the Classpath,
TestNG extension should not be invoked at all. This is how previous versions would work.
Here, onStart() gets called when there are no tests, which was breaking one of my builds.

Reproducer: https://github.com/lprimak/maven-test-compiler-reproducer
./mvnw package -Dexception.from.testng=false

Messages message:

[INFO] --- surefire:3.6.0:test (default-test) @ maven-testng-reproducer ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
[INFO] Running com.flowlogix.maven.jupiter.JupiterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 s -- in com.flowlogix.maven.jupiter.JupiterTest
MyTestNGExtension: onStart called for suite: Command line suite
[INFO] Running com.flowlogix.maven.testng.NextGenTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.flowlogix.maven.testng.NextGenTest
MyTestNGExtension: onFinish called for suite: Command line suite
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]

The methods should only be called one time, as with the previous plugin:
./mvnw package -Pprevious-testng

[INFO] --- surefire:3.5.6:test (default-test) @ maven-testng-reproducer ---
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
MyTestNGExtension: onStart called for suite: Surefire suite
MyTestNGExtension: onFinish called for suite: Surefire suite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.141 s -- in TestSuite

No TestNG extension's methods should be called, as with the previous plugin, since no TestNG tests get found or executed:
./mvnw package -Pprevious-plugins

[INFO] --- surefire:3.5.6:test (default-test) @ maven-testng-reproducer ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.flowlogix.maven.jupiter.JupiterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 s -- in com.flowlogix.maven.jupiter.JupiterTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

The issue is reproducible via maven 3.9.16 and 4-rc-6

Activity

  1. Imagelprimak
    added
    bugSomething isn't working
    on Sep 7, 2026
  2. Imageolamy
    self-assigned this
    on Sep 10, 2026
  3. Imageolamy
    added this to the 3.6.1 milestone on Sep 10, 2026
  4. Imageolamy
    added the
    Bug
    issue type on Sep 10, 2026

  5. mpkorstanje commented on Sep 11, 2026

    @mpkorstanje
    mpkorstanje
    on Sep 11, 2026
    Contributor

    Since 3.6.0-M1 Surefire uses the junit platform runner to run junit 3, junit 4 and TestNG tests via vintage engine and testng engine (#3179).

    The testng engine executes testng with a dry run to discover tests. So your ISuiteListener implementation should be invoked more than once. Up to two times for discovery and once for execution.

    Though I wouldn't expect it to be invoked 3 times prior to execution.

    @olamy does Surefire do discovery one test class/method at a time? I haven't looked at the code in a while.

  6. olamy commented on Sep 11, 2026

    @olamy
    olamy
    on Sep 11, 2026
    Member

    @mpkorstanje, yes, that's the problem. I have been working on fix already and will have a PR soon

  7. Imageolamy
    linked a pull request that will close this issueIssue #3462 TestNG extension onStart/onFinish called too many times #3470on Sep 11, 2026

  8. olamy commented on Sep 11, 2026

    @olamy
    olamy
    on Sep 11, 2026
    Member

    possible fix here #3470

  9. lprimak commented on Sep 11, 2026

    @lprimak
    lprimak
    on Sep 11, 2026
    Author

    I put comments into the PR.

  10. javydreamercsw commented on Sep 30, 2026

    @javydreamercsw
    javydreamercsw
    on Sep 30, 2026
    Contributor

    Came to report the same sort of issues. This release has been a nightmare. I'm not sure why this was not considered as a major release since it is not backwards compatible.

  11. olamy commented on Oct 10, 2026

    @olamy
    olamy
    on Oct 10, 2026
    Member

    10. Came to report the same sort of issues. This release has been a nightmare. I'm not sure why this was not considered as a major release since it is not backwards compatible.

    see comment here #3470 (comment) and the comment above #3462 (comment)

  12. Imageolamy
    removed
    bugSomething isn't working
    on Oct 10, 2026
  13. Imageolamy
    added a commit that references this issue on Oct 10, 2026

    Issue apache#3462 Skip the TestNG engine when scanning test classes

    00e4bcf
  14. Imageolamy
    closed this as completedin #3470on Oct 11, 2026
  15. Imageolamy
    added a commit that references this issue on Oct 11, 2026

    Issue #3462 TestNG extension onStart/onFinish called too many times (#…

    a35e23b

  16. olamy commented on Nov 8, 2026

    @olamy
    olamy
    on Nov 8, 2026
    Member

    #3572 reverts the scan change from #3470 because it broke #3548. TestNG listeners get called once per scanned class again during the scan, like in 3.6.0.

    To avoid it, skip the work in your listener when testng.mode.dryrun is true:

    if (Boolean.getBoolean("testng.mode.dryrun")) {
    return;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions