Scalatest: Incorrect test count summary with ScalaTest 3.0.5/Scala.js 0.6.22

[djneades]

I am programmatically generating tests based on data using an overridden nestedSuites. Here is a simplified example:

package com.araxis.muve.web.test

import org.scalatest.FunSuite

class Test extends FunSuite {

  case class TestCase(expectedResult: Boolean, description: String)

  val testCases =
    List(TestCase(true, "Test should be true."), TestCase(false, "Test should be false."))

  override def nestedSuites: Vector[FunSuite] = {
    val suites = for { tc <- testCases } yield
      new FunSuite {
        override def suiteName = s"${tc.description}"

        test("Assert test case is true") {
          assert(tc.expectedResult === true)
        }
      }

    suites.toVector
  }
}

This approach works as expected when the tests are run with the JVM. It also works fine with ScalaTest 3.0.4 and Scala.js 0.6.21. With ScalaTest 3.0.5 and Scala.js 0.6.22, however, the tests run correctly, but the result summary always shows a total of 0 tests run (at least, when using the Node JS environment to run the tests).

Here is the output from running the above test suite under sbt 1.1.1 with ScalaTest 3.0.5 and Scala.js 0.6.22 (note both the incorrect Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0 line and the No tests were executed. message):

[IJ]root> mwJS/testOnly com.araxis.muve.web.test.Test
[info] Compiling 1 Scala source to C:\Hg\MW\js\target\scala-2.12\classes ...
[info] Done compiling.
[info] Fast optimizing C:\Hg\MW\js\target\scala-2.12\mw-test-fastopt.js
[info] Test should be true.:
[info] - Assert test case is true
[info] Test should be false.:
[info] - Assert test case is true *** FAILED ***
[info]   false did not equal true (Test.scala:18)
[info] Run completed in 71 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[error] Failed tests:
[error]         com.araxis.muve.web.test.Test
[error] (mwJS / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 25 s, completed 28 Feb 2018, 16:38:45
[IJ]root>

For completeness, here is the same run, but with ScalaTest 3.0.4 and Scala.js 0.6.21 (note the correct test-count summary and the *** 1 TEST FAILED *** message):

[IJ]root> mwJS/testOnly com.araxis.muve.web.test.Test
[info] Compiling 1 Scala source to C:\Hg\MW\js\target\scala-2.12\classes ...
[info] Done compiling.
[info] Fast optimizing C:\Hg\MW\js\target\scala-2.12\mw-test-fastopt.js
[info] Test should be true.:
[info] - Assert test case is true
[info] Test should be false.:
[info] - Assert test case is true *** FAILED ***
[info]   false did not equal true (Test.scala:18)
[info] Run completed in 11 seconds, 492 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error]         com.araxis.muve.web.test.Test
[error] (mwJS / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 20 s, completed 28 Feb 2018, 16:42:16
[IJ]root>

One further observation. If I add the option below to the .jsSettings for my sbt crossProject, the previously working ScalaTest 3.0.4/Scala.js 0.6.21 test run will fail in exactly the same way as it does with ScalaTest 3.0.5/Scala.js 0.6.22 without this option. Perhaps something has recently changed with ScalaTest or Scala.js in this area?

scalaJSSemantics ~= {
      // don’t leak class names; see http://bit.ly/2HwSOqj
      _.withRuntimeClassNameMapper(RuntimeClassNameMapper.discardAll())
    }

[sumnulu]

Can confirm it.

Scala: 2.12.5
Scalatest: 3.0.5
Scala.js: 0.6.22

Test.scala:

package testingTest
import org.scalatest.AsyncFeatureSpec
class MySpec extends AsyncFeatureSpec {
  feature("Feature 1") {
    scenario("Scenario 1 should fail") {
      fail("Not implemented yet")
    }
    scenario("Scenario 2 should pass") {
      assert(true)
    }
  }
}

Out:

[info] MySpec:
[info] Feature: Feature 1
[info] - Scenario: Scenario 1 should fail *** FAILED ***
[info]   Not implemented yet (MySpec.scala:9)
[info] - Scenario: Scenario 2 should pass
[info] Run completed in 112 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[error] Failed tests:
[error]     testingTest.MySpec
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 9 s, completed Mar 26, 2018 4:42:20 PM
1. Waiting for source changes... (press enter to interrupt)

[djneades]

I have updated the title to reflect the more general nature of the problem, as reported by @sumnulu.

[gzm0]

Starting Scala.js 0.6.22 (or precisely starting at scala-js/scala-js@22a8d7dc52669c537154f21519752a8b6680a174), the test adapter started executing tasks bound to the master runner.

This surfaces a latent bug in ScalaTest's Scala.js runner: Test counts are only ever updated via messages to the master runner. However, if a task is created on the master, notifyServer is set to None and no summary message is sent. As a result, tasks executed on the JS VM hosting the master runner do not contribute to summary count.

[cheeseng]

Hi all, thanks for the investigation, highly appreciated, I am working a fix now.

[cheeseng]

fyi I submitted a PR to fix this problem:

https://github.com/scalatest/scalatest/pull/1371