Scalatest: Incomplete output with AsyncWordSpec

[el-dom]

I'm using scalatest 3.0.7 on scala 2.12.8 and sbt 1.2.8

I have six testsuites in total with four of them using AsyncWordSpec and the other two using WordSpec and FreeSpec

The async tests look like this example

class FederalStatesRepositorySpec extends AsyncWordSpec {

  "FederalStatesRepository" should {

    "read entry by id from database" in withDatabase { repo =>
      val resById = repo.get(2)

      resById.map(res => assert(res.isDefined && res.get.code == "DE-BW"))
    }

    "read entry by code from database" in withDatabase { repo =>
      val resByCode = repo.getByCode("DE-HB")

      resByCode.map(res => assert(res.isDefined))
    }
}

I've set fork in Test := false and logBuffered in Test := false in build.sbt

When executing sbt test, all six suites are always executed, but not all of them generate output in sbt

E.g. first test execution looks like this

[info] PublicHolidaysRepositorySpec:
[info] PublicHolidaysRepository
[info] TimeslotsRepositorySpec:
[info] TimeslotsRepository
[info] - should read entry by id from database
[info] - should read entry by id from database
[info] - should read public holiday by date and federal state
[info] - should read non-existant entry from database
[info] - should read non-existant entry from database
[info] - should read all entries from database
[info] - should read all entries from database
[info] - should write entry to database
[info] - should write entry to database
[info] - should delete entry from database
[info] SlotRoutesSpec:
[info] SlotRoutes
[info] - should return NoContent for POST requests to calculate route
[info] - should return NoContent for PUT requests to reserve route
[info] ApiRoutesSpec:
[info] The ApiRoutes
[info]   when requested the first route
[info]   - should respond with 'hello!'
[info]   when requested the second route
[info]   - should respond with 'world!'
[info]   when requested an invalid route
[info]   - should respond with 404
[info]   when route fails
[info]   - should respond with 500
[info] Run completed in 4 seconds, 13 milliseconds.
[info] Total number of tests run: 23
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 23, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

and second execution like this

[info] FederalStatesRepositorySpec:
[info] FederalStatesRepository
[info] - should read entry by id from database
[info] - should read entry by code from database
[info] - should read non-existant entry from database
[info] PublicHolidaysRepositorySpec:
[info] PublicHolidaysRepository
[info] - should read entry by id from database
[info] - should read all entries from database
[info] - should read public holiday by date and federal state
[info] - should write entry to database
[info] - should read non-existant entry from database
[info] - should read all entries from database
[info] - should write entry to database
[info] SlotRoutesSpec:
[info] SlotRoutes
[info] - should return NoContent for POST requests to calculate route
[info] - should return NoContent for PUT requests to reserve route
[info] ApiRoutesSpec:
[info] The ApiRoutes
[info]   when requested the first route
[info]   - should respond with 'hello!'
[info]   when requested the second route
[info]   - should respond with 'world!'
[info]   when requested an invalid route
[info]   - should respond with 404
[info]   when route fails
[info]   - should respond with 500
[info] TimeslotsRepositorySpec:
[info] TimeslotsRepository
[info] - should read entry by id from database
[info] - should read non-existant entry from database
[info] - should read all entries from database
[info] - should write entry to database
[info] - should delete entry from database
[info] Run completed in 2 seconds, 723 milliseconds.
[info] Total number of tests run: 23
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 23, failed 0, canceled 0, ignored 0, pending 0

So according to the summary, all my tests are executed, but the output i see is completely arbitrary. Since the blocking tests are always shown, i figure it has something to do with the AsyncWordSpec
Is there a way to have consistent output with async specs?

[KarelCemus]

Any progress or workaround to this issue?

[johnduffell]

I think it's an SBT bug, not scalatest
https://github.com/sbt/sbt/issues/5245

[johnduffell]

I think there are two bugs, I have narrowed down the scalatest bug to that AsyncEngine doesn't increment its completedTestCount when an ignored test is encountered. I am trying to raise a PR now. That fixes the output if you pass in -eU to scalatest.
However I think there is another bug that causes the arbitrary output, it could be SBT or scalatest has that bug.

[johnduffell]

Actually, it is a scalatest bug, I think this PR should fix it: https://github.com/scalatest/scalatest/pull/1843
(and for 3.2.x https://github.com/scalatest/scalatest/pull/1844 )

The bug I was fixing earlier was only affecting ignored tests (and was the most annoying for us), this one affects any ASync tests run from SBT.
The pass/fail status of the test run overall is not affected by either of these bugs.

[ThijsBroersen]

This issue is bothering me already for quite some time and the issue looks stale. Is the solution complicated? Any way I can contribute? Any leads on how to solve this issue?

[johnduffell]

@ThijsBroersen not sure if you saw that I made a PR to fix it yesterday https://github.com/scalatest/scalatest/pull/1843

[ThijsBroersen]

@johnduffell nice! Somehow I did not read your comments, they were not shown (or I did not refresh/cache correct). To my screen I responded to @KarelCemus comment from 14th of July 2019.