Bloop: Scalatest argument `-h` does not work with test

[shwin]

Bloop test doesn't seem to like this ScalaTest option (eg in a build.sbt):

testOptions in Test ++= Seq(
    Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/test-reports")
  )

Steps to reproduce:

$ git clone [email protected]:shwin/ashwin-hello.git
$ sbt bloopInstall
$ bloop test root

This is the output:

Compiling 1 Scala source to /Users/ashwinraja/test/ashwin-hello/.bloop/root/scala-2.12/classes ...
Done compiling.
Compiling 1 Scala source to /Users/ashwinraja/test/ashwin-hello/.bloop/root/scala-2.12/test-classes ...
Done compiling.
[E] Unexpected error during remote test execution: 'Socket closed'.
[E] Internal error when running tests: sbt.ForkMain$Run$RunAborted: java.net.SocketException: Broken pipe (Write failed)
===============================================
Total duration: 0ms
No test suites were run.
===============================================

But sbt test works just fine.

Bloop test works with this build.sbt:

testOptions in Test ++= Seq(
  )

$ sbt bloopInstall
$ bloop test root
HelloSpec:
The Hello object
- should say hello (30 milliseconds)
Execution took 30ms.
1 tests, 1 passed
All tests in example.HelloSpec passed.

===============================================
Total duration: 30ms
All 1 test suites passed.
===============================================

Let me know if I need to provide anything else!

[jvican]

@shwin Thank you for this reproduction.

The "Socket closed" exception is likely to happen because of an exception is thrown by the scalatest framework. My best guess to explain this behavior is that "target/test-reports" is not an absolute path and the framework cannot convert it to an existing directory. Therefore, I propose replacing its value in the sbt build by an absolute path and creating that directory to make sure it exists. If this works, then we have our culprit.

This is what I think we need to fix the issue (assuming my previous analysis does explain the bug):

1. Figure out a way to print the exception stack trace of any test framework error, so that next time an error happens we know why directly.
2. Check if the -h option is passed to a scalatest framework.
   2.1. If it's passed and it's an absolute path, let it as is.
   2.2. If it's passed and it's a relative path, I propose to project.out.resolve("test-reports") and print a warning telling the user:
   
   
   
   • The new location where test reports are stored.
   
   
   • To use an absolute path in the configuration of its build tool if the user really wants to modify the place where test reports are stored (and remove the warning).
   
   

Does this make sense?

[shwin]

@jvican I think that makes sense; the only thing I'm a little unclear on, is why sbt test works?

That is, if scalatest is throwing an exception, why does it happen only in bloop and not when you do a regular sbt test, which populates that directory (even if it doesn't exist, etc) just fine?

The relative path/directory not existing does seem to be the issue; if I change it to an absolute path and the directory exists, bloop works fine; just strange that sbt doesn't care :)

[jvican]

The relative path/directory not existing does seem to be the issue; if I change it to an absolute path and the directory exists, bloop works fine; just strange that sbt doesn't care :)

It's very likely that sbt is making sure it's absolute and/or creating the directory in case it doesn't exist :smile:

[jvican]

@shwin Are you interested in taking a stab at this?

[tpolecat]

I'm seeing this again with the -l option passed to ScalaTest, which is used to exclude tests with the given tag. In the bloop config this shows up under test/options/arguments/ as

"args" : [
  "-l",
   "gem.test.Tags.RequiresNetwork"
],

Thanks @jvican for pointing me at this issue. It's not urgent, I can just remove those lines from the config and I'm fine.