Scala-dev: Assess `-Xsource:3` using community build

[SethTisue]

This came up with @smarter at https://gitter.im/typelevel/general?at=6086d87e81866c680c48df4f

I've recently added a bunch of Scala 3 syntax under that flag to allow us to deprecate and potentially repurpose some old syntax without harming cross-compatibility with Scala 2, but this hinges on people actually using that flag so it'd be good to know if there's any issues with it

We should at least try it and see what happens.

[SethTisue]

@smarter the behemoths are busy today with the sbt 1.5.1 upgrade, but I should be able to do this in the next few days

[SethTisue]

run in progress at https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2703/

[SethTisue]

42 green repos, but also lots of failures, most notably scalatest,kind-projector,utest,shapeless,scalatest-3-0; transitively, those prevent most of the build from running

sampling of failure causes:

• scalatest: "method without a parameter list overrides a method with a single empty one"
• kind-projector: "Either[Int, _] takes no type parameters, expected: 1", I suppose because it's using the old wildcard syntax
• utest: "symbol literal is unsupported; use Symbol("hello") instead"
• shapeless: "Malformed literal or standard type 'a"

fixes for these could easily be submitted upstream, and then we could re-run and get more results

this is one of those times where we don't primarily want a community build that rewires all the dependencies; we would primarily want to build each repo separately

it would not be super hard for me (or somebody) to write a script that glues all the proj/*.conf files together in an alternate why where each repo is built in its own space, with check-missing: false so that all dependencies would be retrieved from Maven Central et al. at least, I don't _think_ I would run into a bunch of weird trouble doing that

on the other hand, I don't have any further time for this either this week or next, but I could probably return to it after that

[smarter]

fixes for these could easily be submitted upstream

I'll see if I can look into it. I think the compiler could also stop emitting errors on symbol sytnax since it's still supported in Scala 3 under a language feature. But for kind-projector at least, the error likely comes from a test exercising the ? syntax since it's still supported even if it's deprecated, and upstream might not want to get rid of it right now.

on the other hand, I don't have any further time for this either this week or next, but I could probably return to it after that

That would be great! No pressure though :).

[smarter]

• https://github.com/scala/scala/pull/9602
• https://github.com/scalatest/scalatest/pull/2020
• https://github.com/typelevel/kind-projector/pull/186

[smarter]

@SethTisue do you think you could try running this again now that all the known issues are fixed?

[SethTisue]

@smarter sure, after https://github.com/scala/community-build/pull/1418 lands

[SethTisue]

queued: https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2787/

[smarter]

https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2787/artifact/logs/kind-projector-build.log

[kind-projector] [error] /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/project-builds/kind-projector-b0a85b5f7bb05e14140d78d7b4b2515b9a3ab274/src/test/scala/test.scala:17:7: Either[Int, _] takes no type parameters, expected: 1
[kind-projector] [error]   bar[Either[Int, ?]]
[kind-projector] [error]       ^

Looks like kind-projector isn't up-to-date in the community build since that doesn't match the current sources: https://github.com/typelevel/kind-projector/blob/main/src/test/scala/test.scala#L17

[SethTisue]

~https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2861/~
https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2865/

[smarter]

Some observations:

• Most failures are due to existing warnings turning into errors
• specs2 is failing because it's still using the old ? kind-projector syntax, I'll try to make a PR to fix that.
• For some reason, the * - test syntax from utest doesn't work anymore, but it's deprecated anyway.
• quicklens fails with seemingly weird errors:

[error] /home/smarter/opt/quicklens/quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala:125:38: method apply in object LensHelper cannot be accessed as a member of object com.softwaremill.quicklens.package.LensHelper from package object quicklens in package quicklens
[error] error after rewriting to `package`.this.LensHelper.<apply: error>
[error] possible cause: maybe a wrong Dynamic method signature?
[error]   def modifyLens[T]: LensHelper[T] = LensHelper[T]()
[error]                                      ^
[error] /home/smarter/opt/quicklens/quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala:127:46: method apply in object MultiLensHelper cannot be accessed as a member of object com.softwaremill.quicklens.package.MultiLensHelper from package object quicklens in package quicklens
[error] error after rewriting to `package`.this.MultiLensHelper.<apply: error>
[error] possible cause: maybe a wrong Dynamic method signature?
[error]   def modifyAllLens[T]: MultiLensHelper[T] = MultiLensHelper[T]()
[error]                                              ^
[error] two errors found

Which are actually caused by the private on the definition of LensHelper:

case class LensHelper[T] private () {

which now also implies that the apply method is private. This can be fixed by explicitly declaring these methods as public:

diff --git quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala
index 9675d9d..ff647d3 100644
--- quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala
+++ quicklens/src/main/scala-2.13+/com.softwaremill.quicklens/package.scala
@@ -127,14 +127,19 @@ package object quicklens {
   def modifyAllLens[T]: MultiLensHelper[T] = MultiLensHelper[T]()

   case class LensHelper[T] private () {
-
     def apply[U](path: T => U): PathLazyModify[T, U] = macro QuicklensMacros.modifyLazy_impl[T, U]
   }
+  object LensHelper {
+    def apply[T](): LensHelper[T] = new LensHelper[T]
+  }

   case class MultiLensHelper[T] private () {

     def apply[U](path1: T => U, paths: (T => U)*): PathLazyModify[T, U] = macro QuicklensMacros.modifyLazyAll_impl[T, U]
   }
+  object MultiLensHelper {
+    def apply[T](): MultiLensHelper[T] = new MultiLensHelper[T]
+  }

   case class PathLazyModify[T, U](doModify: (T, U => U) => T) {

So overall this is looking good (except the specs2 and utest failures prevent a lot of projects from compiling).

[smarter]

https://github.com/etorreborre/specs2/pull/953

[smarter]

https://github.com/com-lihaoyi/utest/pull/253

[SethTisue]

https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2881/

[SethTisue]

new run with fixes for specs2, fastparse, fastparse-scalameta, scalameta:

https://scala-ci.typesafe.com/job/scala-2.13.x-jdk11-integrate-community-build/2887/

[SethTisue]

run 2887 shows: BLOCKING DOWNSTREAM: scalatest-3-0 (62), spray-json (53), wartremover (53), json4s (47), scodec-bits (45), fansi (22), akka-stream (21), acyclic (19), scalatags (11), http4s-parboiled2 (9), spire (7), scalamock (2), hasher (1), lift-json (1), decline (1), mainargs (1), tut (1)

for scalatest-3-0 we could just use a JAR from Maven Central instead

spray-json, wartremover, scodec-bits look to me like they probably just need trivial fixes

json4s situation isn't clear to me at a glance

[smarter]

[json4s] [error] def ~>*[B >: A, X2 >: X](f: => Rule[S, S, B => B, X2]) = for (a <- rule; fs <- f*) yield fs.foldLeft[B](a) { (b, f) => f(b) }

That f* is a postfix application of a * operator, which conflicts with the new varargs syntax, looks like this was recently fixed upstream: https://github.com/json4s/json4s/commit/b91c3ff0fce98cb17f2cd0fb2dc330760eeb4442

[SethTisue]

@smarter after https://github.com/scala/community-build/issues/1456 is done, doing this sort of experiment will become easier

[som-snytt]

Can't wait until scala build is -Xsource:3 modulo early definitions.