Scala-dev: provide guidance to project maintainers on including “latest Scala nightly” in their CI matrices

[SethTisue]

we should figure out how to do this and provide a sample .travis.yml that does it. the Scala modules could be a good place to start

[SethTisue]

https://stackoverflow.com/questions/40622878/how-do-i-tell-sbt-to-use-a-nightly-build-of-scala-2-11-2-12-or-2-13 shows how to get the latest nightly in sbt, but doing it in .travis.yml is a different matter

[dwijnand]

I had a little look at how to do this. So I looked for the simplest project I could find that use Travis CI and picked https://github.com/lihaoyi/sourcecode

However it depends on sbt-scalajs, which depends on a fully cross-built version of scalajs-compiler... See https://github.com/scala-js/scala-js/issues/3140 for my conversation with the Scala.js project about this issue.

I expect Scala Native to be similarly difficult (if not worst...).

[sjrd]

In general, for compiler plugins, the only safe(-ish) way to set it up is to clone the source of the relevant compiler plugin and compile/publish it locally, built againt the particular Scala nightly.

This is necessary because scala-compiler.jar does not guarantee any kind of binary compatibility. In fact, it does not even provide any kind of source compat', so even this can fail, but in that case you want to know anyway.

You could try using using the published version of the compiler plugin that is built against the most recent Scala release, but you are open to binary incompatibilities.

[dwijnand]

Using Scala 2.12.4-bin-38628d1 with scalajs-compiler_2.12.3 😢

java.lang.NoSuchMethodError: scala.tools.nsc.backend.jvm.BTypesFromSymbols.initializeCoreBTypes()V
    at org.scalajs.core.compiler.Compat210Component.initializeCoreBTypesCompat(Compat210Component.scala:146)
    at org.scalajs.core.compiler.Compat210Component.initializeCoreBTypesCompat$(Compat210Component.scala:130)
    at org.scalajs.core.compiler.GenJSCode.initializeCoreBTypesCompat(GenJSCode.scala:31)
    at org.scalajs.core.compiler.GenJSCode$JSCodePhase.run(GenJSCode.scala:215)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1432)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1417)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1413)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1516)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:95)
    at xsbt.CompilerInterface.run(CompilerInterface.scala:26)

[dwijnand]

My code snippet:

// When using a nightly build of Scala (i.e when scalaVersion has a "-bin-.." suffix)
// Replace the scalajs-compiler compiler plugin injected by sbt-scalajs
// With the one built for the _previous_ version of Scala
//
// Example:
// When building with Scala 2.12.4-bin-38628d1, which is a nightly build for future Scala 2.12.4
// Replace scalajs-compiler_2.12.4-bin-38628d1 (which doesn't exist)
// with scalajs-compiler_2.12.3
libraryDependencies := {
  /** If scalaVersion has a "-bin-.." suffix, return the previous version.
   *
   *  | scalaVersion       | crossSv        | baseSv         | prevSv         |
   *  |--------------------+----------------+----------------+----------------|
   *  | 2.12.4             | Some("2.12.4") | None           | None           |
   *  | 2.12.4-bin-38628d1 | Some("2.12.4") | Some("2.12.4") | Some("2.12.3") |
   */
  def prevScalaVersion(scalaVersion: String, scalaBinaryVersion: String): Option[String] = {
    val crossVersion = CrossVersion(CrossVersion.patch, scalaVersion, scalaBinaryVersion)
    val crossSv = crossVersion map (fn => fn("") stripPrefix "_")
    val baseSv = crossSv filter (_ != scalaVersion)
    val prevSv = baseSv collect {
      case VersionNumber(Seq(x, y, z), Seq(), Seq()) if z > 0 => s"$x.$y.${z - 1}"
    }
    prevSv
  }

  val prevSv = prevScalaVersion(scalaVersion.value, scalaBinaryVersion.value)

  val libDeps = libraryDependencies.value
  prevSv match {
    case None         => libDeps
    case Some(prevSv) =>
      val scalaJsCompiler =
        compilerPlugin("org.scala-js" % s"scalajs-compiler_$prevSv" % scalaJSVersion)
      libDeps map (m => if (m.name == "scalajs-compiler") scalaJsCompiler else m)
  }
}

[sjrd]

Well, guess what? You found an actual binary incompatibility between 2.12.3 and 2.12.4-bin-whatever, introduced by https://github.com/scala/scala/commit/5f5d52593750d41045e690e661b28d700e52cd0b#diff-96d929f42cdd21128555c2ea328fbf95

Not sure how the community build passes, btw, because it's also not source compatible. Apparently calling that initialize() method is not as critical as was calling initializeCoreBTypes(). @lrytz, any clue?

See also:

• Commit where we start calling that method: https://github.com/scala-js/scala-js/commit/58d9c45c6427abcde0c88f9119678a3c9852d953
• Issue that commit fixed: https://github.com/scala-js/scala-js/issues/2027
• The original issue in scala-dev: https://github.com/scala/scala-dev/issues/64

[lrytz]

Maybe I misunderstand, but I'm surprised this hasn't come up before: the scala-compiler jar is not binary compatible across minor versions.

[sjrd]

The new thing here is that @dwijnand is putting himself in harm's way from bin incompatibility. The community build always builds from source, so only source incompatibilities are a problem. But @dwijnand is playing with fire by linking a compiler plugin built for a different version of the compiler. I had warned against this in my first comment in this thread.

(The reason I pinged you is only mildly related to the main topic of the discussion; I was surprised that the renaming of initializeCoreBTypes() did not cause a source incompatibility, which would have angered the community build.)

[lrytz]

OK, I read the rest of this ticket now, so there's no misunderstanding about binary / source compatibility for scala-compiler :)

Indeed, calling initialize is no longer necessary for things to work, give me a minute to explain that in detail.

[lrytz]

So the backend has quite a bit of state that is derived from the symbol table. Since types of symbols might change between runs, this has to be re-computed in each compiler run. This used to be done in the initializeCoreBTypes method (and some others).

Now we have a bit of an abstraction, state is created through perRunLazy, which creates a LazyVar, which caches the value and keeps a reference to the initializer, so it can be re-invoked. The new initialize method now re-initializes all LazyVars. So calling initialize is no longer necessary to get the initial values computed.

If you're running multiple compiler runs, you should probably call GenBCode.initialize in each run (or parts of it).

[sjrd]

Hum. So IIUC, now Scala.js with 2.12.4 is vulnerable to crashes or inconsistent behavior in the PC?

[dwijnand]

I'm just experimenting the viability of people testing Scala nightly builds in their projects. They would be configured with allow_failures anyways.

[dwijnand]

@sjrd Would the Scala.js project be willing to setup nightly releases of Scala.js against Scala nightlies?

[sjrd]

If someone provides the engineering time to set it up and keep it running, sure. But on my own I can't take that on, I'm afraid.

[lrytz]

Hum. So IIUC, now Scala.js with 2.12.4 is vulnerable to crashes or inconsistent behavior in the PC?

The PC doesn't run until the backend, so there's no issue there. This affects cases where a Global instance remains resident and is re-used for compilation (REPL, our JUnit tests). As far as I know, sbt and partest create a new Global instance each time.

There's no change wrt 2.12.3: the backend needed to be re-initialized in each run as well.

[milessabin]

Bracketing all the other discussion, I think a JVM-only build with the latest Scala nightly would be very useful. I'd be happy to add it to the shapeless build.

[SethTisue]

one new possibility here is for projects to scrape a decently current version out of the new nightly.properties file this repo now provides: https://github.com/scala/community-builds/blob/2.12.x/nightly.properties

[SethTisue]

to make this more concrete, here's what I have in one repo:

resolvers += "scala-integration" at "https://scala-ci.typesafe.com/artifactory/scala-integration/"
def nightly(branch: String): String = {
  val url = s"https://raw.githubusercontent.com/scala/community-builds/$branch/nightly.properties"
  val props = new java.util.Properties
  props.load(new java.net.URL(url).openStream)
  props.getProperty("nightly").ensuring(_ != null)
}
crossScalaVersions := Seq("2.13.0", "2.12.9", "2.11.12",
  nightly("2.13.x"), nightly("2.12.x"), nightly("2.11.x"))
scalaVersion := crossScalaVersions.value.head
publish / skip := scalaVersion.value.containsSlice("-bin-")

but I'm not sure how to write a .travis.yml that works with this, any suggestions?

[dwijnand]

Add the 3 nightly options to the yaml and make it call a wrapper command that does the lookup and calls ++.

Think plz 2.12-nightly test.

[milessabin]

FWIW, I've been leaning quite heavily on the dottyLatestNightlyBuild task while working on shapeless 3. Something like that for Scala 2 would be very helpful.

[SethTisue]

we wanted to do this for Scala.js (context: https://github.com/scala/community-build/issues/1009)

but @sjrd didn't want to modify his main CI matrix since the "latest" aspect would make the builds nonreproducible

so he set up a separate repo https://github.com/scala-js/scala-js-test-scala-nightly — it has a .travis.yml that clones the main Scala.js repo and runs the tests against the Scala version from scala/community-builds/$branch/nightly.properties

this approach may be of interest to other maintainers as well

[milessabin]

I've shamelessly pilfered this for shapeless here.

[SethTisue]

proposal to add this to sbt: https://github.com/sbt/sbt/pull/5464 cc @martijnhoekstra

more relevant to interactive use than to typical CI, but scala-runners offers it, https://github.com/dwijnand/scala-runners

[SethTisue]

One thing I dislike about my advice above is that it forces you to use script: sbt +test (note the +) rather than controlling the Scala version using scala: entries in your matrix. Which is a bit sad because matrix thing is nice (you get a separate job for each version, with its own log) and is pretty standard in Scala OSS.

I guess if you really want the matrix thing, you could rig a thing where instead of controlling the Scala version with scala:, you control it with env: SCALA_VERSION= matrix entries, and then hook that up to something in the sbt build that overrides the Scala version appropriately if the environment variable is set (including to some magic value such as "2.12.x-nightly").

Or just hope https://github.com/sbt/sbt/pull/5464 lands at some point :-)

[SethTisue]

@dwijnand am I imagining it, or did you do a version of this in one of your repos recently? (scala-rewrites?)

[dwijnand]

Yeah: https://github.com/scala/scala-rewrites/pull/35