Scala-dev: Windows jobs on Jenkins hang on sbt startup since sbt 1.3 upgrade

[SethTisue]

both 2.12.x and 2.13.x are affected, so we should fix on 2.12.x and merge the fix forwards.

• 2.12: https://github.com/scala/scala/pull/8660, https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/
• 2.13: https://github.com/scala/scala/pull/8525, https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.13.x-integrate-windows/

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1642/console

++ SBT='java -Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4 -Duser.home=y:/jenkins -Djava.io.tmpdir=y:/jenkins/tmp -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native -Dsbt.ivy.home=y:\jenkins\workspace\scala-2.12.x-integrate-windows/.ivy2 -jar C:\sbt\sbt-launch.jar -Dsbt.override.build.repos=true -Dsbt.repository.config=y:\jenkins\workspace\scala-2.12.x-integrate-windows/scripts/sbt-repositories-config'
++ java -Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4 -Duser.home=y:/jenkins -Djava.io.tmpdir=y:/jenkins/tmp -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native '-Dsbt.ivy.home=y:\jenkins\workspace\scala-2.12.x-integrate-windows/.ivy2' -jar 'C:\sbt\sbt-launch.jar' -Dsbt.override.build.repos=true '-Dsbt.repository.config=y:\jenkins\workspace\scala-2.12.x-integrate-windows/scripts/sbt-repositories-config' -warn setupPublishCore generateBuildCharacterPropertiesFile publishLocal
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? Build timed out (after 200 minutes). Marking the build as aborted.
Build was aborted

as originally reported a couple weeks ago here, but we merged the upgrades anyway, which is fine. this has now worked its way to the top of my priorities

the jobs run scripts/jobs/integrate/windows — one place to start would be to add -debug to the sbt invocation (on a branch) to get more info about where it's getting stuck

[SethTisue]

😱

[SethTisue]

running with -debug isn't helping: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1643/console

the command line invocation is java ... -jar 'C:\sbt\sbt-launch.jar' ..., one possibility is that the sbt launch jar is simply too old. perhaps we should try upgrading that before considering other hypotheses

doc on how to ssh to the Windows worker is at https://github.com/scala/scala-jenkins-infra/blob/master/doc/client-setup.md#windows-setup , it's old but probably still works

once I'm in I could try replacing the launch jar; I could also run jstack to see where sbt is getting stuck

[SethTisue]

@soronpo notes that this might be https://github.com/sbt/sbt/issues/5128 /cc @eed3si9n

[OlegYch]

someone should really fix that lib to not use powershell, or remove it altogether

[SethTisue]

I haven't had any luck sshing to the Windows worker — I suspect something changed the last time Adriaan redid stuff (chef->ansible)

I see over on the sbt ticket that setting -Dsbt.coursier.home might be a workaround, trying that next

(the things I'm trying are commits to https://github.com/SethTisue/scala/commits/sd-666)

[SethTisue]

okay, now at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1644/console we see:

[error] Caused by: java.io.IOException: Cannot run program "powershell.exe": CreateProcess error=2, The system cannot find the file specified
[error]     at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
[error]     at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.runCommands(Util.java:176)
[error]     at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.getWinDirs(Util.java:147)
[error]     at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.getWinDirs(Util.java:120)
[error]     at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.ProjectDirectories.fromPath(ProjectDirectories.java:221)
[error]     at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.ProjectDirectories.from(ProjectDirectories.java:274)
[error]     at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.coursierDirectories(CoursierPaths.java:82)
[error]     at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.computeCacheDirectory(CoursierPaths.java:44)
[error]     at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.cacheDirectory(CoursierPaths.java:70)
[error]     at lmcoursier.internal.shaded.coursier.paths.CachePath.defaultCacheDirectory(CachePath.java:93)
[error]     at lmcoursier.internal.shaded.coursier.cache.CacheDefaults$.location$lzycompute(CacheDefaults.scala:16)
[error]     at lmcoursier.internal.shaded.coursier.cache.CacheDefaults$.location(CacheDefaults.scala:16)
[error]     at lmcoursier.internal.shaded.coursier.cache.FileCache$.apply(FileCache.scala:1088)
[error]     at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:134)
[error]     at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60)

this appears to be https://github.com/soc/directories-jvm/issues/29 (to which https://github.com/sbt/sbt/issues/5386 is a reference)

I think the next step here is probably for me to get ssh access to the Windows worker, otherwise the troubleshooting process (pushing changes to scripts/jobs/integrate/windows) is just too cumbersome

[eed3si9n]

I'd be curious to know if powershell can be invoked from ssh as well.

Also setting COURSIER_HOME environmental variable might be a workaround?

[SethTisue]

Also setting COURSIER_HOME environmental variable might be a workaround?

I did that (well, -Dsbt.coursier.home, not COURSIER_HOME, but it's the same, right?)

Before that change, it was hanging. After, we get Cannot run program "powershell.exe"

[som-snytt]

It's Number 666 because it's the Powers of Hell.

[eed3si9n]

https://github.com/coursier/sbt-coursier/blob/460a79b96d80d4e2d4fe315cea1a5a8587221357/modules/lm-coursier/src/main/scala/lmcoursier/CoursierDependencyResolution.scala#L134-L140

    val cache0 = coursier.cache.FileCache()
      .withLocation(cache)
      .withCachePolicies(cachePolicies)
      .withTtl(ttl)
      .withChecksums(checksums)
      .withCredentials(conf.credentials.map(ToCoursier.credentials))
      .withFollowHttpToHttpsRedirections(conf.followHttpToHttpsRedirections.getOrElse(true))

It seems like FileCache() or FileCache$.apply is where it's blowing up so there may be a difference between -Dsbt.coursier.home and COURSIER_HOME.

[alexarchambault]

I confirm that the COURSIER_HOME env var should currently work better than the sbt.coursier.home Java property. That needs to be fixed in coursier. (The problem is that calling FileCache() evaluates some default values, and one of them runs that powershell command at the end, all of that before users have a chance to call withLocation.)

[alexarchambault]

~About the Windows error, I wonder if that can be non-existing directories in PATH. I'm going to try that locally when I can reach my Windows machine.~ Edit: seems fine on my Windows 10 machine.

[dwijnand]

I confirm that the COURSIER_HOME env var should currently work better than the sbt.coursier.home Java property. That needs to be fixed in coursier.

This is a minor aside, but it would be good to finishing understanding this. Also the coursier.cache sys prop, mentioned in https://github.com/sbt/sbt/issues/5128.

[alexarchambault]

That setting in your windows worker config is worrying. It seems it clears the PATH, and only puts custom directories there. (Which could cause the Cannot run program "powershell.exe".)

[SethTisue]

@alexarchambault good catch! at https://scala-ci.typesafe.com/computer/jenkins-worker-windows-publish/configure I added /cygdrive/c/Windows/System32/WindowsPowershell/v1.0 to the PATH setting (based on some random googling of where to find PowerShell on Windows Server 2012), and now https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1649/ seems to be proceeding

in the script, I will set both COURSIER_HOME and sbt.coursier.home, just in case, but I'll also comment it with a link back to this discussion.

so this seems resolved; leaving the ticket open for now until I finish PRing the change, testing it on both 2.12.x and 2.13.x, and so on.

[dwijnand]

at scala-ci.typesafe.com/computer/jenkins-worker-windows-publish/configure I added /cygdrive/c/Windows/System32/WindowsPowershell/v1.0 to the PATH setting

Can we make that change in scala/scala-jenkins-infra, run it and revert the override?

[SethTisue]

Can we make that change in scala/scala-jenkins-infra, run it and revert the override?

I'm afraid I no longer know how that stuff works. At one time it was Chef-based and I more or less understood it (and I wrote of bunch of additional documentation about it, too), but it all got redone and now I don't know it anymore.

As best I can recall, the last time this came up, Adriaan said something like: go ahead and make the change directly in the Jenkins UI, we persist those changes every so often so they aren't lost. Something like that? I'm not really sure.

I have now made many such changes to the Jenkins jobs configs. (And perhaps a few to the worker configs, too, less sure about that.)

[dwijnand]

OK, let's chat next week on how we can drop test dropping the workaround.