Coursier: Dependency versions seem to be determined globally instead of specifically per configuration in sbt 1.3.x

[kamarius]

Sbt in versions 1.3.x seems to resolve dependency version conflicts globally rather than per configuration as in earlier versions.

As an example, let's say I do have a dependency defined in compile scope with a specific version and the same dependency occurs in another version in test scope (transitively in my case and in higher version). It seems that even though the test extends compile configuration and not the other way round, the version I observe in compile classpath seems to be the one derieved from the test dependencies.

A small example: https://github.com/kamarius/sbt-conflict-resolution

Steps:

1. sbt "show compile:fullClasspath"

Result:

In sbt versions 1.3.x the kafka-clients library will be resolved to 2.3.0 version (as is defined in tests as transitive dependency from embedded-kafka lib) while sbt 1.2.8 resolves it to 1.0.0 as defined in the compile scope.

Is this expected?

[alexarchambault]

Yes, this is expected. Resolution is done once for configurations that are related (only one resolution is run for compile / runtime / test for example).

[kamarius]

Which configurations are considered related? Do you know if there is any documentation summarizing this behavior? The most surprising part for me was that even though we have a strict policy set up in our build the dependencies were silently promoted in compile scope. And there was no indication either via evicted or any kind of warning in sbt.

Could at least a warning be added when a compile dependencies are changed based on other configuration for which there is no explicit relation?

[eed3si9n]

Let me paste the entire build file here since it's a small one:

name := "sbt-dependency-resolution"
version := "0.1"
scalaVersion := "2.11.12"
val kafkaClients = "org.apache.kafka" % "kafka-clients" % "1.0.0"
val embeddedKafka = "io.github.embeddedkafka" %% "embedded-kafka" % "2.3.0"

libraryDependencies ++= Seq(
  kafkaClients,
  embeddedKafka % Test
)

Compile configuration depends on 1.0.0 of kafka-clients, and Coursier silently evicts that and uses kafka-clients 2.3.0 that was pulled in from Test's transitive dependency? This is a surprise to me, and it's scary the more I think about it. Compile is the configuration that gets published out to Maven Central or deployed to production, and test configuration is just for testing.

Could we please make this behavior configurable?

[dwijnand]

Reminds me of https://github.com/sbt/sbt/issues/3201.

Personally I think Coursier's behaviour is a regression on sbt/ivy's behaviour, which already wrong as described in https://github.com/sbt/sbt/issues/3201.

[avdv]

Yes, this is expected. Resolution is done once for configurations that are related (only one resolution is run for compile / runtime / test for example).

Sorry, that is a deal breaker for me. I have been wanting to use coursier for so long, I cannot even remember. But there was always some glitch that caused me to disable it again. Seeing it integrated into SBT is a really nice thing, but if it does not behave in a sensible way, it is of no use. Still, :heart: and :+1: for your hard work @alexarchambault !

[OlegYch]

@eed3si9n in my testing the published artifact ends up using dependency from Compile config, despite it not being being during either compilation or testing
that is the real problem, because you publish something you haven't compiled or tested
test dependencies upgrading compile dependencies is actually a good a thing, because you want to test the thing you compiled, not something completely random, most of the times

[OlegYch]

at the very least we need to ensure that published classpath is the same as used for compilation
it would be great if we can do the same for testing and compilation (which is the case as things stand), but is probably less important