Coursier: Resolver-side shading

[ryan-williams]

This is a proposal for a large feature that I think the JVM ecosystem needs, and coursier seems like a promising place to do it.

The idea is basically to resolve conflicting transitive-dependencies by creating shaded copies of each version of a conflicted library, and rewriting class-files in [other dependencies that depend on the conflicted one] to point at the shaded version they wanted.

This is different from normal shading workflows because it happens at resolution time; today, we generally hope that upstream libraries shade commonly-problematic deps (e.g. guava) before publishing, but that's usually not done, and anyway is the wrong place to solve the problem: the right set of things to shade from a given library depends on what else is on the classpath, which is different in every downstream library that uses it.

Here are relevant illustrations from an unconference presentation I gave about this idea at NEScala 2017:

"Before" / Today:

"After" / Future:

If this makes sense to ppl then I'd like to help build this here, after dreaming about it for a long time. I have a lot of thoughts about how to do it that we can discuss.

Also, if there is prior art I may have missed, pointers are welcome!

To pre-empt one possible comment: a lot of people I've talked to have said that they thought JDK9 modules solve this (and even that the point of JDK9 modules was to solve this!), but my impression is that they don't; quoting the spec:

It is not a goal of the module system to solve the version-selection problem, which is best left to build tools and container applications.

Looking forward to thoughts here (and also hoping to discuss/work on it with folks at Scala Spree NYC tomorrow 😀)

[stuhood]

I think that this is a fantastic idea. While I would of course love for someone to implement it in a build system ("our" build system) rather than in a dependency resolver (which should generally do one thing, well), solving it in coursier seems like it would more efficiently help the ecosystem-in-general.

For bonus points: to avoid creating "too many" clones of libraries, you might consider hashing/comparing the public APIs of different versions of libraries to determine whether they are API compatible before cloning them. So for example: if v1 and v2 are API compatible, you'd choose one of them rather than cloning/shading. There is prior art for this in rust's cargo.

[ryan-williams]

Thanks @stuhood, that all makes sense.

While I would of course love for someone to implement it in a build system ("our" build system) rather than in a dependency resolver (which should generally do one thing, well), solving it in coursier seems like it would more efficiently help the ecosystem-in-general.

I hear that; to me, this is part of "doing resolving well" that, coincidentally, no resolvers do today 😄

For bonus points: to avoid creating "too many" clones of libraries, you might consider hashing/comparing the public APIs of different versions of libraries to determine whether they are API compatible before cloning them. So for example: if v1 and v2 are API compatible, you'd choose one of them rather than cloning/shading. There is prior art for this in rust's cargo.

gtk about this in rust, and yea there are a lot of interesting possible optimizations like this.

FWIW, in the specific case you've cited, my default position is to not take that shortcut, bc it still results in some code being run against code that it wasn't built against. v1 and v2 being API-compatible doesn't mean they do the same thing (indeed, presumably they do something different, otherwise why are they two different versions?), and I am approaching this problem with a conservative posture about preserving "correctness" in the sense of "not running code against dependencies other than those which it was built/tested against".

Fingerprinting all the way down to individual methods, and the transitive-closures of other classes and methods they reference, to identify cases where the byte-code is identical, and doing deduplication like you've described, is definitely "in scope" for some (later) version of this effort, I think.

[alexarchambault]

In which module(s) of coursier would you see this added?

IMO, it can be an extra feature of the sbt-shading plugin (which would shade the B and C dependencies, while at the same time shading in them the v1 and v2 of D), and / or of the fetch command in the CLI (which would generate some extra JARs with shading / renaming applied, and print those instead of the downloaded JARs).

I guess the latter case would be more straightforward to implement. But sbt users would only benefit from it via the sbt plugin…

@stuhood Couldn't pants benefit from having the fetch command having that feature?

[ryan-williams]

yea @alexarchambault I'm imagining this being somewhere more "core" to coursier than the sbt-plugin; accessing it via the CLI, and across any build tools that use coursier (e.g. pants) is a goal!

The basic interface I'm imagining is:

• input: list of Maven coords
  
  
  
  • typically: the direct dependencies of a library being compiled/run (regardless which build tool is being used)
  
  
  • letting a user feed dependencies in from the coursier CLI could also be useful
  
  
• output: a "safe" classpath containing those dependencies and their trans-deps
  
  
  
  • in the common case this would be a list of local-filesystem paths, like SBT's fullClasspath today
  
  
  • some of these would be unmodified .jars under ~/.{ivy2,m2,coursier}, also like today
  
  
  • others would be rewritten JARs under target; they are specific to a project and are outputs of build tasks
  
  

(this framing came out of discussions with @jvican and @smarter at the Spree on Tuesday)

Notes on publishing

Here's a bunch of thoughts about different cases we'd run in to when attempting to synthesize "safe classpaths" for libraries and publish them for downstream consumption.

Easy (+ common?) case: conflicting private dependencies

When a conflicted library (D in the images above) is a "private" dependency (not exposed in the public API) of other dependencies (B, C above) of a library we are developing (A), we can publish A as usual, and anything downstream can choose to use resolver-side shading to deal with [the {B,C}→D conflict they now inherit from A] or just take their chances with "legacy" conflict-resolution 😀

Backwards-compatibility with existing conflict-resolution strategies comes for free here.

Note also that a downstream user of A may have other dependencies that bring in Dv3 and Dv4, or other paths to Dv1/Dv2, and that's fine, they can build their own safe classpath, and are not required to rewrite Dv1 and Dv2 references inside B and C in the exact same way as we did when building/testing A, since the published A artifacts contain no direct references to D, renamed or otherwise.

Harder case: conflicting public dependencies

In other cases, A will end up with references – in its byte-code – to D1 and/or D2, in which case downstream libraries need to know that the rewriting happened, and how.

Here, the conflicted class (Opt), is exposed in the public APIs of B and C (resemblance to Spark's past struggles with guava's Optional being exposed in its public API is not entirely coincidental 😀) and, importantly, A calls methods .map on Dv1 and Dv2, so its published bytecode will contain references to those synthetic classes/methods.

In this case, I presume we will stuff some metadata into suitable fields of the published POM / ivy.xml, and downstream libraries will have to provide A with similarly-renamed Dv1 and Dv2 on the classpath.

Additionally, two .jars could be published for A (one with a classifier like -shaded or -safe, though the former is already used in the wild to mean something different; example), so that downstream libraries can opt in or out of needing to abide by these rules, vs deal with the conflicts on D in the old way.

Importantly, we can save on this trouble if A only actually uses ≤1 version of D directly: if B and C publicly expose Dv1 and Dv2 classes (resp.), but A only ever directly uses Dv1 via B, then Dv1 (and B) can be left un-rewritten, Dv2 alone can be rewritten, A will be published with references to plain D (and should probably include D (v1) as a direct dependency), and we can fall-back to the easy / backwards-compatible case above.

Impossible case → compile error

If A actually tries to pass something from Dv1 to something that expects a Dv2, naively rewriting things as above will give a compile-time error, which seems basically ideal:

Additional/Clarifying functionality: support directly depending on >1 version of same library

To resolve the compiler error above, we want to let A convert between a Dv1 Opt and a Dv2 Opt, however it's generally impossible to write such a function today because it requires depending on and using two classes with the same FQN from two different versions of the same "library" (Maven {group,artifact} pair).

Allowing direct dependencies on multiple versions of the same library naturally falls out of a lot of what's discussed above, and is in itself a powerful feature that naturally would have a home in a tool like this.

[alexarchambault]

Closing this for now, as it seems no one's working on it right now. But feel free to keep discussing things here, or to send related PRs anytime.