Coursier: Allow fetching transitive provided dependencies

[dwijnand]

From observations using Coursier as a library in MiMa it seems that transitive provided dependencies aren't returned from fetch. Provided dependencies shouldn't be present when running, but my usage is at compile time (parsing classfiles in the jar).

In particular this means that sbt plugins, that (automatically) have a transitive provided dependency on sbt, don't get sbt as a dependency, resulting in bad MiMa reports.

[alexarchambault]

sbt plugins […] don't get sbt as a dependency, resulting in bad MiMa reports.

I'm not sure I understand how this is a problem.

Provided dependencies of transitive dependencies are always ignored during resolution.

[dwijnand]

Provided dependencies are provided by something for a runtime, but they're non-optional dependencies outside of runtime. MiMa is studying the bytecode of a jar (or a dir of classes), bytecode that can reference bytecode in another dependency. As you say "Provided dependencies of transitive dependencies are always ignored during resolution" which is a problem for me/MiMa.

What MiMa does today, for reasons I haven't discovered yet, is it provides a stub "SytheticClassInfo" when it doesn't find a class on the classpath, which is a class with the given (looked up) name, extends Object and that's it, which means false "missing methods" (etc) errors... 😕

Does that make more sense?

[olafurpg]

I just hit on a use-cases where it would be useful to be able to optionally include jars of provided transitive dependencies. My use-case is to compile source code from the -sources.jar of a library. Currently, I get compile errors from unresolved symbols that origin in provided dependencies.

For example, here is an error compiling the byte-buddy sources

/var/folders/mb/88qbygqj169byvv_68bw4fn80000gn/T/semanticdb-javac5269371498875463369/net/bytebuddy/ClassFileVersion.java:18: error: package edu.umd.cs.findbugs.annotations does not exist
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
                                      ^
/var/folders/mb/88qbygqj169byvv_68bw4fn80000gn/T/semanticdb-javac5269371498875463369/net/bytebuddy/ClassFileVersion.java:291: error: cannot find symbol
    @SuppressFBWarnings(value = "REC_CATCH_EXCEPTION", justification = "Exception should not be rethrown but trigger a fallback")
     ^
  symbol:   class SuppressFBWarnings
  location: class ClassFileVersion
/var/folders/mb/88qbygqj169byvv_68bw4fn80000gn/T/semanticdb-javac5269371498875463369/net/bytebuddy/ClassFileVersion.java:481: error: cannot find symbol
            @SuppressFBWarnings(value = "REC_CATCH_EXCEPTION", justification = "Exception should not be rethrown but trigger a fallback")
             ^
  symbol:   class SuppressFBWarnings
  location: class CreationAction
3 errors

The annotation is defined in the provided find-bugs dependency

❯ cat /Users/olafurpg/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/bytebuddy/byte-buddy/1.10.20/byte-buddy-1.10.20.pom | grep -A 3 -B 1 findbug
    <dependency>
      <groupId>com.google.code.findbugs</groupId>
      <artifactId>findbugs-annotations</artifactId>
      <version>3.0.1</version>
      <scope>provided</scope>
    </dependency>

❯ jar tf $(cs fetch com.google.code.findbugs:findbugs-annotations:3.0.1) | grep SuppressFBWarnings
edu/umd/cs/findbugs/annotations/SuppressFBWarnings.class

@alexarchambault I would be willing to contribute a fix if you would be open to support this functionality. Do you maybe have pointers on how to implement/test such a feature?

[alexarchambault]

@olafurpg I guess this can be added as an optional feature, disabled by default.

I think the following steps should work:

• in the core module
  
  
  
  • adding a boolean flag in Resolution around here, to enable that feature,
  
  
  • the code around here checks whether a sub-dependency (dep0) originating from a dependency (from) should be added to the dependency set, depending on from and dep0 configurations (compile, runtime, provided, etc.), and tweaks dep0 before that if needed. mavenScopes referenced just above it defines which configuration a sub-dependency should have to be allowed in, depending on the configuration of the dependency that pulls it (a runtime sub-dependency of a runtime dependency is accepted, while a provided or runtime sub-dependency of a compile dependency is not - as of now, that is). This code should be tweaked so that it keeps provided dependencies when the flag above is true.
  
  
• in the high level API:
  
  
  
  • add the flag (as an Option[Boolean]) to ResolutionParams, then pass it to Resolution in this method if the option is non empty.
  
  
• in the CLI:
  
  
  
  • add the flag to ResolutionOptions as an Option[Boolean] too, and ensure it is passed to ResolutionParams in this method.
  
  

To test that manually, you can run sbt ~cli/pack in a console, then manually run the coursier CLI in another one with modules/cli/target/pack/bin/coursier resolve …, and see if you get the dependencies you expect when you pass the flag you added or not.

As non regression tests, I'd recommend adding a test in ResolveTests, and run it while manually forcing this variable to true. Once it passes, set it back to its default, and check that the test still passes. This should add metadata under modules/tests/metadata in particular. This directory is a git submodule. The new files need to be added to its repository (open a PR for it there). Then the git submodule reference needs to be updated, along the other changes.

[olafurpg]

@alexarchambault thank you for the detailed pointers! I will try to get into my queue for the coming weeks. I'm hoping to build a large part of Maven Central from source :))