Bloop: Jar dependencies between projects not working without publishing local jars

[tgodzik]

Reported by @ayoub-benali in Metals

Given this workspace:

root
| - A
| - B

with project B depending on project A. Maven can't compile project B because project's A jar isn't available. Probably the generated bloop configuration isn't correct.

Current workaround is to do a mvn install -DskipTests to force JARs to be published locally but this isn't ideal for development .

To Reproduce

1. Clone https://github.com/dcmn-com/herringbone.git and switch to upgrade-scala-2.12 branch
2. install thrift locally (apt-get install or brew)
3. import build with Metals in any text editor

See error

metals: [ERROR] Failed to execute goal on project herringbone-main: Could not resolve dependencies for project com.stripe:herringbone-main:jar:0.0.1: Could not find artifact com.stripe:herringbone-impala:jar:0.0.2 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
metals: [ERROR] 
metals: [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
metals: [ERROR] Re-run Maven using the -X switch to enable full debug logging.
metals: [ERROR] 
metals: [ERROR] For more information about the errors and possible solutions, please read the following articles:
metals: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
metals: [ERROR] 
metals: [ERROR] After correcting the problems, you can resume the build with the command
metals: [ERROR]   mvn <goals> -rf :herringbone-main
metals: build tool exit: 1
metals: time: ran 'mvn bloopInstall' in 7.94s
metals: Maven command failed: /tmp/metals516909384235530052/mvnw ch.epfl.scala:maven-bloop_2.10:1.3.2:bloopInstall -DdownloadSources=true
metals: Import project partially failed, limited functionality may work in some parts of the workspace. See the logs for more details. 
metals: running installed 'bloop bsp --verbose --protocol local --socket /tmp/bsp1697843160558780888/-rbfh4dcwxpo1.socket'
metals: tracing is disabled for protocol BSP, to enable tracing of incoming and outgoing JSON messages create an empty 

Expected behavior
The build to work

Installation:

Operating system: Linux
Editor: Sublime
Metals version: 0.7.0+111-0039eb7c-SNAPSHOT
Bloop version: v1.3.2
Search terms
maven, multi project, bloop

[tgodzik]

@jvican Not sure what we can do here actually coming to think of this. It fails, because we need those dependencies while running bloopInstall, so the only solution is to generate them :/

[tgodzik]

@ayoub-benali I looked into this a bit more.

The dependencies are specified as:

<dependency>
      <groupId>com.stripe</groupId>
      <artifactId>herringbone-impala</artifactId>
      <version>0.0.2</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.thrift</groupId>
          <artifactId>libthrift</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

it only works because Maven publishes those jars. There is nothing we can do here aside from running install. Can the interproject dependencies be specified differently? Not a Maven expert.

[ayoub-benali]

don't know much about Maven to be honest, I am not the author of the project as I forked it.

I just know that when you do mvn compile, maven is able to compile the dependency without having to run mvn install. Why isn't bloop able to do the same ?

[tgodzik]

don't know much about Maven to be honest, I am not the author of the project as I forked it.

I just know that when you do mvn compile, maven is able to compile the dependency without having to run mvn install. Why isn't bloop able to do the same ?

It's maven that is failing due to not having the dependency. There is no way around it - look here https://stackoverflow.com/questions/12223754/how-do-i-setup-maven-sub-projects-with-inter-dependencies

You need to install first for it to work. The only thing we can do is make bloopInstall depend on isntall, but not sure if we want that.

And my guess is that compile already depends on install.

[jvican]

I have a fuzzy idea of what's going on here, but it'd be good if there's someone more familiar with Maven builds than myself and @tgodzik that can look into this and propose a solution that doesn't affect the current fast workflows to export bloop from Maven. One thing I'm afraid is that supporting this use case would mean compiling while we export the build, which is a no-go.

[tgodzik]

I have a fuzzy idea of what's going on here, but it'd be good if there's someone more familiar with Maven builds than myself and @tgodzik that can look into this and propose a solution that doesn't affect the current fast workflows to export bloop from Maven. One thing I'm afraid is that supporting this use case would mean compiling while we export the build, which is a no-go.

I think there is no way around other than make bloopInstall always run after install :/

[jvican]

Let’s then close as wontfix ? If someone figures out another way I’m all ears 👂

[mzarnowski]

It's maven that is failing due to not having the dependency.

it doesn't seem to be maven. Here is a minimal repro: https://github.com/marek1840/bloop_1022
when running mvn compile everything works fine (project B uses project A), even though there is nothing installed in local m2 repo.

How about resolving dependencies this way:

for dependency in module.dependencies:
  if dependency in workspace.modules then depend on a module
  else depend on a jar from repository

I am not sure how feasible it is with the current maven plugin (especially when child modules don't declare their parents), but this way we could avoid building the project when importing.

[jvican]

If that's the issue at hand, that sounds like a good way to solve this problem :+1: I don't understand why the Maven plugin and Maven itself is not doing that like other build tools do... :thinking:

[jvican]

I'd love to have a closer look at this ticket but it doesn't look like there's anyone available to implement @marek1840's proposed improvement and therefore I'm inclined to close.

If someone volunteers to look at this, I'd be more than happy to reopen! There's too many things on our plate right now and improving Maven support isn't a priority even though it'd be really nice to support it as well as we support Gradle or sbt.

I'm trying to keep open only those tickets that are either being worked on by some contributor or that I can actively try to fix some time soon, and this one doesn't belong to any of those categories, hence my rationale to close.