Bloop: Distinguish between compile-time and runtime classpaths

[Duhemm]

Currently, Bloop doesn't make any distinction between compilation-time and runtime classpaths. This situation brings several problems for projects that expect to have different runtime and compile-time classpath:

• Some code may compile in Bloop, but not in the tool that generated the config (for instance, my code references a symbol that appears only on my runtime classpath)
• Some code may compile in the tool that generated the config, but not in Bloop (for instance, a name became ambiguous because of 2 wildcard imports)
• We cannot faithfully represent builds that use Gradle's api vs implementation, sbt's Provided or Pants' strict_deps for instance.

The Bloop configuration file should include different fields for these different classpaths.
Finally, Bloop should use the right classpath to perform the right action.

[Duhemm]

I've been prototyping 2 different implementations of a distinction between compile-time and runtime classpath, in order to evaluate the advantages and drawbacks of each approach.

Use compileClasspath and runtimeClasspath in every target

https://github.com/scalacenter/bloop/commit/ebbf0dfa12c21f92e2127e68acc1e4a3aa77d2a8
Here, each Bloop configuration file will have 2 fields compileClasspath and runtimeClasspath which hold the different classpaths. These two fields can have completely different entries.

Advantages

• The encoding is simpler
• Fewer targets are created and available to the users
• Every field of each configuration file has a meaning

Drawbacks

• Some complex configurations cannot be expressed (eg. different platform at runtime), though we could add more fields to the configuration file or define separate projects to handle these cases.
• The configuration file becomes larger
• Implications with offloading from sbt

Create a new -runtime target for every configuration

https://github.com/scalacenter/bloop/commit/3324f940a760c8e8f1dda6fab6692d3383777be2
Here, we use the same "trick" we used for the test configuration: Every configuration that we expose will now export also a -runtime target which will be used when executing the code. This means that a project foo will export the following Bloop configuration files:

• foo, which is used to compile the main sources
• foo-runtime, which has the configuration for running the product of the main sources
• foo-test, which is used to compile the test sources
• foo-test-runtime, which is used to execute the tests.

The implications of that approach regarding BSP are unclear to me.

Advantages

• More flexibility regarding runtime vs compile time configurations out of the box
• Compilation targets are immutable
• No changes to the configuration file

Drawbacks

• More targets, which may hurt in large build graphs
• Many fields of the configuration files become unused (eg. java options in compile target)
• The UX is going to be difficult to get right (eg. what does bloop test foo-runtime mean?)

This is a more complicated encoding, but it allows us to express more subtle differences between compile-time and runtime configurations, such as using different platforms, for instance.

In my opinion, adding fields for the compile-time and runtime classpaths (and possibly more fields in the future, if we need to) seems like a clear winner. In my opinion, runtime vs compile-time doesn't relate to compile vs test, and we shouldn't try to retrofit the same solution to a different problem.

In terms of user experience, I believe that adding fields is also better, because readers of the configuration file get a complete view of the configuration for a given target, and no configuration value is unused.

Implementation-wise, I also think that distinguishing between compile-time and runtime classpaths in the configuration file is easier and will lead to fewer bugs in Bloop.

@jvican You had concerns regarding compilation offloading from sbt (and, I imagine, other build tools as well). Could you please comment on why the first approach described here would make this hard to implement?
@olafurpg @jvican What would be the implications of the second approach regarding BSP?

[olafurpg]

Thank you for writing up this thorough comparison of the two approaches!

What would be the implications of the second approach regarding BSP?

Some open questions:

• workspace/buildTargets: should -runtime targets be listed? IntelliJ struggles to represent two targets that have the same base directory and relies on a heuristic to merge targets to recover in this scenario.
• buildTarget/jvmTestEnvironment: should Bloop use the classpath of a-runtime or a? If you use the classpath of a then you risk runtime exceptions and if you use the classpath of a-runtime then we need to add special handling in Bloop for targets named -runtime. In BSP, we currently have no special handling for -test targets.

[jvican]

Thanks for looking into this @Duhemm so carefully and following up with a prototype to the second proposal I made to you privately.

Here, we use the same "trick" we used for the test configuration

Hmm, I would not say this is a trick. It's exactly how runtime classpath has been meant to be encoded in bloop from the very beginning. It's part of the design. One build target, one classpath. I've responded to similar runtime classpath requests in the issue tracker in the past with the same argument.

I believe it's wrong to inline runtimeClasspath into the current schema. We already have a classpath in the schema. We also have a resources field in the schema, which specifies resources for both compile and run and which would make sense to configure separately. We have a platform field to control how a project should be run (jvm options, jdk, scalajs link options, scala native). These fields exist in our schema of what a valid configuration is, regardless if they are for compilation or runtime targets. Re-adding them to the schema as runtimeClasspath, runtimeResources and runtimePlatform doesn't make us any favor, adds boilerplate to our schema and goes against the model of one build target, one classpath.

In addition to that, the first proposal has the inconvenience that we rewrite the configuration target to add runtime classpath information, that in some build tools such as sbt and gradle require the execution of runtime source/resource generators. This means that the configuration files always need to be re-parsed by bloop when the runtime inputs change. In the second approach, this doesn't happen, a new target is created, which is what makes sense.

Let me comment on some of the advantages/disadvantages you've mentioned.

Advantages of first approach:

• The encoding is simpler

This is honestly subjective, I don't think it's simpler.

I guess what you mean here is that it's easier for a build tool to export a build because it doesn't need to manage two build targets for a build unit. But this is only needed if you do export runtime configuration. I think this is a fair requirement.

• Fewer targets are created and available to the users

Not necessarily. We can add the runtime build target implementation detail and not leak to the user. We can skip runtime targets from autocompletions, from bloop projects and from build import via BSP so that the user doesn't know they exist.

• Every field of each configuration file has a meaning

I don't understand what you mean here. Every field in the configuration has a meaning. The runtime targets and the normal targets share the same schema. This is normal and it's a feature, I don't think that filling in some fields in one target and others in another one is a disadvantage.

The same point goes for the same disadvantage in the second approach, where you say "Many fields of the configuration files become unused (eg. java options in compile target)".

Disadvantages of second approach
More targets, which may hurt in large build graphs

We would need to have some numbers to know whether this is an issue or not.

The configuration files of runtime targets only contain the name, dependency info, classpath, resources, and platform. The resolution info is not there. The size of these targets is minimal and parsing time is not an issue.

One could say that having one more target will make DAG operations more expensive, but I don't think that's true. Runtime targets are the leaf targets, they are not depended on by any build target, so these targets are not in the way of most of the DAG traversals we do.

In terms of user experience, I believe that adding fields is also better, because readers of the configuration file get a complete view of the configuration for a given target, and no configuration value is unused.

The configuration files are an implementation detail of bloop, they are only to be read by bloop, we don't have any public contract on what people should expect to see there. If things go wrong, the recommended approach is that users use --verbose which dumps the classpath to be used.

The UX is going to be difficult to get right (eg. what does bloop test foo-runtime mean?)

The UX can be kept exactly the same way as it is. There's no reason why we need to leak how this works to users. bloop test foo-runtime could be forbidden. -runtime targets could not be shown in BSP and bloop projects, etc.

buildTarget/jvmTestEnvironment: should Bloop use the classpath of a-runtime or a?

It would have the same UX as before, neither the CLI nor BSP would allow operations on -runtime targets.

I don't think implementing the second approach fully is going to take more time than adding a runtimeClasspath field. In fact, I think the changes to implement this should be quite minimal.

[olafurpg]

@jvican one concerns I have with the -runtime approach is that it introduces special meaning to the project name. This sounds conceptually like a regression in the design of the Bloop JSON config format. It's a different situation from -test because the BSP server does not have special treatment of targets named *-test.

The UX can be kept exactly the same way as it is. There's no reason why we need to leak how this works to users. bloop test foo-runtime could be forbidden. -runtime targets could not be shown in BSP and bloop projects, etc.

This sounds like a lot of edge-cases that need to be handled in order to ensure that -runtime targets don't leak into the UX (progress bars, IDEs, error messages, ...). I am more confident we can introduce runtimeClasspath with less special cases resulting in fewer related issues.

Re-adding them to the schema as runtimeClasspath, runtimeResources and runtimePlatform

The proposal under discussion is only to introduce runtimeClasspath and that is the design that is implemented in the commit https://github.com/scalacenter/bloop/commit/ebbf0dfa12c21f92e2127e68acc1e4a3aa77d2a8. There is no proposal to add runtimeResources or runtimePlatform.

The configuration files of runtime targets only contain the name, dependency info, classpath, resources, and platform. The resolution info is not there. The size of these targets is minimal and parsing time is not an issue.

I did an experiment in a workspace with 1700 Bloop projects (actual codebase used by one of our teams) to measure the size of the classpath field is compared to the other fields. Removing classpath reduces the size of the JSON files by ~90%

❯ du -h json-with-classpath json-without-classpath
244M    json-with-classpath/.bloop
 27M    json-without-classpath/.bloop

I don't think we should use performance as an argument in favor of the runtimeClasspath design. However, if performance was the deciding factor then I think runtimeClasspath has better performance characteristics than creating synthetic -runtime projects.

[olafurpg]

I think it's worth adding that all build tools that are supported by Bloop have a built-in feature to encode runtime classpaths. In sbt there % Runtime, in Bazel there are runtime_deps, in Pants there is strict_deps. I think this should be sufficient evidence to support that runtime classpaths are a primitive feature that should be supported in a JVM build tool.

[olafurpg]

Would it make sense to move runtimeClasspath into the JVM "platform" field? It's not possible to encode runtime classpaths for JS/Native projects. We may also want to rename the field into something like runtimeExtraClasspath to stress that these are additional classpath entries that are added on top of the classpath field.

[jvican]

It's a different situation from -test because the BSP server does not have special treatment of targets named *-test.

Our CLI does have this special treatment. Running bloop test foo is mapped to bloop test foo-test. I agree this approach introduces even more special treatments to the names of the generated configuration files, but I think this is a fair requirement.

This sounds like a lot of edge-cases that need to be handled in order to ensure that -runtime targets don't leak into the UX (progress bars, IDEs, error messages, ...). I am more confident we can introduce runtimeClasspath with less special cases resulting in fewer related issues.

I don't think you require that many special cases. I have in my mind exactly the places where we would need to hook up this presentation logic and they could all reuse the same logic.

There is no proposal to add runtimeResources or runtimePlatform.

There is no proposal yet, but I'm sure it'll come. I can't implement support for runtime classpath in sbt-bloop or another build tool without supporting runtime resources. Our design needs to account for the fact that these fields will be required.

I did an experiment in a workspace with 1700 Bloop projects (actual codebase used by one of our teams) to measure the size of the classpath field is compared to the other fields. Removing classpath reduces the size of the JSON files by ~90%

If the classpath alone is this big, the boilerplate json of writing it to another configuration file is moot, it's just a constant factor. The runtime target only would have runtime classpath info, it would be dominated by the size of the classpath. It doesn't matter if that classpath is inlined in the same config or in an independent file.

I think it's worth adding that all build tools that are supported by Bloop have a built-in feature to encode runtime classpaths. In sbt there % Runtime, in Bazel there are runtime_deps, in Pants there is strict_deps. I think this should be sufficient evidence to support that runtime classpaths are a primitive feature that should be supported in a JVM build tool.

We agree on the need for encoding runtime classpath. But I don't think the fact that most tools encode it this way is sufficient evidence that we need to support runtime classpaths just like everyone does because bloop has a different data model and different requirements than most build tools. And the most important of those requirements is that our config files are not manipulated by users directly, they are auto-generated.

[jvican]

Would it make sense to move runtimeClasspath into the JVM "platform" field? It's not possible to encode runtime classpaths for JS/Native projects. We may also want to rename the field into something like runtimeExtraClasspath to stress that these are additional classpath entries that are added on top of the classpath field.

I like the fact that you scope it to the JVM platform. It eases some of my concerns with this approach and I could make my peace with a classpath field added under runtime JVM platform. This would mean we would add a resources field there too.

But note that you cannot encode it as extraClasspath because the runtime and compile classpath can be disjoint sets of URIs. You can add new URIs to the compile classpath as well as you can remove them or even change the order of the URIs. So if you want to encode the extra classpath to avoid repeating some of the URI entries in the compile classpath and save space, we need a way for you to express that.

[Duhemm]

I like the idea of scoping the runtime classpath / resources to the JVM platform, it makes a lot of sense. Let's try with that.

[jvican]

I suggest that if configuration file size is an issue because of large classpaths and resolution information, we add support for gzipped configuration files foo.json.zip so that build tools such as Pants or Bazel with lots of targets and classpath entries can use them.

Bloop can decode the JSON configuration files on the fly and then parse them and, in exchange, the overall size should be drastically reduced. I believe this is a more promising avenue than creating extraClasspath fields to save space.

[olafurpg]

I agree we should keep the runtime/classpath fields separate, there is no need for extraClasspath. This design allows encoding compile-only dependencies (eg % Provided in sbt).

The file size of the configuration files is not a concern (at least not right now). I mentioned the file size only to demonstrate that the "classpath" field accounts for up to 90% of the size of the JSON files in our codebase. Even if we compress the files, the duplicate runtime/classpath fields result in higher memory usage.

Do we have an agreement to add a "runtimeClasspath" field to the JVM platform section?

[jvican]

Even if we compress the files, the duplicate runtime/classpath fields result in higher memory usage.

Not if we intern them. In fact, we don't need to intern them manually, Paths.get interns them right when they are created at parsing time.

Do we have an agreement to add a "runtimeClasspath" field to the JVM platform section?

Yes, but let's call it classpath.