Bloop: Use a standard configuration format that other tools can reuse

[jvican]

This is a ticket to keep track of collaboration with other tools in the Scala commuinty. Bloop should settle on a configuration format that can be read by any tool (from LSP servers, scalafix to other build tools).

As this format doesn't exist, we should probably pioneer one. Another possibility is to reuse .eclipse configuration files -- that are widely supported by Eclipse, Intellij and vscode (not natively, but some plugin extensions do) -- or directly generate .idea configuration files. However, we may want to encode more information in the file and it's not clear yet whether eclipse configuration files allow it.

Supporting all this configuration files is easy, so we may also consider understanding all of these so that people that heavily use Intellij and rely on sbt-idea to import their projects do not need to run install in the build. This would mean there's a standard, preferred configuration file but bloop can also understand others. This is a nitpick and it's only UX work, so this won't happen anytime soon. But it's a thing to keep an eye on.

My ideal format is json because it can be edited and it's readable (unlike protobuf). This json format should have also a library that provides bincompat readers and writers so that other tools that want to read/write from it only need to depend on the library and use it.

When bloop is public, I'll start summoning people up in this ticket so that we converge to one solution and explore how useful this file would be, and how it could help make better tooling.

[olafurpg]

My 2c, nothing novel just some thoughts

• if you go with JSON, I recommend defining a JSON schema http://json-schema.org/ (for example. Buckescript build system does that https://bucklescript.github.io/bucklescript/Manual.html#_bucklescript_build_system_code_bsb_code. JSON schema has great IDE support in out of the box in IntelliJ + vscode without any plugins.
• if you define your own schema, I suspect it will be a challenge to come up with a widely accepted model for defining projects/targets/modules and their attributes.
• I would consider looking into adding build related extensions to LSP, similar to what they're doing in sbt. I think this could be hugely valuable for IDE integration, esp. when once bloop supports running tests

[ShaneDelmore]

2 more cents, no more novelty. The Clang community has settled on a format and quite a lot of tooling has sprung up around it. I would love if we could do the same in the scala community. It may make sense to re-use the same keys they use for our shared bits of information and then just enrich the file with additional information we need to help us interoperate with existing tooling.

https://sarcasm.github.io/notes/dev/compilation-database.html

My other thought, I would like for this file to contain enough information to be able to determine if artifacts built with these settings are still valid or if they need to be rebuilt. I don't know all of the variables besides the scala source code, maybe it is just classpath, compiler flags, and scalac version?

[propensive]

I know I brought this up in another ticket, but I'll reiterate here, as it's more relevant.

If the file format needs to be read by other tools, JSON is too heavyweight a dependency to require, as it becomes largely inaccessible to shell scripting (which is a particular use-case for Bloop). JSON can be read with jq, but it's not standard.

The existing key/value format works very well for shell scripting, as grep, cut and tr provide most of the functionality required for manipulating keys, even if they're rather blunt tools for the task.

To throw another idea into the pot, but one I'm by no means wedded to, the git file format seems like a reasonably good compromise between human-readability, developer familiarity, and computer-readability. Any git config file can be read with git config --file <file> --get <key>, and it's more reasonable to expect git to be available than jq.

I have no idea about HOCON, but anything which requires complex parsing immediately makes the format less useful. Even changing from a single = delimeter to a space-padded = makes the parsing harder, for example. Needing to remove quotes, or do escaping just introduces opportunities for errors and fragility, as programmers build tools which work adequately for their own common use-cases, and break for some other users with slightly different needs.

In summary, please don't introduce complexity you don't need!

[ShaneDelmore]

HOCON desugars to JSON so any JSON is valid HOCON, and a HOCON parser can read HOCON and emit JSON. Not sure if that helps related to the above points, just trying to fill in some of the blanks regarding HOCON, not an argument for or against.