Metals: Presentation compiler and semanticDB indexing should wait until first compilation is finished

[tgodzik]

Describe the bug

Reported by @jvican - I will most likely pick it up this month.

"Metals is incorrectly assuming that compile products (both semanticdb files and class files) are available in the project classes directories as soon as Metals connects to Bloop. This is not the case.
As a result, Metals often doesn't fully index semanticdb files (which means go to definition often doesn't work) and the presentation compiler produces wrong results (many Anys, for example, and generally doesn't work)
To fix this issue, we should change Metals to not assume the compile products are available and trigger a cascade compile in the whole workspace and wait for its result. Only then Metals can assume there will be some semanticdb and class files in the project dirs."

To Reproduce
Steps to reproduce the behavior:

1. Any first compilation.

Search terms
compilation indexing semanticdb

[olafurpg]

I agree Metals should trigger compiles before doing some parts of its indexing pipeline.

I think we also need to change Bloop to copy classfiles to a new classDirectory even when the project has compile errors. I have seen case where completions/hover/... stop working when you run "connect to build server" or reload VS Code. The reason seems to be that every BSP session starts with empty classDirectory and the classfiles get copied over on the first successful compilation. If the project has a compile error then the classfiles don't get copied over and the presentation compiler has no signatures of symbols that are defined outside the focused text file.

To confirm this is the case, I just introduced a compile error in my project, ran "connect to build server", looked at "classDirectory" from the BSP trace and the directory is empty

❯ tree /Users/lgeirsson/dev/metals/.bloop/metals/bloop-bsp-clients-classes/classes-Metals-Nv2oElmeTueKCy9gRgrfUA==/
/Users/lgeirsson/dev/metals/.bloop/metals/bloop-bsp-clients-classes/classes-Metals-Nv2oElmeTueKCy9gRgrfUA==
└── META-INF
   └── semanticdb

[m4dc4p]

@olafurpg This is fantastic info. I think metals (vs code) does this to me frequently. So far, my solution has been to delete .metals and .bloop and re-import.

[tgodzik]

@m4dc4p That shouldn't be needed, .bloop contain mostly config files (aside from artifact directories) needed for Bloop to compile and reimporting will create the exact same files. Removing the directories inside .bloop might only be needed in case of corrupted class files. .metals doesn't really contain anything that could cause any issues.

Running Developer: Reload window is entirely enough to deal with the issues.

[tgodzik]

This also causes parsing errors to come up for Scala 3 codebases - rambo presentation compiler is used before connecting to Bloop.

[tgodzik]

Revisiting this issue after a while, I think we have it working pretty well right now. The old artifacts are now also copied even if the workspace is broken and though we are not waiting for the compilation to be finished, there is nothing breaking for the users anymore.

Currently in a new workspace:

• we index and load presentation compiler lacking most of the compiled artifacts
• some things will work, especially if it's defined locally
• as soon as the compilation is finished indexes will be updated with the new semanticdb files and presentation compiler will be reloaded with the compiled classfiles

I think, it's not really needed that we wait for the compilation to finish, because this way Metals will start providing some functionalities as opposed to making the user wait, which might cause additional confusion. The drawback here is that indeed some navigation, references might not work 100%, but that might even happen normally when working with a workspace and always waiting for the compilation will not be a good experience.

Alternatively, we could try to do the same as https://github.com/scalameta/metals/pull/2169 , however I believe that current state is satisfactory and it might provide a worse UX experience if we wait for the first full compilation.

[tgodzik]

Let me know if anyone thinks the current situation is not satisfactory, otherwise I think I will close this issue and open new ones if any particular new problem shows up.