Language-server-protocol: Retrieve extended file information

[vladdu]

A text identifier uses an URI to refer to a file. A natural value is the file's path on disk, but more information might be needed. For example:

Eclipse has the notion of a workspace that is a virtual file system where the actual files and folders might be located in different places on disk. The file path can be used as a key, but then the server can't always know which files belong together, which might be important (for example, to use the right system libraries or to know the right configuration for each Eclipse project [=top level folder in the workspace]).

Then I think that it would be useful to be able to let the server know about this extra information.

Such static data can be sent via workspace/didChangeConfiguration, mapping virtual paths to file system locations, but when things change at runtime that might not be the best way. Maybe extending 'workspace/didChangeWatchedFiles could provide info when a location changes. Or a special request from the server could be added, asking for info on a set of files.

I mean this as a discussion opener, I feel there may be non-obvious issues hidden here.

[vladdu]

Using a custom URI scheme (including both paths, for example) makes it difficult to have a generic client that can talk to all kinds of servers.

[felixfbecker]

So if I understand you correctly, your issue is that workspaces in Eclipse don't have a single rootUri under which all project files are under, but rather multiple? Is there a concept of a "primary" rootUri?

Or is the problem only that Eclipse needs to scan (not-opened) files, but cannot use the file system because the files are not available on the file system where the language server is running? If yes, would this suit your needs? https://github.com/sourcegraph/language-server-protocol/blob/master/extension-files.md

[vladdu]

The second. There is a virtual file system hierarchy in an Eclipse workspace, and if I understand correctly, the server doesn't need to understand it, it treats the URIs as opaque IDs.

But the server need to read and parse even files that aren't open, so it needs to know about all of them, so your files extension seems like it would help here. I would though like another method for mapping a virtual URI to an accesible file:// one (where available). If the client and server share a file system, it's a waste to send all files' contents via the protocol. I can't open an issue on your repo, though. I'll try to create a PR.

Are these extensions thought to maybe make it into the main protocol? Some server implementations generate APIs at compile time from a protocol description and it's difficult to extend them.

[felixfbecker]

There shouldn't be a concept of a "virtual URI". The URI is passed from the client and either a URN or a URL. If it is a URL, it doesn't only uniquely identify the resource, but also includes information on how to retrieve (locate) it. So if the client passes you a file:// URL, you are free to read from the file system, if not, you should work with that URI and obtain the content in a different way (for example, it could be ftp://, in which case you could fetch it over FTP if you want to - but it is easier to let the client do that). The overhead is not that big of a deal really. You can request multiple file contents in parallel if you need to. The protocol already syncs file contents over the protocol for opened files anyway.

The extensions are supposed to be proposed to upstream at one point, yes. We just want to test a bit internally before we do that. The files extension has been working pretty great so far.

[vladdu]

If there is a simpler way than I think to solve this, I will be happy.

The Eclipse workspace is a virtual file system over the resources it contains, with projects at the top and folders and files beneath them. The locations on disk of these resources can be arbitrary, only the project description brings them together. Resources may be virtual (in a database or whatnot) or remote. The same file may be part of several projects, each with different settings that influence how the file is parsed, so the server needs to know the location of a document in both the workspace and the file system.

Eclipse uses a "platform:/" URI to describe the workspace hierarchy. This work nicely as a document identifier, but only Eclipse can map it to something usable by the server.

[felixfbecker]

@vladdu I feel like your problems would be solved with the files extension. It allows the LS to query for files in the workspace (or any base URI) and to retrieve text document content by URI. Any other use case you need? Why does the server need to know the location of the file on the (possibly remote) file system? I don't see anything wrong with a platform URL, the LS can still parse it and match the path component if that is what you need to do.

[vladdu]

It's primarily a question of optimisation. If the server can access the files directly (maybe via an alternative way than the primary URI gives), I think it's more efficient to do it instead of sending the content via the protocol. Think of a project with third-party dependencies - these could potentially be many megabytes in size and the server needs them in order to properly analyse the project.

I agree that the files extension is what is needed, the discussion can be continued there. https://github.com/sourcegraph/language-server-protocol/pull/15

[bruno-medeiros]

@vladdu The URI from LSP is not just an identifier, but a means to read files from disk (or other resource locations). As such, you shouldn't use Eclipse's platform: URI, but rather map each Eclipse workspace virtual file to it's actual URI (using getLocationURI). This will usually be "file://" on the filesystem.

For those that are not familiar with Eclipse, yes, it is possible to have multiple Eclipse-workspace files/folders that point to the same filesystem location, yet appear in different paths in the Eclipse-workspace. In addition, these Eclipse virtual files can have different document buffers, even though they map to the same filesystem location...

However, I have argued in the Eclipse community that such behaviour is confusing, if not downright broken, and I stand by that. Not even Eclipse itself handles this sanely all the time: it gets confused when you try to save the file, if you have two different editors open on different Eclipse-workspace files that point to the same filesystem path.

This aspect of the Eclipse workspace model has been a pain when using Goclipse especially, because Go has a more peculiar model of how Go source code should be organized (the Go workspaces etc.). But even when using Rust in Eclipse, I've had problems of having duplicate files in the Eclipse-workspace, each with different buffers, and it's annoying. And fundamentally it's Eclipse's problem.

IMO, because of all the above, it would be a horrible idea to try to have the LSP protocol support the Eclipse workspace model. I didn't even get a reply from the Eclipse team as to why this Eclipse behaviour is supported in the first place... what is the use-case here?

You do have a point tho, in that there is a use case for the issue of having multiple configurations of a same project. However it think this is a different problem altogether, and should be addressed in a different way - not by making possible in LSP to have different document buffers for the same URI, that's just trouble.

[vladdu]

I agree it's not a good way to structure the workspace, but let's not forget that the same level of insanity can be achieved by creating links at the file system level. So a file can have many distinct file:// URIs. If using hardlinks, these would even be quite difficult to detect too.

If we ignore the Eclipse issues, a more generic way to state the problem is (I think) this:

• the "original" (=VSCode's) notion of a LS project is a folder
• sometimes, code is spread over multiple folders = projects that refer each other
• these projects may be located in arbitrary locations on disk and may have different configurations
• in order to process the files correctly, the LS needs to be able to associate them to the right configuration/project and to be able to search the full context (=all projects)
• one way to do that would be by letting the client notify the server of how projects and files belong together (for example via didConfigurationChange or an alternative URI), but then the LS needs to be aware of a client-specific issue and isn't generic anymore (this is what I had suggested in this issue)
• another way is by using one LS per project, but each server would need to analyze (potentially) all projects, wasting resources and time.
• an alternative to this last problem is to let the servers share information; the good people at SourceGraph have something working in that direction (see https://github.com/sourcegraph/language-server-protocol/pull/15#issuecomment-272871770).

I guess the last solution is the cleanest.

[bruno-medeiros]

I agree it's not a good way to structure the workspace, but let's not forget that the same level of insanity can be achieved by creating links at the file system level.

Yes, but if you do that, the responsibility for handling the mess is entirely to you, the user, not the tool. No editor or IDE I know tries to understand filesystem links and behave in a special way if it detects the links point to the same file, etc.. They just treat the files opaquely, as it should be, because trying to figure anything more than that would be too much work for little (if any!) value. The simpler model suffices.

in order to process the files correctly, the LS needs to be able to associate them to the right configuration/project and to be able to search the full context (=all projects)

I'm not sure I'm understanding the problem correctly, but from what I can tell, this is an entirely separate issue of whether LSP or any editor should support a virtual filesystem model, as discussed above.

Also, it's a language specific issue, and not all languages will require extra information to process files for LSP operations. For example, Rust and D do not require this because of how their package managers work: the package managers (Cargo and DUB respectively) can figure everything out from just the file location. Go is close to that, the only extra thing it needs is the GOPATH environment variable.

[vladdu]

I'm not sure I'm understanding the problem correctly, but from what I can tell, this is an entirely separate issue of whether LSP or any editor should support a virtual filesystem model, as discussed above.

What I feel is needed is a way to notify the server of the project configuration. Doing it via a VFS model is generic and I thought it might benefit others too. If there is another generic way to do this, I'm all for it.

Also, it's a language specific issue

Yes, but if there are enough languages, then it might be accepted. Not all must need it.

At this point, I concede that it's probably not worth pursuing this direction. I will look for alternative ways. Thanks everyone for the feedback!