Language-server-protocol: Syntax highlight

[vladdu]

Is the support for syntax highlighting supposed to use the document symbols request? Wouldn't it be reasonable to be able to request this for part of a document too (in case the file is huge and scanning all of it is prohibitive)?

[cdietrich]

@vladdu document symbol is a fictive "open type in current file" in eclipse

[vladdu]

Thanks, but is it? The docs say "The document symbol request is sent from the client to the server to list all symbols found in a given text document." Anyway, what I am after is if there is anything about syntax highlighting. There is an issue about semantic highlighting, but I can' see anything explicit about any highlighting and 'document symbol' seemed like the thing that could provide the required information.

[cdietrich]

well i think the semantic highlighting mentioned in the other issue is about server side highlighting in general

[smarr]

Issue #18 is the only thing so far as it seems.

The document symbol stuff gives you this in VS Code:

[vladdu]

I can't look at the VS Code code right now, but maybe that list gets filtered in the client (using the container field for selecting symbols at the right depth). According to the docs, the symbol information will be about _all_ symbols in the document (including strings, numbers, booleans).

[vladdu]

To make my question clearer: is syntax highlighting even supposed to be supported through the protocol at all (in the current version)?

[egamma]

To make my question clearer: is syntax highlighting even supposed to be supported through the protocol at all (in the current version)?

@vladdu No syntax highlighting is intentionally not defined by LSP. Syntax highlighting is best done by the editor host (sublime, vscode, eclipse, atom) and not the server.

What is on the backlog for the language server protocol is to add support for semantic coloring/classification of tokens.

[vladdu]

I see, thank you. I suppose it's for performance reasons, but I do highlighting in a different process and it is speedy enough. It probably requires the server on the same machine, though.

[smarr]

@egamma Server-side syntax highlighting would be a really great feature, especially for research languages that change frequently. Having to recreate a language grammar in something like the TextMate format can be non-trivial, error prone, and hard to maintain. Having a single 'truth' would make it much easier (at least for me).

[ghost]

@vladdu No syntax highlighting is intentionally not defined by LSP. Syntax highlighting is best done by the editor host (sublime, vscode, eclipse, atom) and not the server.

What is on the backlog for the language server protocol is to add support for semantic coloring/classification of tokens.

@egamma Can you say a bit more about what you mean by "semantic coloring/classification of tokens"?.

Is this intended to be additional highlighting and classification done after the colorization from the editor or is it something entirely separate? In other words, how is the tokenization performed? Are the token ranges computed by the server?

The reason I ask is because there are some languages I would like to use this protocol for that are nearly impossible to do correct highlighting for using the usual editor approach (e.g., textmate grammars). One example is the Agda language. Due to the very flexible syntax, highlighting information needs to be computed in part during type checking and _then_ sent to the editor.

I realize this has some drawbacks but it would still be nice to be able to officially support this use case in the protocol somehow. As @smarr mentions, out-of-band highlighting like this is becoming quite common in research languages. From a technical point of view, it doesn't seem like it should be hard to support although there may be some performance issues with very large files.

[vladdu]

• Even if the protocol supports this, it doesn't mean that servers have to implement it. Even if the server implements it, the clients aren't forced to use it (if it's too slow or incomplete or something) and can implement highlighting locally. In any case, the server needs to tokenize for its internal use, so there is no extra work involved.
• The API could be similar to "document symbols", with the addition of a range for which the tokenization is requested, for performance (clients usually know what part of the document is currently shown). Actually, I didn't get an answer to my original question, where I implied that "document symbols" feels like it should return the right information.

[svenefftinge]

The semantic coloring/token classification @egamma mentioned is exactly what you are asking for. It is covered in #18 and is meant to be an optional addition to the lexical syntax coloring that is done by the editor itself.

[vladdu]

Is it a problem if the protocol also allows for the lexical syntax coloring to be done by the server (optionally)? It could be an extra parameter that specifies what kind of highlighting to return.

[svenefftinge]

The proposal in #124 doesn't imply that the highlighting is done based on semantic information. So I think it could be used for lexical coloring, as well.