Vscode: [semantic] proposals for new standard semantic token types

[aeschli]

The new semantic token provider API comes with a list of standard token types and modifiers.
https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-token-classification

These type serve as a base across languages and by having all/most providers using theme will make easier to write theming rules across languages.

That said, semantic token providers are not forced to stick to the standard, but can add new types/modifiers, or extend existing types as seen in the doc.

This issue is to collect proposals for new types and modifiers. When making a suggestion, please add a description and a small code sample. If it exists, name the corresponding TextMate scope.

The standard token types should be be applicable across multiple languages and be useful for theming. We want to keep the set of standard tokens consistent and coherent.

Proposed types:

| Identifier (extends) | Description | Ref | Sample | TextMate scope |
| ------------- |:-------------:| -----:| -----:| -----:|
| importKeyword (keyword) | keywords related to imports/includes | (1) | __import__ * from x | |
| modifierKeyword (keyword) | keywords describing a modifier | (1) | __private__ void foo(); | |
| docComment (comment) | documentation comments | #96712 | /** */ | |

Proposed modifiers:

| Identifier | Description | Ref | Sample |
| ------------- |:-------------:| -----:| -----:|
| unused | annotated all unused symbols | (2) | let __unusedVariable__; |

References:
(1) https://github.com/microsoft/language-server-protocol/issues/968
(2) https://github.com/microsoft/vscode-languageserver-node/issues/604

[kjeremy]

@matklad

[matklad]

Types

Identifier(extenrs) | Description | Sample
--- | --- | ---
attribute/annotation | syntax for attributes and annotations | #[test] fn smoke() {}, @Test void method() {}
builtinType(type) | non user-defined types | i32, long long int
typeAlias(type) | name of type aliases and typedefs | type Unit = (); typedef int int_alias
union(type) | name of C-style untagged unions| union U { a: i32, b: f32}, union u { int a; float b; };

Modifiers

Identifier(extenrs) | Description | Sample
--- | --- | ---
unresolved | for all unresolved symbols (such symbols should also have a corresponding diagnostics) | let my_resolved = 92; my_un_resolved;

All extra tokens&types defined by rust-analyzer: https://github.com/rust-analyzer/rust-analyzer/blob/9cb55966fe0fee791072f275ac55b90b8ee13e32/editors/code/package.json#L522-L572

Hm, actually, unresolved might want to be a type, rather than modifier. It feels similar to unused, but if something is unresolved you, by definition, can't say which type it is. It is type in rust-analyzer.

[etc0de]

While I think it makes sense to add in things like typeAlias and union since a larger subset of system level languages is for sure likely to use this, these still sound specific enough that I'm wondering if a theme author may not give a different shade to all of these or have trouble figuring out what shade to give: imagine somebody who just did some Python scripting and wants to make a new theme, they'll probably have a hard time judging if something like union needs a separate color and to what it should be close in shade. It's probably enough to color type, but I'm not sure how immediately obvious that is to a theme author...?

As a result, I'm wondering: should the resulting LSP semantic tokens formal specification also include some guidance on which token types and modifier types should be considered as important to have different colors? To sort of establish a baseline on what a theme is expected to cover. Or would the expectation really be that all theme authors pick something for a relatively specific thing like a union in particular, or that they know that it's not required?

While this doesn't directly matter to the protocol implementation on either side of course, I feel like it could probably be pretty relevant for how it all plays together in the end to give some guidance for theme authors here.

[woody77]

Identifier | Description | Ref | Sample
-- | -- | -- | --
documentation | for tokens that are part of documentation | (1) | javadoc, rust docs, doxygen
disabled | for tokens that are turned off by compilation flags. | | #ifdef(foo) ....... #endif in c/c++, #[cfg(foo)] in rust
example/sample | sample code in comments (doc or code) | | https://doc.rust-lang.org/src/std/time.rs.html#175
markdown | these types are markdown (in e.g. comments) | | see above

Note that "documentation" exists today, but without any documentation as to when it's to be used, and what semantic meaning it has, so this is a proposal that comment.documentation would apply to:

• javadoc
• rust doc comments
• doxygen in c++
• other languages that specifically call out "doc comments" separate from "code comments"

disabled is something that I see VSCode do with C++ and #ifdefs, but doesn't seem to be via the semantic types. I see #ifdef'd out code with a muted set of colors (50% transparency?) but the type inspector says it should be the same as not-disabled code.

example or sample sample code in doc comments is parsed, semantically highlighted, and flagged for correctness in some IDEs, but is rendered by the themes in a mix of comment and normal formatting (say normal colors, but in italics).

markdown may be best handled in other ways, but e.g. Rust uses Markdown type headings and links in it's doc comments, but maybe the better way to handle that is is by marking them as markdown types (heading, links, etc.), and applying documentation as the modifier.

References:
(1) https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-token-classification

[DanTup]

Should there be a token type for things like TypeScript's decorators (Dart has similar called annotations):

// TypeScript

function foo() {}

@foo
function bar() {}

// Dart

@mustCallSuper
void foo() {}

I don't think any of the existing ones fit?

[aeschli]

Yes, I agree that a token type annotation would be useful.

[TylerLeonhardt]

We have been talking about annotation in https://github.com/microsoft/language-server-protocol/issues/1067

[woody77]

rust-analyzer also provides something similar (called attribute there, as both a token type and a modifier, since there can be functions within them:

The derive is a function.attribute, and the rest of the item has attribute token type. Although the Debug should maybe be marked as an interface ("trait" in Rust).

[DanTup]

There are types for string and number so should there also be one for boolean?

[dannymcgee]

Hate to resurrect a stale comment thread, but hey, how bout that decorator/attribute/annotation token. :)

I love semantic highlighting but it's killing me that the @ symbol is the only thing distinguishing my function decorators from the functions they're decorating — it makes the code quite a bit less legible.

[dbaeumer]

@aeschli do you have any plans to extend this in VS Code?