Vscode: Provide an API for advanced/semantic source highlighting

[seanmcbreen]

_From @Wosi on October 13, 2015 14:22_

I'm looking for a way to implement an advanced source highlighting
where a language service changes the color of words in a file. The result would look similar to the output Visual Studio creates in C# files.
The screenshot shows C# code in Visual Studio. The word "Customer" is highlighted differently based on its context:

To implement highlighting like this it's necassary to let a language service decide about the color of a word after the grammar based highlighter has done its job.
I would like to see an API for this in the future.

_Copied from original issue: Microsoft/vscode-extensionbuilders#14_

[seanmcbreen]

_From @egamma on October 13, 2015 14:40_

The request makes sense OmniSharp supports semantic coloring and we currently have no API to expose this support.

[seanmcbreen]

_From @Krzysztof-Cieslak on October 13, 2015 14:43_

:+1: Same can be used for F# - Visual F# Power Tools for VS and F# support in Xamarin Studio are both using semantic highlighting.

[jrieken]

We are close but we are missing meta-colours ala _the-color-of-a-keyword_

[Krzysztof-Cieslak]

Any progress with it? :)

[vilic]

Desperately want to know whether there've been any progress on this.

[NPatch]

Me2. Could you or will you expose it as json settings kinda like settings.json or keybindings.json?

[DustinCampbell]

Would love to see this happen. Honestly, it'd be great to have a way to provide colorization _without_ a textmate grammar. The C# extension has incredibly accurate colorization information already available to it in OmniSharp (using the same Roslyn APIs that are used to colorize VS).

[nfantone]

This would be extra nice. And if what @jrieken said a year ago still holds true, it shouldn't be extra hard to implement.

[troshko111]

Opened similar issue before found this one, since Atom is already doing this to some extent (semantic highlighting based on omnisharp data, not regexp tm grammar) I would love If you guys could prioritize this higher and move from backlog, imo this is huge improvement to the productivity and can be a true killer feature

[ferologics]

+1 ⌨️

[kobalicek]

Not just that, customizing some keywords based on project would also be amazing, there is a lot of C++ code that relies on macros or uses some specifics, and ability to highlight that "easily" would be nice.

[Wosi]

Do the changes made for https://github.com/Microsoft/vscode/issues/18317 bring us closer to semantic highlighting?

[kobalicek]

Suppose I want to highlight my own keywords / variables, how can I test this?

My solution https://github.com/kobalicek/reloaded-cpp is just a hack at the moment, rewriting it to use existing C++ parser and just hooking it to my own symbol highlighter would be the way.

[tezine]

+1 to solve this. The lack of C# semantic highlighting is the the only think that keeps me away from vscode. :-(

[lemures-t]

any new update ? &_&

[Jinnified]

so any news on this? vs was awesome except this part

[isaacabraham]

Ping. Would love it for F#.

[FacundoGFlores]

It would be useful if vscode supports semantic highlighting. An example is:

https://marketplace.visualstudio.com/items?itemName=AndreasReischuck.SemanticColorizer

[msafi]

If you look at the 2017 roadmap for VS Code, you can see that adding semantic highlighting extensibility is part of the roadmap: https://github.com/Microsoft/vscode/wiki/Roadmap#for-language-extension-authors

So hopefully we'll get it before the end of this year 🤞

[JohnArcher]

@auchenberg brought me here (thanks!).

If I am not wrong it seems there are slightly different types or understandings of the term "semantic highlighting". I want to use it in JavaScript, and my main wish here is that every identifier/variable has it's own unique color, consistently in the whole project.

With Sublime Text I used the great plugin Colorcoder for doing that. The author refers to this article by Evan Brooks where you can also see a screenshot of how it should look like: https://medium.com/@evnbr/coding-in-color-3a6db2743a1e

Another resource of information is this article for the Atom editor: https://atom.io/packages/language-javascript-semantic

So, I would be super happy if this feature could find a way to VSC!

[sharedprophet]

@JohnArcher I maintain the semanticolor package for Atom, and have been following this thread waiting to be able to port it to VSCode. So I'm right there with you. 😄

[oktonion]

I am with you. One of the most needed features is highlighting of user defined types. Waiting the implementation for C++/C.

[RacerDelux]

Going to be honest, this is the only thing keeping me from using vs code... I am dyslexic so semantic highlighting is a must have.

[iamstelios]

Any updates?

[ghost]

Can we chalk out the plan and issue a checkbox'd TODO list in first comment?

[Jinnified]

if you look at vscode road map for 2018, semantic highlighting was put down as a stretch goal, which is a pretty low priority if you ask me, so realistically you probably won't see it's coming till 2019, or even 2020

https://github.com/Microsoft/vscode/wiki/Roadmap

[E4N]

Please make this a thing, this is also the only issue that keeps me away from Visual Studio Code. It just doesn't feel right in the current state.

[Sawtaytoes]

Sublime Text actually just added support for Semantic coloring in their latest 3.1 update. They call it hashed syntax and appear to have used the same algorithm as the Colorcoded plugin.

https://www.sublimetext.com/blog/articles/sublime-text-3-point-1

It looks almost as easy as turning the feature on or off per color scheme.

[fabiospampinato]

I've recently requested an API for getting the current decorations applied to the document, and it has been marked as duplicate of this.

Hopefully implementing this API should be trivial, after all the editor already knows what decorations in which ranges to apply, and we won't have to wait years.

[mattacosta]

I just wanted to point out that there are two features being requested here. The first is a feature that would let the tokenizer do its job and essentially recolor the token it generates, and the second is an extension point to replace the TextMate tokenizer (since regex-based tokenizers can be inaccurate).

In essence, #1967 should not have been merged into this issue.

[sharedprophet]

@mattacosta I'm not sure that's entirely accurate. I think what's being requested is the ability to use semantic information to determine coloring of tokens. The implementation details aren't prescribed here.

Letting the tokenizer do its job and then recoloring the results works for some cases (i.e. color-coded variable names) but not all (i.e. muting C++ #if blocks that are disabled). Replacing the TextMate tokenizer could allow for all cases but potentially make it difficult to implement something like semanticolor that augments virtually any installed language package with color-coded variable name functionality.

Ideally, I think, we want some kind of ability to convey syntax tree info to/from/among language servers and use that for determining scopes. I think that would cover all use cases. Since that may be a tall order, enabling language servers to provide scope info (replacing the TextMate scopes for that language) would be the next best thing.

[mattacosta]

@sharedprophet That's not the point I was trying to make. It was that while you can give a highlighter semantic information for the purpose of recoloring¹, you cannot give a highlighter semantic information for the purpose of correcting inaccurate tokenization. So while there are ways to implement semantic coloring (at least partially) using either tokenizer, only a tokenization API could solve the second issue.

¹ That is to say providing some input in order to get a different output value, either at an early stage for preprocessor directives or at a later stage for syntax trees. In both cases the tokenizer will do its job.

[MathiasMagnus]

I'd love to use VS Code prime time for C++ development, but the lack of proper C/C++ coloring is a deal breaker. I'm currently refactoring a huge CMake-ified C code base and it's crucial to get proper coloring in order to see if a given typedef/macro is visible in the given compilation unit (and perhaps see what it expands to in the given configuration). Dimming inactive code regions is a must. There's too much text to hover over and see if IntelliSense lights up. Coloring is infinitely faster.

I honestly cannot imagine how other languages get by in VS Code without this ability.

[dannymcgee]

Happy New Year!

+1 for this. I'm a little bit obsessed with syntax highlighting and it drives me absolutely bonkers that Intellisense recognizes this as a class, but the syntax highlighter just sees it as a generic variable.

[MathiasMagnus]

@dannymcgee You know what drives me bonkers? That collaborative code editing with remoting an entire session over the wire, even across IDEs of different flavors is implemented before text coloring. 🤯

[bn-l]

@dannymcgee You know what drives me bonkers? That collaborative code editing with remoting an entire session over the wire, even across IDEs of different flavors is implemented before text coloring. 🤯

QOL improvements are hard to point at in meetings and say "hey, look, I did that!"--no matter how useful or how requested they are.

I honestly cannot imagine how other languages get by in VS Code without this ability.

yup.

[MathiasMagnus]

@bn-l I can imagine, but isn't the entire purpose of an IDE is QOL? :) I write template-heavy code in C++, or I'm porting macro-heavy C code, and coloring something with the given compiler options makes all the difference in the world. In fact, I'd gladly free that 8 GB of space my big VS install takes and never look back if this feature were implemented. Until then, I'll keep enjoying cross-platform squigglies which probably won't come to VS Code, like ever.

[oktonion]

Could we just take a second to appreciate the lots of attention to the issue that is just hanging there like 3+ years with more than 260 likes and is a request for basic feature that any Intellisense-like code highlighter has. 👍🏼

[RacerDelux]

@oktonion The lack of this feature is the ONLY reason I do not use vscode. I have just gotten so used to it, and it is a very helpful feature.

[mdsitton]

Yeah same every time I come around to try out vscode. I run into this issue and end up searching for themes that work well with my language of choice for hours before giving up and moving back to whatever other editor(usually sublime text).

[KamasamaK]

Since people continue to post here asking for this and in case anyone didn't see, this is already on the Roadmap for 2019.

[kerams]

@KamasamaK It's been on there since at least 2017 (as semantic highlighting).

[MathiasMagnus]

@mdsitton I do the same. On a fresh install I only had the MSVC build tools and Code installed as my primary (and only editor). I managed for half a year when cross-platform builds and squiggles arrived for VS and I changed back. It's light years ahead. I could live with having to fire up two Code instances (one on Windows and one inside WSL), but the semantic coloring has been in VS since... forever? Not having coloring, inferior debugging capabilities, one of the cutting-edge CMake goodies... It's too much to give up when I'm already on Windows.

[KamasamaK]

@kerams You're right. I was about to say it was promoted from "stretch goal", which is how it was classified since the initial 2018 roadmap was published, but to be honest now I'm not sure since I see _nothing_ is classified as a stretch goal right now.

[mdsitton]

Good to hear that it's finally on the roadmap. I also finally found a theme that handles C++ and C# semi acceptably "Kary Pro Colors". Looks like I may be finally trying it out for a more extended period of time for once.

Still looking forwards to this being implemented!

[SamB]

The standalone variant of the editor, "Monaco", provides something sort of like ITokenizationSupport: https://microsoft.github.io/monaco-editor/api/modules/monaco.languages.html#settokensprovider

We might want something a bit different from Monaco here, though, because:

1. Maybe we don't want to have to choose between "fast" and "with named scopes" on a per-language basis
   
   
   
   1. The "EncodedLanguageId" bits seem ... fiddly
      
      
      
      
      
      • What can we do with these other than set them to the wrong value?
      
      
      
   
   
   2. I imagine that doing an RPC per line might be somewhat expensive. The JavaScript API could easily abstract over whatever optimizations might be appropriate, but the LSP clearly cannot. (And you're gonna need the LSP support to talk to Roslyn, right?)
   
   
   3. Of course, LSPs will also need some source locations so they can find the code in their data structures...
   
   

I'm thinking injection grammars would probably not be viable here, but embeddedLanguages shouldn't be too hard.

[gulshan]

This tree-sitter based syntax hailighting seems pretty advanced-
https://github.com/microsoft/vscode/issues/50140#issuecomment-493786052

[gulshan]

Relevant #77140 and #77133

[dannymcgee]

Just noticed some blessed soul added the joggy boi emoji to this feature on the roadmap and it fills my heart with hope. I believe in you, VS Code.

[HighCommander4]

Just wanted to mention that, while we wait for the VSCode team to release an official API for semantic highlighting, in the C++ community we've been using editor.setDecorations() to implement semantic highlighting in extensions like vscode-clangd and vscode-cquery, and it has been working reasonably well.

[aeschli]

We have started working on a vscode.d.ts API for semantic highlighting.
Please have a look at https://github.com/microsoft/vscode/issues/77133. It's a a proposal on how to classify tokens as well as how to do theming. It would be great if you can give feedback there if we're heading in the right direction or if there's something missing.

There will be an other proposal on the provider API.

[alexdima]

Done via https://github.com/microsoft/vscode/issues/86415