Vscode: [grammars] provide alternative to TextMate grammars

[tomq42]

TextMate isn't sufficient for many languages.

We have been integrating in to the lower level, in the src/vs/languages directory and using Modes.IState and supports.TokenisationSupport. There needs to be a way of writing an extension that can do this, which at least currently there doesn't seem to be,

Thanks.

[Tyriar]

@aeschli I'm interested in the thinking behind moving away from our own tokenization in favor of tmLanguages.

[aeschli]

@Tyriar For performance reasons we want the tokenizers to run in the render process. As we don't want user code to run in the render process we went for declarative tokenizers.
First we had Monarch support as well, but before the API deadline decided to go fully for text mate to keep the API simple.
Also, in order for our theming support to work well, we want all tokenizers to emit TextMate tokens.

We are aware of the limitations and problems of TextMate, and we are open to allow other types of tokenizers, but no work is planned in this area at the moment.

[tomq42]

That's a shame. It's also slightly "unfair". In the sense that it means that Microsoft can write language modes that can do things that other people can't....

There would be zero chance of a pull request to the core of vscode being accepted for a new language mode, so we are left being unable to write a sensible language mode for vscode. We have no problem in Eclipse, there's no problem in writing similar things there, they have no issue with things running in the render thread.

At least in Monarch there was a "pop state" facility, which as far as I know has no equivalent in TextMate. In Monarch you could shift states to an explicit state, and then "pop". So you could write "subroutines". The facility made it least _possible_ to write our language mode in Monarch, even if it was _much_ harder work than doing it the low level way, which is what we ended up doing.

[tambry]

Tokenizing languages where a single token might be split onto multiple lines is near impossible (without very complicated workarounds) using TextMate. (see https://github.com/Microsoft/vscode-textmate/issues/32)
Monarch would be a huge improvement, allowing better language support for more complicated/nuanced languages. Though being able to write a tokenizer using an API would be even better.

[EvgeniyPeshkov]

Hello everyone. I've developed and published syntax highlighting extension based on Tree-Sitter. It provides universal syntax coloring engine for almost any programming language (currently, C and C++ are supported OOTB). Constructing entire syntax tree, Tree-sitter efficiently overcomes all limitations of built-in TextMate grammars. It's very easy to add support for a new language. I'm planning to write HowTo in the next couple of days, but you can figure it out from source code, that is very simple and straightforward. Contributions are welcome. I've been using it by myself for a month, so I suppose it's ready for public use. At least extension can be useful until VSCode core provides stronger syntax parser.

You can install it from VSCode Marketplace.
Or download .vsix package from GitHub page and install it manually.
Please note, that extension published in VS Code Marketplace will only work in Windows-x64.
For other operating systems, please download pre-compiled .vsix package.
This will be fixed in the near future with one of the next updates.
Alternatively, you can build extension from sources.