Monaco-editor: setTokensProvider unable to override existing tokenizers

Created on 17 May 2018 · 4Comments · Source: microsoft/monaco-editor

monaco-editor version: 0.13.1
Browser: N/A
OS: N/A

Behaviour

monaco.languages.setTokensProvider should override existing service with new one but that won't happen. This issue is affecting monaco-editor-textmate package.

monaco-editor-textmate/src/index.ts#L41-L54

The example below should set new tokenizer which tokenizes everything as comment.block.css

Repro

mkdir monaco-test
cd monaco-test
npm i monaco-editor
touch index.html

index.html

<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
</head>

<body>

    <div id="container" style="width:800px;height:600px;border:1px solid grey"></div>

    <script src="node_modules/monaco-editor/min/vs/loader.js"></script>
    <script>
        require.config({
            paths: {
                'vs': 'node_modules/monaco-editor/min/vs'
            }
        });

        function State() {
            this.clone = () => new State()
            this.equals = (other) => other === this
        }

        require(['vs/editor/editor.main'], function () {
            function setTokenizer() {
                monaco.languages.setTokensProvider('css', {
                    getInitialState: () => new State(),
                    tokenize: (line, state) => {
                        return {
                            endState: new State(),
                            tokens: [{
                                startIndex: 0,
                                scopes: 'comment.block.css'
                            }],
                        }
                    }
                })
            }

            // before model creation :(
            setTokenizer()
            var editor = monaco.editor.create(document.getElementById('container'), {
                value: [
                    'html, body {',
                    '\tmargin: 0;',
                    '}'
                ].join('\n'),
                language: 'css'
            });
            // after model creation :(
            setTokenizer()
        });
    </script>
</body>

</html>

*question

Source

NeekSandhu

👍1

Most helpful comment

The challenge here is that the built-in CSS language (shipped via monaco-languages) gets lazy-loaded and will register a tokenization provider after it gets loaded.

So, when a model with the language css is created, the listener will execute and it will create a <script> tag to load the css colorizer. Once that is loaded, it will set the tokenizer.

If you want to use monaco-editor as is, the only way would be to use a setTimeout or try to poll on <script> tags.

One cleaner solution would be to create a monaco-editor distribution which does not include monaco-languages (the basic colorization for several languages). This can be easily done by cloning this repo, running npm install, commenting out the relevant lines in /metadata.js, and running npm run release. The /release/ folder will have a monaco-editor without monaco-languages.

alexdima on 17 May 2018

👍2

All 4 comments

The challenge here is that the built-in CSS language (shipped via monaco-languages) gets lazy-loaded and will register a tokenization provider after it gets loaded.

So, when a model with the language css is created, the listener will execute and it will create a <script> tag to load the css colorizer. Once that is loaded, it will set the tokenizer.

If you want to use monaco-editor as is, the only way would be to use a setTimeout or try to poll on <script> tags.

alexdima on 17 May 2018

👍2

cc @epatpol - You might wanna express your thoughts on this matter

NeekSandhu on 17 May 2018

So if I understand correctly, the issue here is that the css tokenizer will override the tokenizer you could have set beforehand? If we call setTokensProvider after the css tokenizer has been loaded than it will work (hence the setTimeout and polling workarounds you mentionned).

I assume there's no event fired once that css tokenizer has been loaded right? Otherwise that would have been a better solution than a possible race condition or polling.

epatpol on 17 May 2018

I had the same issue with Javascript tokenizer. When typescript.worker.js was loaded my custom tokenizer was overridden to defaults. To fix this issue i used getJavaScriptWorker function, which returns a promise. Check out code sample below:

monaco.languages.typescript.getJavaScriptWorker().then(
      resp => {
        console.log('Worker has beed loaded')
        monaco.languages.setMonarchTokensProvider('javascript', yourCustomProvider)
      },
      err => {
        console.log('Worker load error: ', err)
      }
    )

Seems like it works. It still looks a little bit tricky for me, so i think there should be a better solution.