Che: Support for Language Server Protocol

[svenefftinge]

##The VSCode team has recently published the version 2.0 of their language-server protocol. The goal is to define common API for editors such as VSCode or Che to interact with language services like

• TypeScript language services
• OmniSharp (C#)
• Tern or the recently announced Orion JS services
• Ensime (for Scala and Java)
• and probably more (please add them to this ticket).

We have started using this protocol for communication between Che and Xtext and would like to move the Che part here and collaborate on the development. @evidolob: could you create a branch, against which we can make a PR with the basic projects we have created so far?

Tasks in this Epic includes part 1 of implementation:

• [x] Dependency: Agent deployment and management #1823
• [x] Language Server Protocol: Respect Language Server capabilities on IDE client #1789
• [x] Shows errors and warnings inside editor.
• [x] Intellisense - auto-completion in a context aware setting
• [x] Goto Definition #1423
• [x] Language Server Protocol: Find Reference #1421
• [x] Language Server Protocol: Implement client part for "Go To Symbol" feature #1791
• [x] Workspace symbols to list project-wide symbols matching a query string.
• [x] Language Server Protocol: Implement document formatting #1788
• [x] Language Server Protocol: Hover feature #1800
• [x] Language Server Protocol: Signature Help feature #1801
• [x] Language Server Protocol: Refactor Symbol navigation UI #1809
• [x] Language Server Protocol: Support incremental updates #1425

Language Server Agents.

LS agents are injected into workspace configuration:
“agents” : [“org.eclipse.che.ls.json”, “org.eclipse.che.ls.csharp”]
LS agents and LS binaries are located at remote storage. For the time being it is Codenvy Update Server. Agents are available by the following urls:
https://codenvy.com/update/repository/public/download/org.eclipse.che.ls.json
https://codenvy.com/update/repository/public/download/org.eclipse.che.ls.csharp
When workspace is created agents are being applied. Basically it means agents script execution over running instance. After that LS can be started on demand when user opens files associated with installed LS.

• [x] Dependency: Agent deployment and management #1823

Bug:

Code Assistant and Code Validation don't work for C# #2509

Initialize process:

• User opens file with LS support.
• LS Editor provider calls vs-agent to initialize LS.
• LanguageRegistryService on vs-agent calls LanguageServerRegistry to find and start LS.
• LanguageServerRegistry launch LS.
• When new LS is launched and initialized, LS capabilities passed to IDE.
• IDE use LS capabilities to create editor and open file.
• [x] Language Server Protocol: Respect Language Server capabilities on IDE client #1789
  

  Diagnostics

RI: C#, JSON

• [x] Shows errors and warnings inside editor.

Completion (IntelliSense)

RI: C#, JSON

• [x] Intellisense - auto-completion in a context aware setting

Is a context-aware code completion feature in some programming environments that speeds up the process of coding applications by reducing typos and other common mistakes. Attempts to do this are usually done through auto completion popups when typing, querying parameters of functions, query hints related to syntax errors, etc.

Goto Definition

RI: C#

• [x] Goto Definition #1423

Perform search for symbol definition and allows to navigate.

Find References

RI: C#

• [x] Language Server Protocol: Find Reference #1421

Search project-wide references for the symbol denoted by the given text

Document Symbols

RI: JSON

• [x] Language Server Protocol: Implement client part for "Go To Symbol" feature #1791

Allow to view and navigate document symbols. In most cases it’s can be described as flat outline

Workspace Symbols

RI: C#

• [x] Workspace symbox to list project-wide symbols matching a query string.
  Workspace symbols is the list project-wide symbols matching the query string. It’s allow to navigate not only for file or class name but for method or field.

Document Formatting

RI: JSON

• [x] Language Server Protocol: Implement document formatting #1788

Format all document.

Document Range Formatting

RI: C#, JSON

• [x] Language Server Protocol: Implement document formatting #1788

Format selected range in document.

Document on Type Formatting

RI: C#

• [x] Language Server Protocol: Implement document formatting #1788

Automatically invoke formatting after special character typed.

Hover

RI: C#, JSON

• [x] Language Server Protocol: Hover feature #1800

The hover shows useful information, such as types of symbols, or, in the case of CSS below, the shape of the HTML that would match a certain CSS rule:

Or for java hover may show javadoc:

Signature Help

RI: C#

• [x] Language Server Protocol: Signature Help feature #1801

Shows signature information at a given cursor position. For strongly typed programming languages it’s may shows method parameters types:

[svenefftinge]

Also we have defined the interfaces of the protocol in Java. And made it available here : https://github.com/TypeFox/ls-api

[evidolob]

I create branch named "language-server" https://github.com/eclipse/che/tree/language-server

[nadako]

Nice, this is exactly what I had in mind when I started developing haxe-languageserver - that not only VSCode can implement and make use of that protocol, but any editor/IDE! Looking forward for it to be implemented in Eclipse, so I could try adding Haxe support for it, using the language server.

[TylerJewell]

@tolusha @evidolob @gazarenkov @vkuznyetsov - please, can we link in any _dependency_ issues related to agent deployment and end user UX that are related to how a user will add / remove language servers within a workspace? without seeing the full end user workflow issues that we need to handle, it will be hard to understand what the real definition of done is here.

[tolusha]

https://github.com/eclipse/che/issues/1823

[kaloyan-raev]

I want to prototype a PHP IntelliSense support in Che. I want to take advantage of the Crane language server, which provides the PHP IntelliSense in VSCode: https://github.com/HvyIndustries/crane

As far as I understand, the support for the language server protocol in Che is currently a work in progress and it is not available in the master branch yet.

Could you tell me how to setup my dev environment, so I can start experimenting?

[TylerJewell]

@evidolob @tolusha @vparfonov - FYI - can we document some setup tasks for the community to test our work as we go along?

[kaloyan-raev]

I was able to build the branch from the root pom.xml after deleting a couple of test files that were causing compilation errors.

Now I am exploring how the C# support is implemented. I have a Java workspace in my local Che. I created a new C# file. If I open it in the editor, I get an "Initializing Language Server for cs" progress message and the IDE freezes.

It looks like the C# language server cannot be started. Any hints how to troubleshoot?

[kaloyan-raev]

Same story with the JSON language server. There the code for launching the server is simpler and I found that the /home/user/che-agents/ls-json/launch.sh script is missing. I guess this is related to #2032 and these launch.sh are not available yet...

[tolusha]

@TylerJewell
What setup tasks do you mean? Everything is in branch, except @evidolob code part.

@kaloyan-raev
I am checking...

[tolusha]

@kaloyan-raev
I've just fixed agents. No need to pull changes.
json ls requires only a few seconds to be initialized, but c# requires several minutes.
c# ls works properly if you have project.json file in your project.

I tested everything on codenvy/debian_jdk8 container

[kaloyan-raev]

@tolusha Thanks for the magic you have done! I created new project and the Json language server now starts (I would say immediately).

How do I check if the language server works correctly? I don't see any meaningful code assist, and it's hard for me to imagine any for a generic json file.

Does the validation come from the language server too?

[tolusha]

@kaloyan-raev
To use code assist just add any json scheme into your file and press Ctrl+Space
For instance:
{
"$schema": "http://json-schema.org/draft-04/schema#"
}

[kaloyan-raev]

@tolusha Thanks! This works.

I noticed a few issue with the code assist. Perhaps, you already know them:

• The list of proposals is not sorted.
• The list of proposal is not filtered. Regardless the input I type, the proposals list always contains all available proposals and always the first one is selected.
• There is some {{}}} garbage inserted when applying the code assist.

There is also a concept issue with the json editor. I've never seen $schema to be declared for popular json files like bower.json and composer.json. Yet users would expect code assist and validation to work for such files. Is there a way for Che to tell the json language server what schema to use, if there is no $schema declared in the file?

[tolusha]

@kaloyan-raev
Thx.
We aware of them.

[gazarenkov]

Splitted epic moving its second part to #2109 including:

Language Server Protocol: ShowMessage Notification feature #1793
Language Server Protocol: ShowMessage Request feature #1794
Language Server Protocol: Telemetry Notification #1796
Language Server Protocol: DidChangeConfiguration Notification feature #1798
Language Server Protocol: DidChangeWatchedFiles Notification #1799
Language Server Protocol: Document Highlights #1802
Language Server Protocol: Rename feature #1806
Language Server Protocol: Push notification to client on new language server #1427
Language Server Protocol: Ensure order of Events #1426

[TylerJewell]

@slemeur - on the "Document Symbols" - can you please take a look at the UX for the highlight of the typed characters. It is currently a weird yellow box and ends up appearing unnecessarily heavy to the end users. Please identify an improvement here that we can apply.

[kaloyan-raev]

After I've done some refactoring on the Crane language server (https://github.com/HvyIndustries/crane/pull/143), I am now able to run it in Che and get code completion inside the PHP editor.

I prepared PR #2282 with the changes. Please, review and merge it to the language-server branch.

The PR also contains attachments with the agent that must be uploaded to the Codenvy Update Server.

[kaloyan-raev]

BTW, there is a new PHP language server started a few days ago: https://github.com/felixfbecker/php-language-server.

It is implemented in pure PHP. My next goal is to try it out with Che.

Is it be possible to have more than one language servers for the same editor at the same time?

For the moment, the two language server available for PHP does overlap in features, but complement each other. So, it would make sense to have them both available.

[TylerJewell]

Very interesting work! I do not believe it will be possible - for now - to have more than one language server activated for a source file of a particular type / extension. I do not see us working to try and provide support for multiple LS at the same time for a single file. I think we would require users to register / unregister one before activating another. This would be the common scenario, I would expect.

Also - we will be publishing docs later this week or next week around how to author agents, which are ways to package up language servers into hot-deployable services within a workspace. This will be a nice packaging. We haven't done our own language servers this way but need to.

[kaloyan-raev]

Good to know this limitation for only a single language server per file.

However, I wonder how we can address files with multiple languages this way. Imagine a single HTML file having all of HTML, JavaScript, CSS, PHP and maybe other code.

[kaloyan-raev]

Also - let's pick even a single language with a large community, e.g. JavaScript. There are bazillions of JavaScript libraries and frameworks that may be supported by different language servers. We can have the jQuery support in one language server, but the AngularJS in another.

I don't see it happening to have the complete JavaScript support in a single monster language server. Same for other popular programming languages.

If the goal is to reuse the language servers created by the wide community, I think it is crucial to have a design that supports multiple language servers per single file.

[gazarenkov]

Hi,

It is interesting practical case and we have not tried to implement something like this yet... of course.

I really doubt LS specs have or will have something like requirements for activation LS for particular "embedded" language (like JS inside HTML) - like recognizing a file fragment, it is always triggers the file IMO (by extension).

So the most probably and reasonable (indeed a single monster LS with all possible options looks ugly) option to me to have it as a responsibility of "main" LS to activate "embedded" LSs.
So as for this example HTML LS would act as a "proxy" to recognize JS syntax and call JS LS or just try to call all dependent until have some "good response" and send it back to the client.

I would say our Agent concept applying to this case suites quite well for delivering "embedded" languages LSs as a dependency of main LS. I.e. HTML LS Agent may declare that it requires JS LS Agent as a dependency and so both of them should be installed.

May be pretty similar for JS libraries.

Disclaimer: take it as a random thoughts, I am not really a tech expert in LS
@evidolob

WDYT?

[kaloyan-raev]

This main HTML LS proxying to dependent LS might be a solution. However, the HTML LS must be extensible, so additional dependent LS can be injected together with the strategy to recognize the respective embedded code. It is more or less clear that CSS and JS a often embedded in HTML, but it might be JSP, PHP or template scripts. Hardcoding all this knowledge in the HTML LS does not look flexible.

It also does not respond to the issue to have more than one LS for the same language, e.g. adding support for jQuery, AngularJS or other frameworks.

Perhaps, we should take a look how VS Code handles this (if it is at all). I'll experiment in the next weeks and will report my findings here.

[vazexqi]

How can I get started on trying out the language server feature in Che? I've checked out the language-server branch and built (ignoring tests). And I've launched it by doing

docker run --rm -t \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e CHE_DATA_FOLDER=/Users/nchen/che \
    -e CHE_HOST_IP=10.10.42.22 \
    -e /Users/nchen/Development/my-eclipse-che-developer/che/assembly/assembly-main/target/eclipse-che-4.8.0-SNAPSHOT/eclipse-che-4.8.0-SNAPSHOT \
    codenvy/che-launcher:nightly start

I launched a new workspace with a .json file but I don't see any of the features at the "Assistant" menu item. Invoking ctrl+space also doesn't do anything.

How can I check if the language server has been initialized? I don't see any errors and my workspace is working for the sample Node.js project.

I've attached a screenshot of what it looks like.

[kaloyan-raev]

Che does not send the json/schemaAssociations notification to the language server yet. Hence you need to declare the schema explicitly in the json file.

Add the following:
"$schema": "http://json.schemastore.org/package"

Now you should have code completion working when hitting Ctrl+Space.

Note the current limitations/issues I reported a few comments above: https://github.com/eclipse/che/issues/1287#issuecomment-239362279

[vazexqi]

@kaloyan-raev - I did try that as well but I still don't get the auto-completion (or any of the other features of the language server protocol like symbols).

I also don't see much network activity between hitting ctrl+space. Is it supposed to be sending some request back to the server?

[kaloyan-raev]

It works for me this way, but I run Che differently.

I run it with the bin/che.sh script executed from the che/assembly/assembly-main/target/eclipse-che-4.8.0-SNAPSHOT/eclipse-che-4.8.0-SNAPSHOT/ folder.

The language server is started as a child process of the che agent (the client). The communication is done via a pipe between the standard input/output streams of the two processes. So, no TCP networking.

[vazexqi]

@kaloyan-raev - Yes, I would like to run it using bin/che.sh as well but I've hit #2313 on my Mac (just filed that bug). So I'm using the docker launch method as my workaround.

I do see the following in the logs, so I am pretty sure the agents are running.

2016-09-02 10:03:14,644[kspaceManager-0]  [INFO ] [c.p.d.m.DockerInstanceProvider 596]  - Machine with id 'machinewbevysr1enau40tb' backed by container '2c2d5aec62aca6065cc3d6fc622ce07d2a6c0e61166ae964716c4eebb4231d68' has been deployed on node 'localhost'
2016-09-02 10:03:15,160[kspaceManager-0]  [INFO ] [.e.c.p.d.m.DockerAgentsApplier 101]  - Starting org.eclipse.che.ls.json agent
2016-09-02 10:03:26,010[kspaceManager-0]  [INFO ] [.e.c.p.d.m.DockerAgentsApplier 101]  - Starting org.eclipse.che.ls.csharp agent

I also see that the agents have been downloaded to the workspace at /home/user/che-agents/

Any other way to find out more information on why it's failing?

[kaloyan-raev]

Can you check in the Terminal view if there is a node process running? The JSON language server is executed in a Node process.

Execute a command like ps aux | grep node.

[vazexqi]

I got this working after sticking consistently with the way that was suggested in #2313 with the latest version of the language-protocol branch.

I have a couple of questions (if this is not the right place to ask, please re-direct me):

1. Are there any concerted efforts between Che's implementation of the language server protocol (LSP) and https://github.com/eclipselabs/eclipse-language-service? I know both of them are using typefox-ls underneath. But what about the presentation/filtering/auto-completion logic? I'm asking because I'm interested in working on LSP in both classic Eclipse and Che and would like to know what might be a good way moving forward.
2. I notice that you have to post the vscode implementation at https://codenvy.com/update/repository/public/download/org.eclipse.che.ls.json.binaries. What's the origin of this? Is it from the vscode/extensions/json/server repo or from the vscode-json-languageservice repo? Any modifications necessary?
3. Is there a list of work items to track the current LSP bugs? I noticed that auto-completion for JSON, at least, was buggy with it not filtering based on prefix, suggesting {{somevalue}} wrapped in the double parentheses, etc. Brought up in https://github.com/eclipse/che/issues/1287#issuecomment-239362279 as well by @kaloyan-raev
4. What is the timeline for LSP support in Che?

[TylerJewell]

@vazexqi - Hi Nick. This is really amazing. This is the thread where we are tracking all of the 1st phase work to be done on the language server protocol. There is another epic for the second phase work.

To your questions:

1: All of the people that are working on the various LSP items were in a hackathon together in Switzerland. There is the most coordination on the typefox API and the Java server implementation. I am not aware of the various client products coordinating how they implement things in the editor. That is going in slightly different directions, but I think it will be easy to consolidate different editor implementations over time. It is more important to verify that different language servers work similarly in different products. I am also very interested in getting coordination around a public registry for language servers, and IBM, Red Hat, and Codenvy have all shown interest in helping there. But i don't see this work starting before 2017.

2:
@evidolob @vparfonov can answer a bit better than me. But I think this is our nexus repository where we are hosting the packages that can be downloaded by workspaces when they are started. It is our intention for the first release with Che to package up LS as "agents" which are a new concept that we just merged last week. Agents are packages of software that can be dynamically injected or removed from a workspace runtime after it is started. We used to have a lot of stuff that we place into a workspace to make it remotable like our terminal and SSH. Now we package this as an agent and dynamically deploy it during the workspace boot process. LS will be done the same way and it allows us to deploy them as part of a stack definition or when users ask for it. As an initial implementation all of our LS agents will have to be packaged and hosted on the Che nexus which codenvy manages. But it would be better if we had a LS registry that is similar to how maven central works at the eclipse foundation. That is a step 2 thing.

3:
I'd like to have all of the bugs for this implementation logged as separate issues and then added as cross-link into this issue. They will get resolved and included as part of the first release. I want to organize all of the check-box items on the language server protocol on this epic a bit better - so will move them to the top of the issue for tracking, and then will add in a check box for each bug cross-linked.

4: LSP support is going to be part of the 5.0 major release, targeted for this fall. When all of the issues in this epic are completed, we will release it. Given our current progression - it's maybe only a couple sprints left, so maybe by end of October.

[StevenDoesStuffs]

Will this include the typescript lang services? I see the part up top, but do not know if it's a priority.

[TylerJewell]

That language server for typescript, unfortunately, is not open source. I believe. The last time I checked was a couple months. So I don't believe we can include it unless Microsoft changes their license policy.

We will include every open source server we can find.

[StevenDoesStuffs]

https://github.com/Microsoft/TypeScript/blob/master/src/server/server.ts
https://github.com/Microsoft/TypeScript/tree/master/src/server

Is that of any significance?

And also, xtext will still be a thing right?

[vparfonov]

Move:

• [ ] Language Server Protocol: Code Lens #1804
• [ ] Language Server Protocol: Code Lens Resolve #1805

to the Support for Language Server Protocol - part 2

[TylerJewell]

@vazexqi - you asked about delivery timeframes - this looks like it will be completed for M2 or M3 of the 5.0.0 release train, which is very near term, likely this month.

[felixfbecker]

Could you add https://github.com/felixfbecker/php-language-server to the list?

[StevenDoesStuffs]

@TylerJewell Any updates on the typescript lang server? Or are we out of luck?

https://github.com/Microsoft/TypeScript/blob/master/src/server/server.ts
https://github.com/Microsoft/TypeScript/tree/master/src/server
Is that of any significance?

[TylerJewell]

@StevenTheEVILZ - we will have to wait and see what Microsoft chooses to do with it. We'll package language servers as they become available.

[StevenDoesStuffs]

@TylerJewell I;m probably missing something, but isn't that licensed under Apache 2.0 like the rest of the typescript package? And also, I'm not even sure if that is a language server; I don't see it using anything from languageserver-node.

[TylerJewell]

Our understanding is that Microsoft has a language server for typescript, but has not chosen to release it yet. It's only for vscode right now. This is the list of language servers that have been published. Not all are open source. https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations

[StevenDoesStuffs]

I see. TY

[StevenDoesStuffs]

@TylerJewell Sorry to bother you again, but then what is the thing I sent you?

[TylerJewell]

I don't know - we are not following what is happening in the typescript libraries. Microsoft will let us know if they add more language servers in the open source realm. We'll also publish the ones that we include soon, too. Unless, of course, you wanted to adapt Typescript into a language server for others to use?

[StevenDoesStuffs]

@TylerJewell Yeah, I was thinking about writing a typescript lang server. I just wanted to make sure that there isn't already one ready to be implemented.

[TylerJewell]

I'd get in touch with Microsoft on their forums and ask about their progress. If they are planning something, you would want to know about it before doing the work.

[StevenDoesStuffs]

K. TY

[tolusha]

@felixfbecker
When we have API and UI to inject ls agents into a machine, the php ls will be among them.
Or are you talking about a bit different list?

[felixfbecker]

Just referring to the list in the issue

[StevenDoesStuffs]

@felixfbecker @TylerJewell I think che is already using crane as its php ls:
https://github.com/eclipse/che/pull/2282

[kaloyan-raev]

@StevenTheEVILZ This is true. I submitted #2282 with the Crane server. That day it was more or less the only available PHP language server.

Since then we have also https://github.com/felixfbecker/php-language-server on the scene. It shows faster progress than Crane so far. Also it is written in pure PHP, which has a greater potential to attract the PHP community for contributions and to reuse existing PHP tools.

I believe very soon we should replace Crane with https://github.com/felixfbecker/php-language-server.

[felixfbecker]

I also use the existing https://github.com/nikic/PHP-Parser for parsing instead of rolling my own, imo this is a much more solid foundation. I also care about as-high-as-possible test coverage.

[TylerJewell]

This is a first cut of the documentation that we need to write around language servers. These are my notes from a 90 minute demo / review with the language server team this morning.

A language server has two phases of management within Che: installation and initialization. The installation is about how the language server gets installed into the stack, so that it is present for Che to turn on or off. The initialization process is the launching of the language server so that it is a running process within the workspace, and that the editor can connect to it.

The installation of a language server must happen within the definition of a workspace, or as part of a stack. Language servers are packaged within "agents" which are scripts that install the langauge server and its dependencies into a workspace. The agent installation can happen either as part of a stack (when the workspace is first started), or dynamically when a file of an associated type is first opened in the editor. It's recommended, for performance, to always include the associated agent within a workspace / stack definition.

Shortly, we will be providing a GUI within the dashboard that lets users add / remove agents to a workspace or a stack. This will also be part of the stack editing process. Stacks can choose from various agents that they want included.

There can be different types of agents added into the system other than language servers. For example, we have an SSH and terminal agent, which provide the services you expect the workspace to have.

Currently, all agents must be pre-defined within Che and saved in this location as part of the Che repository. We are thinking about a public registry for agents where they can be added and removed, but this is a future activity.

Each agent is saved as a JSON file. The name of the file without the extension is the ID for the agent within the agents list used by the dashboard or other products. So the CSharp agent is identified as "org.eclipse.che.csharp". Inside of workspaces and stacks, there is a new data object called agents : [] which is an array of agents that are to be included. The array would list the identifiers of each agent to include.

Agents have a name, a set of other agents that they depend upon, properties and a "script", which defines how the agent's package is to be installed into the workspace. This is not the startup of the agent, but just the installation. The script section of the agent is difficult to read, but we also include the same scripts in this scripts directory and our CI systems convert those scripts to embed them inline within your agent during the packaging phase.

The scripts that you must provide with an agent have a large if block where you provide installation logic for each Linux distribution that we support. You can follow our templates for how to build agents of your own.

Once a language server is included in the workspace as an agent, then there needs to be a launcher that controls when the language server is started and stopped. Technically, agents can be start / stopped for any number of reasons. For language servers, those agents will be typically start / stopped when the editor opens a file of a certain extension or when a plugin of a certain type is created. Right now, launchers need to be authored as Che extensions that are packaged with Che itself. We will also look to make these dynamic in the future.

Write a launcher extension
Add the launcher as a module

Once you have added this extension, you can test your Che by just rebuilding the plugin-language server module and also rebuilding the core assembly.

[kaloyan-raev]

@TylerJewell This was very helpful. Thanks for taking the time to write it down!

[kaloyan-raev]

@TylerJewell It would be helpful to have a wiki page that lists all supported Linux distros and versions, and a minimal Che stack for each of them. This will help testing the LS agent installation and initialization.

[mickaelistria]

I reported https://github.com/Microsoft/language-server-protocol/issues/76 about setting up a public registry/marketplace of language servers that all clients could use.

@vazexqi about Che and eclipse-language-server, there isn't going to be much room for collaboration on the UI parts as the UI stacks are quite different. However, there is already collaboration on the common parts such as the protocol itself and the LSP4J api. The good way to move forward if you're interested in LSP is to contribute to both ;)