Neovim: Language Server Implementation (LSP)

[tjdevries]

@justinmk, I have been looking into trying to write some sort of shim/plugin/client (still not sure what the name should be) to provide (at least some amount of) integration between the language server protocol and Neovim.

I have read your discussion in Gitter a bit ago and it got me interested in trying to implement some part of it. However, before I got working on implementing it, I wanted to try and figure out if I was understanding the idea of the integration correctly.

I made this diagram to explain some of what I was thinking. Obviously there are a lot more scenarios to cover, but this is what I was thinking.

Also, do you know what language this client should be written in? I was thinking one of the remote plugins. It appears that Microsoft is working with this (https://github.com/Microsoft/vscode-languageserver-node) for a lot of their stuff, and we might be able to use it as well. Do you know which of the neovim node libraries would work well with this?

Thanks.

[KillTheMule]

Awesome, this things seems to be getting traction: https://internals.rust-lang.org/t/introducing-rust-language-server-source-release/4209.

[tjdevries]

Yeah, it looks like there are several implementations coming along quite nicely (https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations). I think this would be a pretty killer feature for neovim.

[justinmk]

The node implementation would be a good start. A lua implementation could be shipped with Neovim later.

[tjdevries]

Here's where I'll be working on this: https://github.com/tjdevries/nvim-langserver-shim

I will try and make it so we can swap out the remote plugin later (so get a lua implementation when a SDK for the language protocol is available).

[euclio]

cc #4982

[tbelaire]

Just to note, I had started on a python client with the intent of then writing vim and neovim, and making sure I can support the Rust LSP.

[purpleP]

Not sure that this is the right piece for it, but I have this idea for a while.

If there would be a thing that would take a language specification (BNF etc) and make a parser for that language (should be totally possible and there is YACC).

The parser can generate AST and it should be possible to create a DSL for linting rules, so that not only linter designers could write them but users too. Which would solve a problem where pylint (python linter program) would complain in wrong places and won't where it should have been.

This tool should have integration with editors similar to what Microsoft proposes.

This would give smarter completions that take types (possibly inferred for dynamically typed languages) into account.

Next thing is refactorings. Because refactoring is just a function from one AST to another it should be possible to create fairly simple DSL or service for that also. So language specific refactorings could be added easily.

This thing could also remove the need of ctags as it could store all information about various program components (like classes, variable names etc) in some database like sqlite, mongo etc) and where ctags gives you ability only to jump to tag that is for the word under cursor this thing could integrate with Unite plugin and show things like all classes or all variables or all strings etc (more complex queries).

[tjdevries]

If I'm understanding correctly, that's pretty much a language server as defined by microsoft's protocol.

[justinmk]

If there would be a thing that would take a language specification (BNF etc) and make a parser for that language (should be totally possible and there is YACC).

Check out the scintillua project for a library of PEG grammars for various languages. I'd really like to see this used for syntax highlighting in Nvim.

Because refactoring is just a function from one AST to another it should be possible to create fairly simple DSL or service for that also.

Refactoring is not AST => AST. Useful refactoring depends heavily on semantic analysis (type system) or more sophisticated inspection.

This thing could also remove the need of ctags as it could store all information about various program components

Microsoft Language Server Protocol abstracts those things into a common interface (this could be the DSL you mentioned) that can be used by any front end. AFAIK it is not a one-size-fits-all solution for storing, indexing, querying.

[purpleP]

@justinmk As I've said type checker will be also implemented in my proposed solution. And ANY code change is AST -> AST and so is refactoring. As to scintilla it only have lexer, right? Whereas I'm talking about parser.

[purpleP]

@tjdevries I've read briefly about Language Server. It differs from my idea in two ways. First I'm talking about implementation and protocol, whereas MLS is only a protocol. Second, in my solution protocol differs a bit. For example MLS doesn't define any means of querying the code base. Only jump to definitions. But I could be wrong about it.

One of the reasons I'm talking about this is that my proposed solution could be compatible at least partially with MLS, so if neovim would support that it would be good.

[prabirshrestha]

I would also be very interested in how this goes.

Given that my work machine is windows and neovim hasn't been working well there 😞 I managed to write LSP client in pure vimscript that works asynchronously in both vim8 and neovim.

Currently it just the client part.

https://github.com/prabirshrestha/vim-lsp

[tjdevries]

Oh this is nice. I will take a look at this later today. Maybe we can
combine our work somewhere!

On Sun, Nov 6, 2016, 4:22 AM Prabir Shrestha [email protected]
wrote:

I would also be very interested in how this goes.

Given that my work machine is windows and neovim hasn't been working well
there 😞 I managed to write LSP client in pure vimscript that works
asynchronously in both vim8 and neovim.

Currently it just the client part.

https://github.com/prabirshrestha/vim-lsp

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/neovim/neovim/issues/5522#issuecomment-258669135, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEQo0-Zwd_lQ8-KnCa6hZrCVfQZpSxWbks5q7Zw-gaJpZM4Kd6cE
.

[prabirshrestha]

I'm all for working together.

[tjdevries]

I'll try and post a screen cast of some features that I've gotten working, but I think I've got:

• [x] Use sourcegraph/langserver-go to test
• [x] Functions to make messages / message dictionaries quickly in Neovim.
• [ ] Implement various actions in (similar to) this order:
  
  
  
  • [x] Initialize
  
  
  • [x] Shutdown
  
  
  • [x] textDocument/didOpen
  
  
  • [x] textDocument/references
  
  
  • [x] Hover
  
  
  • [x] Go to definition: https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#goto-definition-request
  
  
  • [x] Find references: https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#find-references-request
  
  

all set to go (at least in a very basic form). So it's working quite well :D

I'm excited about the results.

[tjdevries]

Big shoutout to @prabirshrestha for his work on vim-lsp which I modified a bit and included in the shim.

[mhartington]

Very excited to see your progress @tjdevries. I've recreated something similar for typescript but using the python api

https://github.com/mhartington/deoplete-typescript/blob/master/rplugin/python3/nvim-typescript/__init__.py

[prabirshrestha]

Glad to see this coming out very well. @tjdevries awesome job.

@mhartington unfortunately typescript (tsserver) does not implement the language server protocol :( https://github.com/Microsoft/TypeScript/issues/11274 There does seem to be an effort by sourcegraph team https://github.com/sourcegraph/javascript-typescript-langserver

It will be interesting to see these LSP implementation working inside vim. https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations

[tjdevries]

I have confirmed it working on at least the sourcegraph go langserver and their python langserver. So that's pretty exciting.

Documentation will come out soon with some more ease of use type things :) I'll keep you all posted when that's ready.

[dumblob]

How did you solve the issue with concurrent edition of the same buffer (e.g. split buffer - possibly among different clients)? The Microsoft Language Server protocol still does not support it (https://github.com/Microsoft/language-server-protocol/issues/23#issuecomment-234992961 ).

[hauleth]

@dumblob but where is the issue? Vim has things enabled mostly per buffer, not per window, so there should be no problem with anything when using LS.

[dumblob]

@hauleth I'm sorry, but I don't understand what you mean. What are "things enabled per buffer"? How does it relate to e.g. split buffer edition among different clients?

[hauleth]

If you meant different instances of nvim then it would be the same problem among all editors, so nothing nvim specific and IMHO not an issue as when you edit files in different places simultaneously, then there always be problems. On the other hand, if you were talking about different vim windows then this is not an issue as LS would be operating on buffers not windows and windows shares the same buffer.

Am I missing something?

[prabirshrestha]

Something like file change notification would be useful to have in vim and neovim to create better integration for language server. Started a discussion in vim_dev at https://groups.google.com/forum/#!topic/vim_dev/oubHkRbdQW0. Some more discussion about this at https://github.com/neovim/neovim/issues/1380 including a WIP PR at https://github.com/neovim/neovim/pull/1791

[prabirshrestha]

In case anyone is curious to see demos :)

Async goto defintion in typescript using vim-lsp. More info at https://github.com/Quramy/tsuquyomi/issues/57#issuecomment-269136151 This works in both vim and neovim.

async completion in vim (Fork of completor.vim with python removed implemented in pure vim script. Currently doesn't work yet in neovim as I'm waiting for neovim to support lambdas. More info at https://github.com/tjdevries/nvim-langserver-shim/issues/11) Wished async completion was just part of core like completfunc and omnifunc

Currently simulating async completion with timers.

function! s:emoji_completor(args, done)
  call timer_start(1000, {timer->a:done([{'word': ':smile:'}, { 'word': ':+1:'}])})
  " for sync directly call a:done()
  " call a:done([{'word': 'smile:'}, { 'word': '+1:'}])
endfunction

call completor#providers#register({
      \ 'name': 'emoji',
      \ 'filetypes': ['javascript'],
      \ 'completor': function('s:emoji_completor'),
      \ 'trigger': completor#providers#trigger_from_chars([':'])
      \ })

2017 might finally be the year of async vim and vim as an IDE :)

[Shougo]

@prabirshrestha Why don't you use deoplete?

[prabirshrestha]

My work machine is windows and language is typescript and I can't get neovim with python working. Details at https://github.com/neovim/neovim/issues/5360 This means I'm back to using vim until then and deoplete doesn't work in vim 8 yet 😢

Also I don't think the current deoplete interface would satisfy my needs. I will file an issue in deoplete to discuss about the requirements. I need something like denite's __async_gather_candidates

Since I wanted to support vim too I posted my thoughts about async completion here https://github.com/tjdevries/nvim-langserver-shim/issues/11 which describes the interface in bit more details.

[Shougo]

My work machine is windows and language is typescript and I can't get neovim with python working. Details at #5360 This means I'm back to using vim until then and deoplete doesn't work in vim 8 yet 😢

Hm. deoplete should support Vim8, but it is not easy.
I must implement neovim-python compatibility layer.

Also I don't think the current deoplete interface would satisfy my needs. I will file an issue in deoplete to discuss about the requirements. I need something like denite's __async_gather_candidates

Hm. But why __async_gather_candidates is needed?
deoplete is already asynchronous.

[prabirshrestha]

I might be wrong on this but isn't gather_candidates supposed to return a list which is synchronous? Even though the gather_candidates is called asynchronously due to remote plugins in neovim the actual implementation should be synchronous on the python side.

Since gather_candidates is synchronous:

• I cannot buffer the completion items. Added more details at with example https://github.com/Shougo/deoplete.nvim/issues/407
• If I need to call asynchronously even without buffering required I would need to have some mechanism to wait for the async call to complete. Somewhat like c# task.Result which is like calling Wait(). https://msdn.microsoft.com/en-us/library/dd321468(v=vs.110).aspx

[prabirshrestha]

Got a fork of go-vimlparser working as a an LSP server while I wait for the reply at https://github.com/haya14busa/go-vimlparser/issues/26

Here is a demo of :FindDocumentSymbols using vim-lsp repo. Someone else should definitely pick up on vim language server.

[Shougo]

@prabirshrestha OK. deoplete should support the asynchronous feature.
It is not easy though...

[timeyyy]

The implementation of the protocol in lua could be simplified by using the haxe sd and generating lua. I've been playing with haxe and can reccommend.

[garyburd]

@timeyyy See langserver.lua for a partial implementation of the protocol in Lua.

[prabirshrestha]

Landed up writing my own async completion in pure vim script that works both on vim8 and neovim which depends only on timers. It is a fork of https://github.com/roxma/nvim-complete-manager. Huge props goes to roxma for his work.

Here is the fork: https://github.com/prabirshrestha/asyncomplete.vim. Beside removing python dependencies and supporting vim 8 I also added some features in this fork. Here is the demo in both vim8 and neovim on windows. timers is the only requirement.
Buffer completion and Ultisnips completion works synchronously.
Go completion works asynchronously via async.vim which is used normalize the job apis in both vim 8 and neovim so there is no if has('nvim') checks at all in the completion source plugin.

Pretty happy with the progress as you can see that there is no lag in the UI. Since it is pure async the initial gocode completion is a bit slow but as the server warms up the completions starts to be very fast.

Currently this is an experimental plugin as I have not locked down the apis yet. In long term I would like to also add the python features I removed from the original. The why section is a good read. https://github.com/roxma/nvim-complete-manager#why. The plan would be to allows the users to register custom core providers and provide vim and python implementation by default. (Core provider is the one that handles fuzzy matches. I want this to also be extensible so rather the writing all the algorithms in core, I would expect external people to write better ones and let the community decide which they favor). Vim users will be limited to vimscript for fuzzy matches including blocking UI while matching and sorting is taking place during autocompletion, while neovim users can enjoy the benefits of remote plugins and custom fuzzy matchers in python and more responsive UI. Ideally one could also write the code provider as an executable instead so even vim users can enjoy the speed. Once everyone moves to lsp protocol having multiple core providers might not be a problem since most of them would had been handled by the language server instead.
I plan to do something very similar in vim-lsp so that even the users have option to use the core parser in vim script or take advantage of the remote plugin that neovim provides.

The goal is to have consistency between vim and neovim and then optimize for each separately without breaking.

[roxma]

Hi,

I'm planning on adding built-in language server support into nvim-completion-manager, at least for code completion, with python implementation.

But I have not reached a decent design. so I've just made a working one from scratch, nvim-cm-php-language-server, just for demonstration.

[prabirshrestha]

@roxma That is awesome. Glad to see more people on boarding to the language server protocol.
Looking at the source it doesn't seem like it is following strict protocol. I was actually very curious how writing to stdin for didOpen and didChange would affect the perf. Seems like most of node client-server uses node IPC to communicate.

[autozimu]

I made a neovim plugin to support Language Server Protocol at

https://github.com/autozimu/LanguageClient-neovim

It still have rough edges, but right now is usable. (Well, at at least for Rust Language Server, I haven't got time to cover other server implementations yet.)

Thanks and happy hacking!

[roxma]

Since LanguageClient-neovim already has nice interface, I'm deprecating my previous nvim-cm-php-language-server

Here's a simple php language server package that could be easily setup to work with LanguageClient-neovim, and it is well integrated with nvim-completion-manager

[Shougo]

The pull request is here.

https://github.com/neovim/neovim/pull/6856

[prabirshrestha]

Started reworking on my vim-lsp plugin again in dev branch. (still lot of refactoring to do and finalizing the public apis). https://github.com/prabirshrestha/vim-lsp/pull/11

Currently these are supported.

• :LspGetWorkspaceSymbols
• :LspGetDocumentSymbols
• You can send custom request and notifications to the server. (Registering server notifications not supported yet)

call lsp#send_request('server_name', {
    \ 'method': 'initialize',
    \ 'params:' {....},
    \ 'on_notification': {server_name, response->lsp#log(server_name, response)},
    \ })

call lsp#send_notification('server_name', {
    \ 'method': 'initialize',
    \ 'params:' {....},
    \ })

Calling these methods will automatically flush the buffers (i.e. wait for server start, initialize, didOpen and didChange. I had to actually go far as writing an async helper method in vim script to coordinate this https://github.com/prabirshrestha/vim-lsp/blob/31743d44463cd62358503f83d4a87fd54b818cdd/autoload/lsp/utils/step.vim)

function! s:ensure_flush(buf, server_name, cb) abort
    call lsp#utils#step#start([
        \ {s->s:ensure_start(a:buf, a:server_name, s.callback)},
        \ {s->s:is_step_error(s) ? s:throw_step_error(s) : s:ensure_init(a:buf, a:server_name, s.callback)},
        \ {s->s:is_step_error(s) ? s:throw_step_error(s) : s:ensure_open(a:buf, a:server_name, s.callback)},
        \ {s->s:is_step_error(s) ? s:throw_step_error(s) : s:ensure_changed(a:buf, a:server_name, s.callback)},
        \ {s->a:cb(s.result[0])}
        \ ])
endfunction

• Also managed to get asyncomplete source for lsp. https://github.com/prabirshrestha/asyncomplete-lsp.vim

Here is a sample .vimrc on how to use it.

Plug 'prabirshrestha/vim-lsp', { 'branch': 'dev' }
Plug 'prabirshrestha/asyncomplete.vim'    " optional only required for autocompletion
Plug 'prabirshrestha/asyncomplete-lsp.vim' " when server is initialized and it supports completion, asyncomplete source is automatically registered. no need to call asyncomplete#register_source in .vimrc

au User lsp_setup call lsp#register_server({
     \ 'name': 'rust',
     \ 'cmd': {server_info->['rustup', 'run', 'nightly', 'rls']},
     \ 'whitelist': ['rust'],
     \ })

au User lsp_setup call lsp#register_server({
    \ 'name': 'ts',
    \ 'cmd': {server_info->has('win64') ? ['cmd', '/c', 'javascript-typescript-stdio']  : ['sh', '-c', 'javascript-typescript-stdio']},
    \ 'whitelist': ['typescript'],
    \ })

Are there any plans for supporting remote lua plugins in neovim? I would like to support both vimscript and remote lua plugin specially since some of these parsing are very expensive. Another option I have been thinking is to have rpc support in async.vim so that it works in both vim8 and neovim.

[justinmk]

are there any plans for supporting remote lua plugins in neovim?

Yes.

[HerringtonDarkholme]

The most difficult problem I encounter in vim Langauge client is completion.

LSP's completion item is very rich and expressive, way beyond Vim's builtin completion mechanism.

Detail: https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#textDocument_completion

For example, a server can return a completion item as snippet. In VSCode selecting this item wil trigger snippet expansion.

Item can also have TextEdit field, selecting such item will apply text edition. For example, changing characters before completion position.

Some servers, like Java LS, will even uses AdditionalTextEdit for things like import library.

I wonder how far we can support LSP using current completion in Vim.

[roxma]

For example, a server can return a completion item as snippet. In VSCode selecting this item wil trigger snippet expansion.

I believe the snippet support is possible with existing plugins: deoplete+neosnippet+LanguageClient-neovim
or nvim-completion-manager+neosnippet/ultisnips/snipmate+LanguageClient-neovim

But I haven't seen a language server with this capabilities yet (I've tested php-language-server, go language server, rls, python). Anyone point me a direction so that I can test it on my laptop?

[HerringtonDarkholme]

Hi @roxma, thanks for reply! You can try https://github.com/vuejs/vetur/tree/master/server.

Example test code

<template>
  <div cl|></div>
</template>

The | indicates cursor position. VLS will return a snippet which expands to

<template>
  <div class="|"></div>
</template>

Note the position of cursor. It's inside of quotation.

Edit: update code demo

[BonaBeavis]

Hi, I am experimenting with LanguageClient-neovim and java and I stumbled on an issue. The java lsp responses with its own schema jdt://... for following a dependency definitions: autozimu/LanguageClient-neovim#392.
Is there a handler interface in neovim which could be extended to handle this issue?

[matheuslessarodrigues]

do any of these LSP plugins work with csharp? couldn't figure it out on how to make it work

[norcalli]

Guess I can close this now :P