Deoplete.nvim: async gather candidates

[prabirshrestha]

I'm looking into providing deoplete source for language server protocol clients. Let me know how to implement the source.

Note: The client it self is started using vim script using jobs. This means all communications needs to be async.

Here is how it is going to work.

1. start the language server based on file type. (This is async via jobs.)
2. send initialize request to the language server
3. get the initialize response. this is async too.
   3.a. if the language server support completion, set the deoplete' files type keyword completion pattern from the initialization response. export interface CompletionOptions { triggerCharacters: string[] }. For typescript triggerCharacters would be ['.'] for cpp it would be something like ['.', ':', '>'].
4. deoplete should now call gather candidates based on 3.a
5. gather_candidates should call the job started in 1 and ask for textDocument/completion. https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#completion-request
   Since it is async I can't return empty list. I need to wait for the completion items to arrive. This completion items are buffered which means I may get the first five completion items in 1 sec and then in another 2 sec I might get more completion items based on the response. interface CompletionList { isIncomplete: boolean; items: CompletionItems []}

Ideally the source implementation would need to look something like this pseudo code. Note a async/await this means it is not blocking. We are waiting for partial results to come and as it comes we refresh the completion menu. (Ignore my python skills here :))

class Source(Base):
    def get_complete_position(self, context):
        # get from triggerCharacters
        # if server not initialized might be we can just return -1

    async def gather_candidates_async(self, context):
        # call the language server job created in vim script

        while True
            let result = await self.vim.call('getCompeltions')
            self.append_candidates(result)
            if result.isIncomplete == False {
                break
            }

Similar interface would also be needed for Denite. For example if I want to find all the classes used in the project (workspace). The server could first asynchronously return only items from the file and then as it computes more files it could send more classes.

Some more details at https://github.com/tjdevries/nvim-langserver-shim/issues/11

[Shougo]

Similar interface would also be needed for Denite.

This feature is already implemented in denite.

[tweekmonster]

I think support for a language server should be a separate source. And async def would add a hard requirement of Python 3.5 for deoplete, which isn't ideal.

I think a better option for maximum backward compatibility is to introduce a new class:

from . import Base

class AsyncBase(Base):
    completions = []

    def gather_candidates(self, context):
        return self.completions

    def on_event(self, context):
        if context['event'] in ('TextChangedI', 'TextChanged'):
            self.completions = ['whatever']

(or add the attribute is_async to the existing class)

This is off the top of my head, but it could happen in one of the following ways:

• deoplete could iterate over AsyncBase sources in a coroutine/thread to trigger certain events
• deoplete could iterate over AsyncBase sources in the main thread and run on_event() in coroutines/threads

gather_candidates() would return cached results when deoplete collects the completions. With this being a subclass of Base, deoplete can skip these sources at its discretion.

Again, this is off the top of my head, but I think using a coroutine for this kind of task is not ideal and threading might be a better option. The reason is that deoplete will need a way to cancel the background sources if they're still running by the time a new event comes in. Otherwise, we'll be back in the same situation of sources blocking, but in queued background tasks instead.

[prabirshrestha]

I'm not familiar with python so definitively shouldn't be taking my suggestions on the best practices on how to implement it. @tweekmonster your suggestion seems good. Cancellation is definitely important for intellisense like making http requests to get the packages name and version in package.json.

After re-reading the spec carefully again seems like it actually doesn't need to stream the completions. rather it says. This list it not complete. Further typing should result in recomputing this list.. @Shougo would this one be easier to implement. Although I do see cases where streaming could be useful. Might be having a separate base class would make it easy so in future if someone needs to support stream it could be achieved.

[Shougo]

It will be high priority for me.
But I must refactor the deoplete implementation before that.

[prabirshrestha]

@Shougo I explained the algorithm of how VsCode provides intellisese at https://github.com/roxma/nvim-completion-manager/issues/30#issuecomment-283281158. It also explains the isIncomplete flag. This might be helpful for deoplete too (Currently I'm not familiar with internals of how deoplete works. Would be good to have a doc like nvim-completion-manager on why it is fast)

[Shougo]

Thank you for the infomation.
I will look into it.

[Shougo]

My next task is to implement the feature.
Please wait.

[Shougo]

I have implemented the feature!