Neovim: Autocomplete Improvements

[jfelchner]

The autocomplete in Vim is abysmal when compared with IDEs. Even the plugins which try to improve it such as NeoComplCache and SuperTab can't do what they need to do in order to be on par with Emacs or even SublimeText.

So my question, will neovim's refactorings allow either vim itself or plugin makers to be able to create a reasonable autocomplete system?

[thiderman]

What's abysmal about it and in what way are the other editors being better at it? Since you don't really state any problems it's hard to know if it would be possible to improve anything.

My main gripes with the completion system are:

1. Speed, because of vimscript.
2. The API is clunky at best. You have to write a function that does two separate things (determine string to base completion on, and generating suggestions based on that) and you have to write it in a way that means it will be called twice with different parameters. At least for me, this increased friction and made it take longer until I could be productive.
3. The popup that shows generated suggestions is tricky to add formatting to.

As to how these can be fixed, it's probably difficult to know right now. A discussion would still be useful though!

[jfelchner]

@thiderman my main gripe, although there are many is that you cannot source multiple omnicomplete sources in one menu.

So for example if I'm in a CSS file, I want autocomplete to show me CSS tags/values but I also want it to autocomplete other values which may not be "typical" CSS values but that I may have used elsewhere in my CSS files.

Or I'm in a programming file and I want it to complete based on keywords in the files in my buffers, but I _also_ want it to look in my tags file and add those to the suggestions.

Finally I want it to be able to intelligently weight each of these such that it gives me the "correct" answer the majority of the time.

[jfelchner]

For the record, sourcing multiple omnicomplete sources in one menu is _technically_ possible but it is a dirty, dirty hack which is very prone to errors (if it even works at all) and therefore I do not consider it a solution.

[Soulou]

Have you played with https://github.com/Valloric/YouCompleteMe It is, according to me, the best completion available for vim.
Ok it's a bit more difficult to install than à simple vim plugin because as vim script is slow, the extension has been built with C++ and Python, so you need to compile it first.
The result is quite impressive.

[jfelchner]

@Soulou if by "best completion available in Vim" you mean "best completion available in Eclipse" :wink: You have to run an Eclipse server (at least for Ruby) to get anything approaching the level of autocomplete that I think that Vim should provide out of the box (or at least available via an easy to install plugin).

I agree that YouCompleteMe _is_ probably the best completion system out there for Vim and comes closest (assuming one goes through the work) to a modern autocomplete system, but it is still (IMO) a dirty hack.

Sorry, my standards are really high. This is the number 1 reason why I was thinking of switching from Vim to something else. I'm expecting an IntelliSense quality autocomplete. I'm no fan of Microsoft, but IntelliSense is the best autocomplete system I've ever used (but let's not digress into whether it's good or not I just care about having a badass autocomplete available in my favorite editor :+1:)

[Shougo]

Hi, I'm the author of neocomplete auto plugin system.

Yes, Vim has good completion system, but it is not suitable for auto completion.
Well, it is IDE like feature.
Vim is not aimed for auto completion.

1. Culture:
   Vim has not auto completion system. Current auto completion systems are dirty hack includes neocomplete and YouCompleteMe. I want to get auto completion official support for the editors.
2. Speed and asynchronous:
   VimL is slow, Yes(Python is 2x+ faster).
   And Vim's asynchronous completion support(input_check()) is very limited.
   I cannot use it for neocomplete/neocomplcache.
3. Popup API problem:
   Vim's completion(API) problem is very limited and incomplete for VimL.
   I cannot implement features like auto-complete.el or unite.vim.

[Soulou]

@jfelchner I agree with you, YouCompleteMe is close to a dirty hack, and it works more or less well according to the language you are using. For python, jedi is doing a great job, for go gocode is doing a great job, I haven't tried to install anything for ruby, but I see what you mean.

So the question becomes, is it a goal for neovim to include IDE like features or not, but obviously if the work can be easy for plugins, it would be great !

[torpak]

It would be really awesome if neovim could provide common frontends for IDE tasks like code completion, jump to declaration/definition, go to error, debugger integration, and so on. Plugins could then provide the backends and the user would have the same UI independent of the programming language they are using.
Neovim could already include backends for lua and vimscript which would make plugin development a breeeze.

[tarruda]

I agree that we need a better completion API, and it's something I intend to implement and expose with msgpack-rpc, but it won't be done at this moment so I'm closing the issue. We can reopen later.

[davidhalter]

@tarruda If you're ever reopening this (or discussing it somewhere else), please discuss this first with the current VIM autocompletion plugin designers. It would be too bad if we end up with the same mess again.

This would include the authors of YCM, NeoComplCache, jedi-vim and a few others.

[justinmk]

@davidhalter I recall that you and valloric had a good discussion about the role of YCM and jedi and whether both are needed or should they be merged, etc. And in that discussion @Valloric mentioned his ambition to make YCM a generalized completion middleware for any number of engines (Jedi, clang) for any number of frontends (Vim, etc). But I can't find that discussion. Can you link to it here?

I think that if YCM becomes a pseudo-standard target, or at least is thinking hard about how to achieve that, then Neovim should look carefully at the design decisions there.

If YCM can deliver a Windows build then my next question would be: why not ship it with Neovim?

[tssm]

@justinmk I think this is the discussion you are looking for.

[justinmk]

@taecilla That's it! Thanks :)

[jfelchner]

@tarruda I think this was closed prematurely. I think there's a huge amount of good discussion that could be had here by all of the current experts in this area. I understand the need to keep the issues "clean" but it also makes it more difficult for those interested to find it.

[tssm]

I was reading the discussion mentioned by @justinmk a couple of days ago, thinking on Neovim, and it seems to me that YCM is more than an improved "omni completion". It talks to clang which does static analysis of the code, like jedi-vim does for Python, and Tern for Vim for JavaScript, so you also have syntax checking, refactoring and jump to definition. Loved it! But I don't like of a monolithic tool like YCM. I think it would be nice if Neovim replaces the omni completion and tags functionality with an API that would get input from glue code (Vim scripts that I explicitly install, like jedi-vim or Tern for Vim) between it and the static analyzer tool (Jedi and Tern respectively). After all, there are lots of programming languages, and I just want integration for those that I use, which are less than ten.

[justinmk]

@jfelchner This discussion is marked as reopen-later, so it's not lost. After Neovim has reached its milestones we'll reopen all issues with that tag. Are you saying you think this should be kept open so that others can find the issue by searching the "Open" issues?

[tarruda]

@jfelchner just wait a little bit more, when I get the rpc API working(in a way that is can be easily extended) this issue can be reopened since there will probably be bounties for better autocompletion: https://docs.google.com/forms/d/1TjpCEJbQ6PehftrRsuz54X3HL6abjCUuuiipFCmmHV4/viewanalytics?usp=form_confirm

[justinmk]

it would be nice if Neovim replaces the omni completion and tags functionality with an API that would get input from glue code [...] between it and the static analyzer tool (Jedi and Tern respectively)

@taecilla My understanding from https://github.com/davidhalter/jedi-vim/issues/119 was that YCM is trying to fill that role. It has two components: a common interface, and a back-end. YCM uses jedi as the backend for python, but the completer interface is still the same.

The common interface is what would be interesting for Neovim to look at and learn from.

Quoting @Valloric:

I see ycmd as the one completion server that editors talk to; ycmd itself ships with and connects to various semantic engines (Jedi, libclang, omnisharp etc.) but provides one common interface to talk to all of them.

[davidhalter]

@tarruda My main concern is actually not the form of communication, but the definition of the API. The current way of doing autocompletion in VIM is just wrong on some levels. It supports async behavior to a certain degree, but it's not good enough.

Also: If you're ever discussing the way how any GUI can be written (or drawn), please mention me, so I can add to the discussion, I've had several issues with that.

[Valloric]

I'll refer people to my comments on the issue @taecilla linked; YCM will become an editor-agnostic code-completion (and more than that) middleware, exposed as a server. It actually already works this way, the client-server split was done months ago in YCM internally, I just need to pull out the server into a separate GitHub repo, write docs and clean up a few things (but finding the time to do this... _sigh_).

The more time I spend improving YCM and analyzing how other editors/IDE's do code-completion/semantic operations like GoTo etc (and trust me I've spent quite some time examining everything from Eclipse to NetBeans to VisualStudio to IntelliJ to Komodo to... everything, pretty much) the more I realize editors should not be doing this themselves. Editors should just be thin clients around a generic component that does this.

Otherwise everyone just keeps building the same thing over and over again, and each new system fails to match the state-of-the-art (which IMO is probably IntelliJ).

Editors _should_ provide a nice client-side UI for this though; the completion server can provide the editor with completion data given the buffer contents, filepath, line, column and other relevant stuff, but the editor needs to provide the popup window, the as-you-type interaction and the extra windows for displaying function documentation etc. There's plenty of client-side work necessary for a good code-completion experience that doesn't involve building the semantic engines. _That's_ what editors should focus on.

But editors shouldn't be integrating the logic for how to produce the completion data; it's both easier and more sensible to leave that to third-party middleware that the editor can just ship with and use under the covers as a data source.

[philix]

Otherwise everyone just keeps building the same thing over and over again, and each new system fails to match the state-of-the-art (which IMO is probably IntelliJ).

I strongly agree here. :+1: Let's not make with code completion what was done with spell checking in Vim: big spell.c that reinvents the spell checking wheel ignoring good existing solutions.

[Valloric]

I strongly agree here. :+1: Let's not make with code completion what was done with spell checking in Vim: big spell.c that reinvents the spell checking wheel ignoring good existing solutions.

Spell-checking is a great example of the model I'm describing: hunspell is the common backend and the clients (Firefox, Chrome, LibreOffice etc) implement a UI on top of it. That's how it should work instead of everyone re-inventing a spelling engine.

[jfelchner]

While my first reaction was to disagree, after some thought, I've decided I agree with @Valloric. I would be interested in two caveats (either of these would work for me):

• Lightweight enough that it could run constantly without taking up a ton of resources when it's not used
• Something that could be quickly started and stopped when opening and closing the editor

[jfelchner]

I also feel like a system like this would be able to lexically scan a set of files in the project and be smart enough to do things like "jump to definition", etc. It seems to me that those two things have a similar root of responsibility.

[davidhalter]

@Valloric What do you think about adding YCM fully to NeoVIM? At least the client? (YCM would still be a separate project, but adding it to NeoVIM would mean builtin support).

[Valloric]

@davidhalter Sounds like a solid idea; the client would still have to continue working in vanilla Vim though, but I'm not at all against adding extra support for Neovim, as long as it doesn't complicate the client's ability to work well in vanilla Vim. I'm also not against Neovim building it's own client for the YCM server (ycmd).

@jfelchner The YCM server can be easily started/stopped on demand, so if the user is not using it in Neovim you could easily have it shut down. No resource consumption then. It starts up in a second or so. BTW in Vim it's already started and stopped as Vim is started and stopped, but can easily be restarted by the user.

[jfelchner]

@Valloric :thumbsup:

[justinmk]

Reopening under milestone "autocomplete" since this is now a bounty item.

[tssm]

But editors shouldn't be integrating the logic for how to produce the completion data; it's both easier and more sensible to leave that to third-party middleware that the editor can just ship with and use under the covers as a data source.

If this approach is chosen (and just like @jfelchner, I have been convinced by @Valloric), then the omni/user completion and the ctags/cscope interfaces could be deprecated, right? I use Exuberant ctags for PHP completion, but it feels like an old and dirty patch because there is nothing like Jedi/Tern/semantic-engine for PHP (neither for a lot of other languages, and that's why I wrote "deprecated").

I also feel like a system like this would be able to lexically scan a set of files in the project and be smart enough to do things like "jump to definition", etc. It seems to me that those two things have a similar root of responsibility.

Yeah, and YCM also performs syntax checks thanks to clang, and I guess it could provide syntax highlighting as justinmk mentioned in #719 and folding. It all depends on the semantic engine, so Neovim code could be even cleaner, right?

[dontdieych]

@Valloric Thanks for insightful comment. :+1:

[ChrisPenner]

I'm in favour of using the model suggested by @Valloric, one request I have is that Fuzzy completion be supported in this model from the beginning; including case-insensitive fuzzy-complete.
Fuzzy completion helps substantially when typing constants like WINDOW_SIZE_HORIZONTAL and WINDOW_SIZE_VERTICAL which are hard to type and only differ at the end, and could also be used when specifying file paths etc. It's gaining popularity pretty quickly lately.

[jfelchner]

@ChrisPenner I agree.

[jfelchner]

Say what you will about Microsoft, but IntelliSense is freaking amazing.

Check out the video on https://code.visualstudio.com for a look at how they use it with JavaScript. Very impressive.

This is the kind of thing that NeoVim should enable other developers to build plugins for.

[jfelchner]

The other best in breed in my opinion is IntelliJ:

https://www.jetbrains.com/idea/features/code_completion.html

[lithammer]

As far as other terminal applications go, I think bpython does this pretty nice:

[jfelchner]

@renstrom very nice!

[gaving]

Throwing http://blog.atom.io/2015/05/15/new-autocomplete.html into the mix, hands down the nicest autocompletion I've seen yet.

Of course, this can be used as-is with vim-mode for atom in the present moment!

[jfelchner]

@gaving excellent!

Keep the examples coming. I know that if we see what everyone else is doing we can take those ideas, improve on them, and make vim more amazing than ever. 😄

[a-b]

While vim amazingness coming is there any working autocompletion for neovim exists?

[gglanzani]

@a-b Yes, https://github.com/Shougo/deoplete.nvim

I'm using it on a daily basis and, although it's still rough around the edges, it's good.

[7belrhalmia]

@tarruda there is any improvement in neovim autocompletion

[jfelchner]

@7belrhalmia lots of behind the scenes upgrades which will eventually help but nothing will happen with this until probably version 1.0

[7belrhalmia]

@jfelchner but you think that autocomletion will be improved and will be like ide style like insert all function prototype or a popup to show function paramettre

[7belrhalmia]

@jfelchner and how long we have to wait until that will hapen

[jfelchner]

@7belrhalmia I wouldn't even begin to fathom a guess. It's gonna be a while. But they finally have "mini milestones" before 1.0 so we should see some visible forward progress soon.

[choco]

I love vim but I'm fed up with the way the complete popup + preview behaves, so I've been toying with a new design for it. Here's a couple of screenshots of how it would look.
Results list:

Selected element:

Instead of opening a preview window, additional infos would get displayed under the list of results. The current patch I'm using is a rough PoC and since implementing it right would require rewriting a good chunk of popupmenu.c I decided to post here and get some feedback.
The patch would:

• render the kind of the suggestion first, and styles it based on the kind value.
• render the rest of the popup menu as before
• render extra info under the results list with another style attribute

This change wouldn't modify the current API, just change the design of the popup, so if collaborators prefer I can open a new issue instead of cluttering this one that seems more focused on improving the API itself.

[scampi]

@cHoco looks nice!

[jfelchner]

@cHoco beautiful! Tell ya what, go ahead and open new issue and the mods can add your issue to the Autocomplete milestone.

[Shougo]

@cHoco It is really nice! Can you send the Pull Request?

[PixelPirate]

@cHoco This looks really nice! I would love to use this!

[gavsim]

@cHoco Looks great 👍Get that PR opened.

[Valloric]

@cHoco That looks amazing!

I've been saying for years that Vim needs an API that plugins can use to create sub-windows. This could then be used to build whatever the plugins want in terms of code-completion display.

For instance, the screenshot you're showing puts the detailed info window at the bottom of the completion list. Personally, I think that putting it to the right of the completion list is much better, for various reasons. But if the window is being drawn by Vim/Neovim, then I as a plugin author have no choice but to use what you've provided.

So the whole "specific API in Vim for showing code-completions" is IMO too limiting. Better to provide generic support for drawing & placing custom sub-windows (and their styling) in plugins. This supports _so many_ use-cases, of which better code-completion is but one.

[danihodovic]

@cHoco Nice!

I agree with @Valloric. This is a bigger issue than just the completion popup, being able to draw custom windows opens up a world of new use cases for plugin creators.

[justinmk]

I've mentioned in some places that allowing buffers to "float" at arbitrary coordinates (while improving the buffer API) may be the best way to allow plugins to create arbitrary popups/menus (even "widgets", if you will). Though I did not create an issue for this concept.

[Valloric]

@justinmk Making the windows real buffers creates problems though. Suddenly what should be a display-only window can be navigated to (and possibly changed) by the user or (accidentally) by a separate plugin.

As a plugin author, I want a window strictly under my control without worrying about a stray plugin or the user messing with it by accident. (The user should of course have the power to kill any such windows I create. The user is of course free to not use my plugin too.)

[wsdjeg]

@cHoco
great work ! can you open an PR
@artur-shaik
MAYBE YOU SHOILD KNOW THIS ,IF IT IS MERGED,IT WILL BE EASY FOR YOU TO CREATE JAVADOC FOR EVERY METHOD OR FIELDS

[justinmk]

As a plugin author, I want a window strictly under my control without worrying about a stray plugin or the user messing with it by accident

@Valloric Certainly the design is up for discussion. New facilities for such control may need to be invented, but I'll note that some clumsy ones exist already: :set eventignore=all and :noautocmd...

Main goal of re-using buffers for such purpose is (1) to inform new buffer-API design in service of such use-cases, and (2) to benefit users and nvim itself by re-using existing concepts.

[jfelchner]

@justinmk @Valloric either way, the fact that these discussions are being had, and that we all know improvements need to be made (whatever shape they take) is what makes Neovim amazing.

The great communication between plugin authors and Neovim core is outstanding. After all, I think the best philosophy for Neovim is to understand what is the _underlying_ problem that plugin authors need solved and build that. Because once those problems are solved, plugin authors can play around with different higher-level abstractions to find out what works best. And Neovim core remains svelte and compact (well, as much as possible while still maintaining 50 years of backward compatibility :grinning:)

[xcambar]

@cHoco Great work! Off-topic: You have an amazing font and theme. Care to share?

[carlitux]

@cHoco that looks nice!!!

[MichaelMackus]

Nice, @cHoco !! That looks really cool.

Beyond just the styling of the popup, I definitely agree that vim's autocompletion engine could use some more love. The API is terrible in my little experience hacking things in the attempt to "combine" auto completion engines. I'd love some sort of completion frontend integrated into neovim, where plugin authors could just hook in their backend code (independent of language).

[PixelPirate]

Is there actuallly any progress on the new UI or a branch to test it / help?

[mhartington]

@cHoco not to be a pest, but this is awesome. Would love to see this added! Please make a PR or post a gist of the changes

[justinmk]

@mhartington #4432 shows a working POC by @romgrk using the new externalized popupmenu API.

[choco]

I'm sorry for never posting an update, but that's because there was nothing concrete to update about :(
The patch I used in the screenshot was basically just a hack not really ready for use, and the extra info below was actually completely hardcoded (it wasn't generated dynamically for each element).
When I started thinking about how to implement this change the right way, more than a single problem became evident:

1. in the patch, I had hardcoded the check for either 'f' 'c' 'm' etc... but these letters aren't actually standard and different completion plugins/omnicompletetion functions implement different letters etc.. so a first step would be to define all the possible values there (the documentation talks about some of them, but to correctly implement this all should be defined). Then once that is done, since I think everyone would love to costumize them, we'd need a to define for each of these letter types an highlight (more options wohooooo). And don't forget the highlight styles for the other components of the popupmenu (even more options)
2. the design above is able to keep the current behaviour of vim to align the completed word with the suggestion only because I use line numbers, but if one doesn't then the completion text wouldn't be aligned. That's when I thought that maybe people at that point would prefer to write the formatting themselves to better suit their needs, maybe with the same syntax of the status line. My idea was then to reuse the parsing function for status line to add and option to decide how to format the popupmenu (an expression like statusline), but I never had time to get familiar with that code to check if it was really feasible. This approach would also need a new option and not everyone likes them.
3. how to implement the the extra info additional menu? the issue is that drawing custom component like that in vim is rough, and it would also mean reimplement for each one a lot of the same logic. For example the popupmenu needs to check the text orientation, how much space it has to render and a bunch of other conditions to render correctly. The new extra_info popup would basically have to reimplement all of them! Plus I'd need to implement an algorithm to format the text to render inside that box and that would be another good chunk of conditions. So instead of all this reimplementation crap, wouldn't it be better to have an api to draw windows like the popupmenu to make developing these easier and also give access to the plugins? But again, I didn't really have time to get comfortable with the code and figure out a decent way to do this.
4. would it just be better to change to completion api altogether instead of keeping the current one (with the division defined for the popupmenu?

I probably forgot other issues I though about when trying to come up with a decent design for these features, but these were the main ones. To continue working on this change I wanted to post about these issues and discuss the possible best solutions. But I really didn't have time then and when finally I could I had lost interest. Then the refactoring in https://github.com/neovim/neovim/pull/4432 came along and I was not sure how it would affect this, but now that it is merged, if I understand it right, it is geared towards external GUIs.
To conclude, before going forwards I think it would be better to hear the developers opinions about the issues above.

[Shougo]

I have found gonvim.
It implements the auto completion window.
It is awesome!

https://github.com/dzhou121/gonvim

[justinmk]

Resolution/Summary

• > allowing buffers to "float" at arbitrary coordinates (while improving the buffer API) may be the best way to allow plugins to create arbitrary popups/menus (even "widgets", if you will).
  
  
  
  • Floating windows were implemented in 0.4.x by @bfredl in https://github.com/neovim/neovim/pull/6619 .
  
  
• Externalized popupmenu was implemented in #4432, and is used by several UI clients including Oni, Oni2, gNvim, nvim-gtk, veonim, neovim-qt, etc.
• Further structure around floating-windows is tracked by https://github.com/neovim/neovim/issues/9663 .

---

_Locked to keep the summary visible. You can always chat or open a ticket if you have new information/topics to discuss._