Deoplete.nvim: Excessive CPU usge

[magicalbanana]

Warning: I will close the issue without the minimal init.vim and the reproduction instructions.

Problems summary

Crazy CPU usage

Expected

Should not be excessive

Environment Information

• deoplete version(SHA1):
  c5f8165
• OS:
  OSX
• neovim/Vim version:
  Vim 8
• :checkhealth or :CheckHealth result(neovim only):

Provide a minimal init.vim/vimrc with less than 50 lines (Required!)

" Your minimal init.vim/vimrc
inoremap <expr><tab> pumvisible() ? "\<c-n>" : "\<tab>"

set completeopt+=noselect

"close the preview window after completion is done.
autocmd CompleteDone * pclose!

" enable deoplete
let g:deoplete#enable_at_startup = 1

" deoplete-go
let g:deoplete#sources#go#gocode_binary = $GOPATH.'/bin/gocode'
let g:deoplete#sources#go#sort_class = ['package', 'func', 'type', 'var', 'const']

The reproduce ways from neovim/Vim starting (Required!)

1. Open Vim
2. Do anything
3. htop

Screen shot (if possible)

[pdilyard]

I'm experiencing this as well. I believe it started after an update, so I'm going to see if I can roll back to an older version and find when it started.

[magicalbanana]

@pdilyard yeah, I believe so too. I just can't figure out what commit to rollback to.

[pdilyard]

@magicalbanana I made an educated guess and picked 865747e - it passes CI and is before a bunch of commits about processes and threads. That commit seems to be working just fine, though I don't quite have the time to find the exact commit that introduces the problem.

[nsfm]

I'm also experiencing this after updating a few hours ago. It seems to me that the python3 instances stick around after vim is closed, accumulating until you're stuck at 100% CPU usage.

I have g:deoplete#max_processes = 2 set and I can confirm that two additional python3 processes are left running every time I open and close a file. killall python3 does solve the CPU usage temporarily.

[Shougo]

Unfortunately, I cannot reproduce the problem.
It is hard to fix.

[Shougo]

@pdilyard @nsfm
Are you using Vim8?
If so, I need to find better ways to quit Vim8 correctly.

[Shougo]

Please test the latest master version.

[Shougo]

And please test the patch.

diff --git a/autoload/deoplete/util.vim b/autoload/deoplete/util.vim
index a0d5bcc..53af8f3 100644
--- a/autoload/deoplete/util.vim
+++ b/autoload/deoplete/util.vim
@@ -217,8 +217,10 @@ function! deoplete#util#rpcnotify(event, context) abort
   if deoplete#init#_check_channel()
     return ''
   endif
+
   call s:notify(a:event, a:context)
-  if has('nvim') && a:event ==# 'deoplete_on_event'
+
+  if a:event ==# 'deoplete_on_event'
         \ && a:context['event'] ==# 'VimLeavePre'
     while g:deoplete#_stopped_processes < g:deoplete#max_processes
       sleep 50m

[Shougo]

https://github.com/Shougo/deoplete.nvim/issues/628#issuecomment-363263744

@Shougo I just ran dein#update() and seems to work now (no more errors). However, quiting neovim still hangs.

If so, it is after quitting problem?

[Shougo]

@pdilyard @nsfm Please provide more environment information for me.
The information is important.

Your environment is Mac OSX?

[Shougo]

I think it depends on your configuration or installed plugins.
Because, VimLeavePre autocmd is not called.

I have got similar issue from @wsdjeg.

He has used non nested autocmd and quit Vim in autocmd in SpaceVim.
It does not call VimLeavePre autocmd when quitting.

So please test deoplete with the minimal init.vim.
I think it works.

[andymass]

This is happening on linux too.

When using vim 8, deoplete makes the following processes:

python3 -u /home/user/.vim/plugged/nvim-yarp/pythonx/yarp.py 127.0.0.1:44019 2 deoplete
python3 /home/user/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/dp_main.py 127.0.0.1:5847
python3 /home/user/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/dp_main.py 127.0.0.1:5847
python3 /home/user/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/dp_main.py 127.0.0.1:5847
python3 /home/user/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/dp_main.py 127.0.0.1:5847

After quitting vim this process consumes excessive cpu (> 225%) and does not quit:

python3 -u /home/user/.vim/plugged/nvim-yarp/pythonx/yarp.py 127.0.0.1:44019 2 deoplete

I have reproduced with the following minimal vimrc:

set nocompatible

call plug#begin('~/.vim/plugged')

if has('nvim')
  Plug 'Shougo/deoplete.nvim', { 'do': ':UpdateRemotePlugins' }
else
  Plug 'Shougo/deoplete.nvim'
  Plug 'roxma/nvim-yarp'
  Plug 'roxma/vim-hug-neovim-rpc'
endif

call plug#end()

let g:deoplete#enable_at_startup = 1

[Shougo]

It is reproduced in neovim?

Please test the patch.

https://github.com/Shougo/deoplete.nvim/issues/637#issuecomment-363606021

[andymass]

It is reproduced in neovim?

Not in neovim for me. Only in vim 8.

Please test the patch.

The patch works for me, but adds a noticeable delay to vim 8 quit. There is no delay in neovim.

[Shougo]

OK. So please the patch instead.

diff --git a/autoload/deoplete/util.vim b/autoload/deoplete/util.vim
index a0d5bcc..30e7b43 100644
--- a/autoload/deoplete/util.vim
+++ b/autoload/deoplete/util.vim
@@ -218,10 +218,10 @@ function! deoplete#util#rpcnotify(event, context) abort
     return ''
   endif
   call s:notify(a:event, a:context)
-  if has('nvim') && a:event ==# 'deoplete_on_event'
+  if a:event ==# 'deoplete_on_event'
         \ && a:context['event'] ==# 'VimLeavePre'
     while g:deoplete#_stopped_processes < g:deoplete#max_processes
-      sleep 50m
+      sleep 10m
     endwhile
   endif
   return ''

[andymass]

10m works for me.

By the way, if let g:deoplete#max_processes = 1, is the the asynchronous completion feature disabled? Or what about let g:deoplete#max_processes = 0?

[Shougo]

By the way, if let g:deoplete#max_processes = 1, is the the asynchronous completion feature disabled?

Yes. But it is slower implementation than before.

I will implement serial completion support.
https://github.com/Shougo/deoplete.nvim/issues/638

[Shougo]

Or what about let g:deoplete#max_processes = 0?

It is the same result with let g:deoplete#max_processes = 1.

[andymass]

Okay, thank you for your help and development Shougo

[wessm]

@Shougo since the latest update I do not seem to have any problems anymore with CPU usage, and also no delay when closing neovim!

Thank you!

[Shougo]

OK. I will close the issue later.

[clason]

I unfortunately still have exactly that problem (zombie Python process freezing neovim on exit; no completion) on current master with neovim on macOS (but did not yesterday before the updates); will try to find a minimal reproducible configuration.

[Shougo]

I unfortunately still have exactly that problem (zombie Python process freezing neovim on exit; no completion) on current master with neovim on macOS (but did not yesterday before the updates); will try to find a minimal reproducible configuration.

It is neovim? If so, it seems your configuration or plugins problem.
Please create the minimal vimrc.

[clason]

Indeed, it is with neovim, and certainly a plugin interaction (I suspect it's https://github.com/lervag/vimtex/, specifically the omnifunc source). I'm currently working on the minimal vimrc.

[Shougo]

@clason Please see it.

https://github.com/Shougo/deoplete.nvim/issues/637#issuecomment-363629829

deoplete needs own VimLeavePre autocmd.

[clason]

No, I don't think that's quite it, because a) it worked yesterday and b) the freeze happens while nvim is running (no completion). I'll know more when I isolate the minimal configuration and get the log file.

[zhuangya]

the freeze happens while nvim is running (no completion)

yup, same here, neovim

and below is my plugin list, hope it helps:

https://gist.github.com/zhuangya/156fd40c05c5ab99964b7d774cbcbd8b

[Shougo]

Oh, it seems...

Please test the patch.

https://github.com/Shougo/deoplete.nvim/issues/471#issuecomment-363660845

[clason]

OK, here's the minimal vimrc

call plug#begin('~/.vim/plugged')
    Plug 'lervag/vimtex'
    Plug 'Shougo/deoplete.nvim'
call plug#end()
let g:tex_flavor='latex'
let g:deoplete#enable_at_startup = 1
if !exists('g:deoplete#omni#input_patterns')
  let g:deoplete#omni#input_patterns = {}
endif
let g:deoplete#omni#input_patterns.tex = g:vimtex#re#deoplete

The problem are the lines with the input_patterns; without them I don't see the freezing.

[Shougo]

omni source...
Please test above patch.

I know the slowness problem.

[zhuangya]

not sure if this helps:

(when i launch neovim with deoplete.nvim enabled, and these error msgs appear)

[clason]

Yes, that patch fixes it!

[Shougo]

@zhuangya Please create the minimal vimrc. You have installed many plugins.

Like this.
https://github.com/Shougo/deoplete.nvim/issues/637#issuecomment-363695167

[Shougo]

@zhuangya I think it is the same problem.

https://github.com/Shougo/deoplete.nvim/issues/635

Please check your environment.

[Shougo]

Especially, https://github.com/Shougo/deoplete.nvim/issues/635#issuecomment-363331278

[zhuangya]

@Shougo i CheckHealth and re-install/upgrade neovim and msgpack with pip3 in my neovim3 python environment and now everything works as always

thank you for you patient!

[zhuangya]

btw, my dp_main.py does not raise any error,

however, since my neovim python-client is out-of-date, i upgrade them anyway,then everything back to normal.

[Shougo]

OK. It seems neovim package was broken.

[blueyed]

btw: special-casing of VimLeavePre does not seem to be neccessary with #641.
This fixes the hanging on exiting Neovim for me completely.
I think together with the already merged #631 it should fix the busy loops in case of errors / quitting Vim.

[nsfm]

Just to follow up - I'm using Vim 8.0 with patches 1-1436 under Arch Linux. The supplied patch does solve the issue!

[blueyed]

@nsfm
Which patch?

[nsfm]

@blueyed Shougo's patch from 19 hours ago:

diff --git a/autoload/deoplete/util.vim b/autoload/deoplete/util.vim
index a0d5bcc..30e7b43 100644
--- a/autoload/deoplete/util.vim
+++ b/autoload/deoplete/util.vim
@@ -218,10 +218,10 @@ function! deoplete#util#rpcnotify(event, context) abort
     return ''
   endif
   call s:notify(a:event, a:context)
-  if has('nvim') && a:event ==# 'deoplete_on_event'
+  if a:event ==# 'deoplete_on_event'
         \ && a:context['event'] ==# 'VimLeavePre'
     while g:deoplete#_stopped_processes < g:deoplete#max_processes
-      sleep 50m
+      sleep 10m
     endwhile
   endif
   return ''

After applying this change, the python3 processes die when I close Vim correctly with :q.

However @Shougo if I close Vim with kill -9 (or if I just kill the terminal vim is running in), one python instance remains running indefinitely with a CPU consumption of 130%+:

# ps wauxxxx | grep python3                                                                                                                                                                                                                                                 
nsfm      7541  139  0.2 264468 19348 ?        Ssl  16:22   0:32 python3 -u /home/nsfm/.vim/plugged/nvim-yarp/pythonx/yarp.py 127.0.0.1:57105 2 deoplete
nsfm      7547  0.4  0.0      0     0 ?        Z    16:22   0:00 [python3] <defunct>
nsfm      7549  0.6  0.0      0     0 ?        Z    16:22   0:00 [python3] <defunct>

Indeed if I kill multiple instances of Vim this way, the yarp.py processes continue to pile up.

[blueyed]

https://github.com/Shougo/deoplete.nvim/pull/641 gets rid of this whole waiting, please try it also.
See https://help.github.com/articles/checking-out-pull-requests-locally/.

[Shougo]

I have merged #641.
Please test the latest version.