Youcompleteme: go_completer.py should perhaps expect to get literal `null` from `gocode`

[filmil]

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside
the brackets) _before_ filing your issue:

• [X ] I have read and understood YCM's CONTRIBUTING document.
• [ X] I have read and understood YCM's CODE_OF_CONDUCT document.
• [ X] I have read and understood YCM's README, especially the
  Frequently Asked Questions section.
• [X] I have searched YCM's issue tracker to find issues similar to the one I'm
  about to report and couldn't find an answer to my problem. (Example Google
  search.)
• [X] If filing a bug report, I have included the output of vim --version.
• [X] If filing a bug report, I have included the output of :YcmDebugInfo.
• [ ] If filing a bug report, I have attached the contents of the logfiles using
  the :YcmToggleLogs command.
• [ ] If filing a bug report, I have included which OS (including specific OS
  version) I am using.
• [] If filing a bug report, I have included a minimal test case that reproduces
  my issue, including what I expected to happen and what actually happened.
• [ ] If filing a installation failure report, I have included the entire output
  of install.py (or cmake/make/ninja) including its invocation
• [X ] I understand this is an open-source project staffed by volunteers and
  that any help I receive is a selfless, heartfelt _gift_ of their free time. I
  know I am not entitled to anything and will be polite and courteous.
• [X] I understand my issue may be closed if it becomes obvious I didn't
  actually perform all of these steps.

Thank you for adhering to this process! It ensures your issue is resolved
quickly and that neither your nor our time is needlessly wasted.

Issue Details

Provide a clear description of the problem, including the following key
questions:

• What did you do?

I tried a trivial autocomplete in my code.

Include steps to reproduce here.

Sorry, can't share the code with you. I tried to use autocomplete from gocode

Include description of a minimal test case, including any actual code required
to reproduce the issue.

• What did you expect to happen?

YCM should log an error message that means it encountered an unexpected output

Include description of the expected behaviour.

• What actually happened?

I see error like:

TypeError: object of type 'NoneType' has no len

This happens in go's autocompleter code, in the line len (resultdata) != 2. len fails because resultdata == None at this point.

go_completer.py

    if len( resultdata ) != 2:
      _logger.error( GOCODE_NO_COMPLETIONS_MESSAGE )
      raise RuntimeError( GOCODE_NO_COMPLETIONS_MESSAGE )
    for result in resultdata[ 1 ]:
      if result.get( 'class' ) == 'PANIC':
        raise RuntimeError( GOCODE_PANIC_MESSAGE )

I dug a bit and it seems that this is because, for whatever reason, gocode returns the literal value null in its output instead of a non-null JSON object.
null then gets unmarshalled into None and the result fails.

I added a debug logging statement to display output from gocode and here's what it gives me:

2018-09-18 10:40:03,390 - ERROR - stdoutdata; null

2018-09-18 10:40:03,391 - ERROR - Exception from semantic completer (using general): Traceback (most recent call last):
  File "/REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/handlers.py", line 103, in GetCompletions
    .ComputeCandidates( request_data ) )
  File "/REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/completers/completer.py", line 231, in ComputeCandidates
    candidates = self._GetCandidatesFromSubclass( request_data )
  File "/REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/completers/completer.py", line 242, in _GetCandidatesFromSubclass
    raw_completions = self.ComputeCandidatesInner( request_data )
  File "/REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/ycmd/../ycmd/completers/go/go_completer.py", line 154, in ComputeCandidatesInner
    _logger.error("resultdata; " + resultdata)
TypeError: coercing to Unicode: need string or buffer, NoneType found

Here's the code snippet with my modifications for added logging;

    # NOTE: Offsets sent to gocode are byte offsets, so using start_column
    # with contents (a bytes instance) above is correct.
    offset = _ComputeOffset( contents,
                             request_data[ 'line_num' ],
                             request_data[ 'start_column' ] )

    _logger.info( "binary path: " + self._gocode_binary_path)
    stdoutdata = self._ExecuteCommand( [ self._gocode_binary_path,
                                         '-debug',
                                         '-sock', 'tcp',
                                         '-addr', self._gocode_host,
                                         '-f=json', 'autocomplete',
                                         filename, str( offset ) ],
                                       contents = contents )

    try:
      _logger.error("stdoutdata; " + stdoutdata)
      resultdata = json.loads( ToUnicode( stdoutdata ) )
      _logger.error("resultdata; " + resultdata)
    except ValueError:
      _logger.error( GOCODE_PARSE_ERROR_MESSAGE )
      raise RuntimeError( GOCODE_PARSE_ERROR_MESSAGE )

:YcmDebugInfo, some info redacted

Printing YouCompleteMe debug information...
-- Client logfile: /REDACTED/tmp/ycm_WGn62X.log
-- Server Python interpreter: /usr/bin/python2
-- Server Python version: 2.7.13
-- Server has Clang support compiled in: True
-- Clang version: clang version 6.0.0 (tags/RELEASE_600/final)
-- No extra configuration file found
-- Go completer debug information:
--   Gocode running at: http://127.0.0.1:41045
--   Gocode process ID: 144269
--   Gocode executable: /REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/third_party/gocode/gocode
--   Gocode logfiles:
--     /REDACTED/tmp/gocode_41045_stdout_YmqknE.log
--     /REDACTED/tmp/gocode_41045_stderr_EupXn8.log
--   Godef executable: /REDACTED/.vim/bundle/YouCompleteMe/third_party/ycmd/third_party/godef/godef
-- Server running at: http://127.0.0.1:49371
-- Server process ID: 144233
-- Server logfiles:
--   /REDACTED/ycmd_49371_stdout_NH80L1.log
--   /REDACTED/ycmd_49371_stderr_bsy89a.log 

Include description of the observed behaviour, including actual output,
screenshots, etc.

Diagnostic data

Output of vim --version

```
─>$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Feb 20 2018 15:50:47)
Included patches: 1-1401
Modified by [email protected]
Compiled by [email protected]
Huge version with GTK2 GUI. Features included (+) or not (-):
+acl +farsi +mouse_sgr -tag_any_white
+arabic +file_in_path -mouse_sysmouse +tcl
+autocmd +find_in_path +mouse_urxvt +termguicolors
-autoservername +float +mouse_xterm +terminal
+balloon_eval +folding +multi_byte +terminfo
+balloon_eval_term -footer +multi_lang +termresponse
+browse +fork() -mzscheme +textobjects
++builtin_terms +gettext +netbeans_intg +timers
+byte_offset -hangul_input +num64 +title
+channel +iconv +packages +toolbar
+cindent +insert_expand +path_extra +user_commands
+clientserver +job +perl +vertsplit
+clipboard +jumplist +persistent_undo +virtualedit
+cmdline_compl +keymap +postscript +visual
+cmdline_hist +lambda +printer +visualextra
+cmdline_info +langmap +profile +viminfo
+comments +libcall +python +vreplace
+conceal +linebreak -python3 +wildignore
+cryptv +lispindent +quickfix +wildmenu
+cscope +listcmds +reltime +windows
+cursorbind +localmap +rightleft +writebackup
+cursorshape +lua +ruby +X11
+dialog_con_gui +menu +scrollbind -xfontset
+diff +mksession +signs +xim
+digraphs +modify_fname +smartindent +xpm
+dnd +mouse +startuptime +xsmp_interact
-ebcdic +mouseshape +statusline +xterm_clipboard
+emacs_tags +mouse_dec -sun_workshop -xterm_save
+eval +mouse_gpm +syntax
+ex_extra -mouse_jsbterm +tag_binary
+extra_search +mouse_netterm +tag_old_static
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
system gvimrc file: "$VIM/gvimrc"
user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
defaults file: "$VIMRUNTIME/defaults.vim"
system menu file: "$VIMRUNTIME/menu.vim"
fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK -pthread -I/usr/include/gtk-2.0 -I/usr/lib/x86_64-linux-gnu/gtk-2.0/include -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/freetype2 -Wdate-time -g -O2 -fdebug-prefix-map=/tmp/build-debs.U8UMDD/vim-8.0.1401-2=. -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim -lgtk-x11-2.0 -lgdk-x11-2.0 -lpangocairo-1.0 -latk-1.0 -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lpangoft2-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lfontconfig -lfreetype -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE -lm -ltinfo -lnsl -lselinux -lacl -lattr -lgpm -ldl -L/usr/lib -llua5.2 -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.24/CORE -lperl -ldl -lm -lpthread -lcrypt -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpthread -lieee -lm -lruby-2.3 -lpthread -lgmp -ldl -lcrypt -lm
````

Place the output here, or a link to a gist.

Output of YcmDebugInfo

Place the output here, or a link to a gist.

Contents of YCM, ycmd and completion engine logfiles

Add let g:ycm_log_level = 'debug' to vimrc, restart Vim, reproduce the
issue, and include link here to a gist containing the entire logfiles for
ycm, ycmd and any completer logfiles listed by :YcmToggleLogs.

OS version, distribution, etc.

Include system information here.

Output of build/install commands

Include link to a gist containing the invocation and entire output of
install.py if reporting an installation issue.

[filmil]

Sorry for not looking up all the information. I think however that I've dug deep enough to show what the underlying issue is so that it can be fixed: YCM does not expect a value can be None, but it is.

It may be helpful to emit a user-friendlier message in case this happens, just so that it is obvious what is going on. The generic mesage is kind of OK if you have a lot of YCM context. But note that most folks who step on errors won't be YCM mavens.

There seems to be a separate issue with gocode whereby it returns null on my particular query, but I'll dig into that separately.

[micbou]

Thanks for the report. This has been fixed in PR https://github.com/Valloric/ycmd/pull/1098 and will be part of YCM once we update the ycmd submodule.

[bstaletic]

To get the update right away, do the following:

• cd /path/to/YouCompleteMe/third_party/ycmd
• git checkout master
• git pull
• git submodule update --init --recursive
• cd ../..
• `./install.py --go-completer