Neovim: Segfault During Build (helptags)

[treed]

I'm getting a segfault during build:

[..SNIP..]
-- Installing: /home/treed/neovim/share/nvim/runtime/ftplugin.vim
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/pltags.pl
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/efm_filter.pl
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/shtags.pl
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/efm_perl.pl
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/doc/maketags.awk
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/doc/makehtml.awk
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/mve.awk
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/macros/less.sh
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/vimspell.sh
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/ref
-- Up-to-date: /home/treed/neovim/share/nvim/runtime/tools/vimm
-- Generating helptags in /home/treed/neovim/share/nvim/runtime/doc.
CMake Error at /home/treed/neovim/cmake/GenerateHelptags.cmake:25 (message):
  Generating helptags failed: - Segmentation fault
Call Stack (most recent call first):
  cmake_install.cmake:516 (INCLUDE)


make[1]: *** [install] Error 1
make[1]: Leaving directory `/home/treed/neovim/build'
make: *** [install] Error 2
INFO[0236] The command [/bin/sh -c cd neovim && make CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=$HOME/neovim" install] returned a non-zero code: 2

This is on a fresh build for a Docker container. The Dockerfile is at https://github.com/treed/dotfiles/blob/master/Dockerfile.neovim, which derives from https://github.com/treed/dotfiles/blob/master/Dockerfile

Is there an extra log file that would have details of the segfault I can pass along?

[ghost]

Are you absolutely sure that you're running the latest version of nvim? This sounds just like the issue that was fixed by https://github.com/neovim/neovim/pull/2263.

[treed]

After you asked that, I thought maybe I was getting bit by Docker's build cache, but I just tried again with --no-cache and watched it clone a fresh copy:

Step 3 : RUN git clone https://github.com/neovim/neovim.git
 ---> Running in 1386173e88c2
Cloning into 'neovim'...
---> fd45b2b16026
Removing intermediate container 1386173e88c2

and it still fails in the same way.

I've pasted the entirety of the docker build output here: http://sprunge.us/NUET

[fwalch]

There's also some problems with generating helptags on Arch Linux, but only with certain AUR helpers (doesn't work with pacaur, works with packer and plain makepkg). Could it depend on the working directory or something?

[michaeljones]

I am seeing this as well using this very similar docker file:

https://github.com/FelikZ/docker-neovim/blob/master/Dockerfile

I'm not deep into either docker or neovim but it seems like a nice set up for playing around with neovim & different GUI options.

[tarruda]

Are you sure this is a SEGFAULT? I had a similar problem, but debugging with gdb showed that :helptags is failing due to a duplicate tag. Removing the installation directory and reinstalling everything fixed for me.

[michaeljones]

I'm not well qualified to comment but the docker file approach I reported above does a fresh build and I get the error in the original post above which includes Generating helptags failed: - Segmentation fault. (It did this for me a couple of minutes ago with the latest master.)

It builds fine for me outside of docker. Or it did last time I double checked anyway. I thought it might be something to do with the availability of the tty, though I don't know enough to know what the right term to use there is.

[tarruda]

Then it is a different problem. @treed do you get a core dump from the segfault? You use this command to save a backtrace of the crashed program: gdb -q -n -ex bt -batch ./build/bin/nvim core > backtrace.txt

If no core dump is generated, then you may need to enable it with ulimit -c unlimited

[michaeljones]

No luck I'm afraid, I need more experience but this is my docker file attempt:

FROM ubuntu:14.04

RUN \
    apt-get update && \
    apt-get install -y libtool autoconf automake cmake libncurses5-dev g++ pkg-config unzip git curl

RUN \
    ulimit -c unlimited && \
    git clone https://github.com/neovim/neovim.git nvim && \
    cd nvim && \
    make 

RUN apt-get install -y gdb

RUN \
    ulimit -c unlimited && \
    cd nvim && \
    ./build/bin/nvim -u NONE -esX -c "helptags ++t ." -c quit || \
    find / -name "core*" && false

ENTRYPOINT nvim

I get a Segmentation fault (core dumped) in the output but the find command doesn't find a core file. Altering it to find all files created in the last 5 minutes doesn't seem to reveal anything either. Googling docker & core-dump suggests that it isn't necessary a given that they are created.

I've tried something like:

gdb -batch -ex "run" -ex "bt" --args ./build/bin/nvim -u NONE -esX -c "helptags ++t ." -c quit

to try to get a backtrace straight from gdb when running the command but that fails during initialisation of the program with this kind of error:

During startup program terminated with signal SIGSEGV, Segmentation fault.
No stack.

I'm taking a break for now as I don't really know what I'm doing!

[melon3r]

I had a similar issue today. I removed neovim from the system completely and started over and it worked.

[treed]

I'll check to see if I can get a core file later, but I should note that every time I build, it's a completely fresh build from master and completely fresh OS root, so any approach that involves removing anything and starting over won't work here, since that happens on every build anyway.

[ditsing]

I'm not sure if it's the same issue with mine, but I was compiling neovim from master on Ubuntu 14.04 and got the following error while running make install:

Generating helptags failed:
[?1049h[?2004h[?12l[?25h[?25lError
detected while processing command
line:[?12l[?25h[?25lE154:
Duplicate tag "script-here" in file
./if_pyth.txt[?1002l[?1006l(B[?12l[?25h[?1049l[?2004l[?12l[?25h

• 1
  Call Stack (most recent call first):
  cmake_install.cmake:516 (INCLUDE)

In the (messy) message we can see if_pyth.txt and the script-here tag is mentioned. I tried to rename the tag script-here in runtime/doc/if_pyth.txt to something else and the error was gone.

I have absolutely no knowledge of cmake or helptags or anything, hope somebody cloud fix it.

[blueyed]

@tarruda
I've seen the same problem, where removing /usr/local/share/nvim fixed it:

-- Generating helptags in /usr/local/share/nvim/runtime/doc.
CMake Error at /home/daniel/Vcs/neovim/cmake/GenerateHelptags.cmake:25 (message):
  Generating helptags failed:

  - 1
Call Stack (most recent call first):
  cmake_install.cmake:516 (include)

It would be nice if this could be more verbose, maybe by removing the -s in cmake/GenerateHelptags.cmake:16?

But then @ditsing appears to see the error message..

[tarruda]

@blueyed @ditsing error can be fixed by removing the existing runtime directory, but I think @treed is experiencing a different error, I need a backtrace from the core dump to determine the cause

[blueyed]

@tarruda

error can be fixed by removing the existing runtime directory

Yes, that worked for me. But I only found the workaround when seeing this issue here, so making the :helptags more verbose / less silent might have helped already.

[tarruda]

Yes, that worked for me. But I only found the workaround when seeing this issue here, so making the :helptags more verbose / less silent might have helped already.

:+1: Created #2421 for this

[Hettomei]

I had the same problem yesterday.

And today after make install fail, I read " Remove "/usr/local/share/nvim/runtime/doc" and try again" and it worked :)
Thank you.

[ghost]

I'll close this, but please feel free to reopen this @treed if the issue isn't fixed.

[michaeljones]

I'm still getting:

-- Installing: /usr/local/share/nvim/runtime/compiler/ruby.vim
-- Installing: /usr/local/share/nvim/runtime/compiler/xmllint.vim
-- Installing: /usr/local/share/nvim/runtime/compiler/fortran_lf95.vim
-- Installing: /usr/local/share/nvim/runtime/compiler/decada.vim
-- Installing: /usr/local/share/nvim/runtime/compiler/rspec.vim
-- Installing: /usr/local/share/nvim/runtime/menu.vim
-- Installing: /usr/local/share/nvim/runtime/ftplugof.vim
-- Installing: /usr/local/share/nvim/runtime/ftoff.vim
-- Up-to-date: /usr/local/share/nvim/runtime/doc/maketags.awk
-- Up-to-date: /usr/local/share/nvim/runtime/doc/makehtml.awk
-- Up-to-date: /usr/local/share/nvim/runtime/macros/less.sh
-- Generating helptags in /usr/local/share/nvim/runtime/doc.
-- /usr/local/share/nvim/runtime/doc already exists
CMake Error at /nvim/cmake/GenerateHelptags.cmake:31 (message):
  Generating helptags failed: - Segmentation fault

  Remove "/usr/local/share/nvim/runtime/doc" and try again
Call Stack (most recent call first):
  cmake_install.cmake:196 (INCLUDE)


make[1]: *** [install] Error 1

So the error has changed. This is running on a clean docker container with --no-cache to make sure it is super clean. I'm totally clueless about the build procedure but it looks like it must be creating that folder and then complaining that it is already there? Not sure.

I hope this helps a little.

[blueyed]

@michaeljones
Can you enter the container and run the command manually?
Then you could emit the -s (https://github.com/neovim/neovim/pull/2421#issuecomment-92352934) and also use it through gdb.

[blueyed]

@michaeljones
Can you provide the Dockerfile, so others (@tarruda) could reproduce it?

[FelikZ]

@Blueyed https://github.com/FelikZ/docker-neovim/blob/master/Dockerfile

[michaeljones]

Thanks @FelikZ.

@blueyed if I enter the container and run the command manually then it works. I'm sorry for not including that information. I don't know the correct terminology but I assume it is something to do with access to the 'tty' perhaps? Related to that fact that the helptags step actually boots up nvim as far as I can tell.

[michaeljones]

Here is the content of a Dockerfile which separates out the failing command so that the rest is cached for quicker iterations:

FROM ubuntu:14.04

RUN \
    apt-get update && \
    apt-get install -y libtool autoconf automake cmake libncurses5-dev g++ pkg-config unzip git curl

RUN \
    ulimit -c unlimited && \
    git clone https://github.com/neovim/neovim.git nvim && \
    cd nvim && \
    make

RUN \
    cd nvim && \
    ./build/bin/nvim -u None -esX -c "helptags ++t ." -c quit

ENTRYPOINT nvim

I tried removing the -s flag but it didn't seem to have any impact on the output. Though with this Dockerfile the output is different to the output shown above as we're not running make install here.

[yshui]

I have seen a similar crash as well. It's at main.c line 1365, when calling fileno on stderr.

[FelikZ]

@Michaeljones can you provide your docker run full command?

[treed]

I've been unable to get an actual coredump out of the build, but the dockerfile I linked above also reproduces this problem.

[michaeljones]

@FelikZ I'm on a different machine at the moment but I'm pretty sure it is:

sudo docker build -t nvim .

In the same directory as the Dockerfile. I am quite new to docker though.

[FelikZ]

@tarruda I have found a 5709181243536f0a9f23dc4a8d87fe75cc32537d that introduced this build bug.

Dockerfile for success with commit 5709181243536f0a9f23dc4a8d87fe75cc32537d^
Dockerfile for fail with commit 5709181243536f0a9f23dc4a8d87fe75cc32537d

[jszakmeister]

I have seen a similar crash as well. It's at main.c line 1365, when calling fileno on stderr.

That smells like stderr is somehow set to NULL. :-(

[unphased]

I had to apply the sudo rm /usr/local/share/nvim/runtime/doc method to proceed to build neovim (for example just now, commit df3abf4). Both on Centos 7 and OS X 10.10.3. Neither of which are hosting or running under docker.

[jakubgs]

I am still experiencing this issue when trying to create a deb package using checkinstall. It works when just doing make install.

I have tried removing /usr/local/share/nvim/runtime/doc or the whole /usr/local/share/nvim. Doesn't help. I still get:

$ sudo checkinstall --default --type=debian --pkgname=neovim --nodoc --provides=neovim --requires=perl,python
...
-- Up-to-date: /usr/local/share/nvim/runtime/doc/makehtml.awk
-- Up-to-date: /usr/local/share/nvim/runtime/doc/maketags.awk
-- Up-to-date: /usr/local/share/nvim/runtime/macros/less.sh
-- Generating helptags in /usr/local/share/nvim/runtime/doc.
-- /usr/local/share/nvim/runtime/doc already exists
CMake Error at /home/sochan/soft/neovim/cmake/GenerateHelptags.cmake:31 (message):
  Generating helptags failed: - Segmentation fault

  Remove "/usr/local/share/nvim/runtime/doc" and try again
Call Stack (most recent call first):
  cmake_install.cmake:239 (include)


Makefile:85: recipe for target 'install' failed
make[1]: *** [install] Error 1
make[1]: Leaving directory '/home/sochan/soft/neovim/build'
Makefile:103: recipe for target 'install' failed
make: *** [install] Error 2

****  Installation failed. Aborting package creation.

[fwalch]

Can you get a coredump to check what exactly the problem is?

[jakubgs]

Sure thing, here you go.

[yshui]

@PonderingGrower I think you also need to provide the result binary which generates this coredump.

[fwalch]

Actually, I meant for you to take a look a the coredump and post a backtrace :-D

[jakubgs]

Well now I feel silly.

This is a backtrace I've gotten:

#0  0x0000000000000000 in ?? ()
#1  0x00007f28ce4a3cc6 in readlink () from /usr/lib/checkinstall/installwatch.so
#2  0x0000000000625f7e in malloc_conf_init () at src/jemalloc.c:440
#3  0x0000000000626869 in malloc_init_hard () at src/jemalloc.c:682
#4  0x00000000006295d2 in malloc_init () at src/jemalloc.c:292
#5  calloc (num=num@entry=1, size=size@entry=32) at src/jemalloc.c:1123
#6  0x00007f28cdc5c640 in _dlerror_run (operate=operate@entry=0x7f28cdc5c130 <dlsym_doit>, args=args@entry=0x7ffc5b20f6d0) at dlerror.c:141
#7  0x00007f28cdc5c198 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#8  0x00007f28ce49d9d9 in ?? () from /usr/lib/checkinstall/installwatch.so
#9  0x00007f28ce49e000 in _init () from /usr/lib/checkinstall/installwatch.so
#10 0x00007f28ce6be9bd in call_init (l=0x7f28ce8cf4c8, argc=argc@entry=8, argv=argv@entry=0x7ffc5b20f7a8, env=env@entry=0x7ffc5b20f7f0) at dl-init.c:64
#11 0x00007f28ce6beae3 in call_init (env=0x7ffc5b20f7f0, argv=0x7ffc5b20f7a8, argc=8, l=<optimized out>) at dl-init.c:36
#12 _dl_init (main_map=0x7f28ce8d21a8, argc=8, argv=0x7ffc5b20f7a8, env=0x7ffc5b20f7f0) at dl-init.c:126
#13 0x00007f28ce6b11ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#14 0x0000000000000008 in ?? ()
#15 0x00007ffc5b2117e4 in ?? ()
#16 0x00007ffc5b211809 in ?? ()
#17 0x00007ffc5b21180c in ?? ()
#18 0x00007ffc5b211811 in ?? ()
#19 0x00007ffc5b211816 in ?? ()
#20 0x00007ffc5b211819 in ?? ()
#21 0x00007ffc5b211828 in ?? ()
#22 0x00007ffc5b21182b in ?? ()
#23 0x0000000000000000 in ?? ()

And full backtrace.

[yshui]

@PonderingGrower you need to build neovim with debug symbol.

[jakubgs]

How do I do that? I tried adding -g and -ggdb to CC and objdump --syms does show debug symbols, but when I try to use gdb to check the backtrace I still get ?? for the last 10 calls.

[jszakmeister]

RUN \
ulimit -c unlimited && \
git clone https://github.com/neovim/neovim.git nvim && \
cd nvim && \
make

I believe, given your dockerfile, you may already be building binaries with debug.

Given your backtrace, I'm not sure it's Neovim that is broken here. It looks like CheckInstall is intercepting things, and maybe it's causing the problem.

[fwalch]

Can you try again with latest master? There has been a change to the helptags generation.

[jakubgs]

I'm afraid it still fails, and yes, I'm on commit 6609a54.

[rdnelson]

I spent some time today looking into this, and this is indeed a bug in checkinstall, but it actually has nothing to do with helptags. Instead, it has to do with neovim's use of jemalloc. Checkinstall assumes that the dlfcn.h functions will not call any filesystem functions. Unfortunately jemalloc calls readline, which causes this assumption to break, hence the segfault.

If you want to get checkinstall working again you can get the source for it from http://checkinstall.izto.org/checkinstall.git and apply this patch to fix the bug. It causes checkinstall to return ENOENT when je_malloc checks for the /etc/malloc.conf symlink. This might cause problems if you have malloc options set in /etc/malloc.conf, but otherwise, it should be fine.

Unfortunately, this doesn't look like a problem that will go away as long as jemalloc sticks around, so if this patch gets a bit more testing from others, then this might belong on the wiki.

[fxdgear]

I just wanted to throw my 2 cents into the ring using Alpine linux

Same with this comment
from @FelikZ my build works with this SHA 5709181243536f0a9f23dc4a8d87fe75cc32537d

RUN apk add --update git alpine-sdk libtool cmake automake m4 autoconf linux-headers unzip ncurses-dev python py-pip && rm -rf /var/cache/apk/*

WORKDIR /tmp

RUN git clone https://github.com/neovim/libtermkey.git && \
  cd libtermkey && \
  make && \
  make install && \
  cd ../ && rm -rf libtermkey

RUN git clone https://github.com/neovim/libvterm.git && \
  cd libvterm && \
  make && \
  make install && \
  cd ../ && rm -rf libvterm

RUN git clone https://github.com/neovim/unibilium.git && \
  cd unibilium && \
  make && \
  make install && \
  cd ../ && rm -rf unibilium

RUN  git clone https://github.com/neovim/neovim.git && \
  cd neovim && \
  git checkout 5709181243536f0a9f23dc4a8d87fe75cc32537d^ && \
  make && \
  make install && \
  cd ../ && rm -rf nvim

[justinmk]

Should be fixed since https://github.com/neovim/neovim/pull/3546

[zachgersh]

@justinmk I just tried building this in alpine (by grabbing master) and I am still running into what I believe is the same problem:

make[3]: Entering directory '/tmp/neovim/build'
[100%] Generating syntax/vim/generated.vim
[100%] Generating copy_docfiles
[100%] Generating doc/tags
Segmentation fault
make[3]: *** [runtime/doc/tags] Error 139
make[2]: *** [runtime/CMakeFiles/runtime.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [nvim] Error 2
runtime/CMakeFiles/runtime.dir/build.make:70: recipe for target 'runtime/doc/tags' failed
make[3]: Leaving directory '/tmp/neovim/build'
CMakeFiles/Makefile2:620: recipe for target 'runtime/CMakeFiles/runtime.dir/all' failed
make[2]: Leaving directory '/tmp/neovim/build'
Makefile:149: recipe for target 'all' failed
make[1]: Leaving directory '/tmp/neovim/build'
Makefile:55: recipe for target 'nvim' failed

[justinmk]

@zachgersh Are you sure it's not https://github.com/neovim/neovim/issues/4183? See this comment:

https://github.com/neovim/neovim/issues/4183#issuecomment-181291487

Please try the same steps, then rm -rf build/ to remove the cmake cache, then build again.

[zachgersh]

@justinmk - are you saying in the same dockerifle build, wait for it to fail, rm -rf build/, and then run the build again?

Also wanted to note it passed the _FORTIFY_SOURCE_ACCEPTABLE check from the other issue.

[justinmk]

are you saying in the same dockerifle build, wait for it to fail, rm -rf build/, and then run the build again?

No. As mentioned in https://github.com/neovim/neovim/issues/4183#issuecomment-181291487 , some environments get past our _FORTIFY_SOURCE_ACCEPTABLE check. Please force -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 (as explained in https://github.com/neovim/neovim/issues/4183#issuecomment-181291487 ) then try again. To try again, rm -rf build/ ensures that cmake does not reuse the previous config.

[zachgersh]

@justinmk that actually didn't have any affect. Decided to go with disabling jemalloc and that did the trick.

[justinmk]

@zachgersh Thanks for the followup. What's different about "Alpine" that makes jemalloc a problem? See also https://github.com/neovim/neovim/issues/3434

[zachgersh]

@justinmk - I am actually not sure. I would love to be able to give you some more debug output or the Dockerfile I am using to reproduce the problem. What's really weird to me is that my failure case presents as this bug but the fix is what was done for #3434. I am hoping that it isn't just a red herring.

[fxdgear]

FWIW using my dockerfile https://github.com/neovim/neovim/issues/2364#issuecomment-144010069

I'm still getting

runtime/CMakeFiles/runtime.dir/build.make:70: recipe for target 'runtime/doc/tags' failed
Bus error
make[3]: *** [runtime/doc/tags] Error 135
make[2]: *** [runtime/CMakeFiles/runtime.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [nvim] Error 2
make[3]: Leaving directory '/tmp/neovim/build'
CMakeFiles/Makefile2:620: recipe for target 'runtime/CMakeFiles/runtime.dir/all' failed
make[2]: Leaving directory '/tmp/neovim/build'
Makefile:149: recipe for target 'all' failed
make[1]: Leaving directory '/tmp/neovim/build'
Makefile:60: recipe for target 'nvim' failed
The command '/bin/sh -c git clone https://github.com/neovim/neovim.git &&   cd neovim &&   make' returned a non-zero code: 2

_edit_ actually it's not the exact same. I removed this line: git checkout 5709181243536f0a9f23dc4a8d87fe75cc32537d^ && \

[laur89]

Issue reproduces also on Debian 4.3.5-1 (non-docker).

[andrewrynhard]

I am building in the same environment as @zachgersh and getting

Scanning dependencies of target runtime
make[3]: Leaving directory '/root/neovim-0.1.4/build'
make[3]: Entering directory '/root/neovim-0.1.4/build'
[100%] Generating syntax/vim/generated.vim
[100%] Generating copy_docfiles
[100%] Generating doc/tags
/bin/sh: line 1: 32756 Segmentation fault      (core dumped) /root/neovim-0.1.4/build/bin/nvim -u NONE -i NONE -e --headless -c helptags\ ++t\ . -c quit
make[3]: *** [runtime/CMakeFiles/runtime.dir/build.make:71: runtime/doc/tags] Error 139
make[3]: Leaving directory '/root/neovim-0.1.4/build'
make[2]: *** [CMakeFiles/Makefile2:621: runtime/CMakeFiles/runtime.dir/all] Error 2
make[2]: Leaving directory '/root/neovim-0.1.4/build'
make[1]: *** [Makefile:139: all] Error 2
make[1]: Leaving directory '/root/neovim-0.1.4/build'
make: *** [Makefile:60: nvim] Error 2

[jorgicio]

This issue is still not fixed, and as mentioned in a comment above, disabling jemalloc did the trick.
Tried with Gentoo Linux, with cmake 3.5.2.