Mastodon: Error installing cld2 gem with gcc 6+

[yookoala]

Problem

The current Gemfile includes cld2 for language detection functionality, which seems to be incompatible with gcc-6+.

I tried to do a simple bundle install on my Ubuntu 16.10 machine (which comes with gcc 6.2.0 by default). When processing the cld2 gem installation (effectively gem install cld2 -v '1.0.3'), I got some spectacular error messages:

Building native extensions.  This could take a while...
ERROR:  Error installing cld2:
    ERROR: Failed to build gem native extension.

    current directory: /home/foobar/.rvm/gems/ruby-2.4.1/gems/cld2-1.0.3/ext/cld
/home/foobar/.rvm/rubies/ruby-2.4.1/bin/ruby -r ./siteconf20170505-6029-169rn4c.rb extconf.rb
checking for -lstdc++... yes
creating Makefile

current directory: /home/foobar/.rvm/gems/ruby-2.4.1/gems/cld2-1.0.3/ext/cld
make "DESTDIR=" clean

current directory: /home/foobar/.rvm/gems/ruby-2.4.1/gems/cld2-1.0.3/ext/cld
make "DESTDIR="
compiling internal/cldutil.cc
cc1plus: warning: command line option ‘-Wimplicit-int’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wdeclaration-after-statement’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wimplicit-function-declaration’ is valid for C/ObjC but not for C++
In file included from internal/cldutil.h:24:0,
                 from internal/cldutil.cc:20:
internal/cldutil_shared.h:404:54: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
                                       uint32 quadhash) {
                                                      ^
internal/cldutil_shared.h:432:62: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
                                           uint64 longwordhash) {
                                                              ^
cc1plus: warning: unrecognized command line option ‘-Wno-self-assign’

...
(skipped)
...

internal/cld_generated_cjk_uni_prop_80.cc:7089:1: error: narrowing conversion of ‘-9’ from ‘int’ to ‘CLD2::uint8 {aka unsigned char}’ inside { } [-Wnarrowing]
internal/cld_generated_cjk_uni_prop_80.cc:7089:1: error: narrowing conversion of ‘-9’ from ‘int’ to ‘CLD2::uint8 {aka unsigned char}’ inside { } [-Wnarrowing]
cc1plus: warning: unrecognized command line option ‘-Wno-self-assign’
cc1plus: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1plus: warning: unrecognized command line option ‘-Wno-parentheses-equality’
Makefile:209: recipe for target 'internal/cld_generated_cjk_uni_prop_80.o' failed
make: *** [internal/cld_generated_cjk_uni_prop_80.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/foobar/.rvm/gems/ruby-2.4.1/gems/cld2-1.0.3 for inspection.
Results logged to /home/foobar/.rvm/gems/ruby-2.4.1/extensions/x86_64-linux/2.4.0/cld2-1.0.3/gem_make.out

Temp Solution (Need Documentation)

I found similar issues on some gist in Github. Then I tried this(*) before another bundle install:

sudo apt-get install gcc-5 g++-5 -y
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5;

Then bundle install works.

Probably need to document this somewhere. Just don't know where...

(*) Need to sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 62 --slave /usr/bin/g++ g++ /usr/bin/g++-6 afterwards to revert default gcc version to 6.2.0.

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [ ] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[lexpierce]

Solution only works on platform which has a 'compat' gcc available. This does not work on Fedora 25, for example, which is 100% gcc-6

[Komic]

Same thing here (I should have opened a bug from the beginning). I've tried fiddling with the -Wno-narrowing flag, but without success (don't trust me though, someone else probably should give it a shot). While downgrading gcc works, it does feel a bit dirty imo. :/

[mjankowski]

Yes, I agree that "install a different gcc version" is not an acceptable fix here.

Can you try the flag on this pr with -std=c++03 ?

[mikoim]

cld2 looks to be no longer maintained. Another method is to replace cld2 with cld3.

[Wonderfall]

This must be resolved as soon as possible. Debian Stretch will soon be released and the official ruby image based on Alpine Linux may update in the future (Alpine 3.5 already makes use of gcc6). Both will use GCC 6+.

[Gargron]

Wow! Looks like cld3 was created by @akihikodaki!

[askl56]

Is cld3 production ready? I'm running into the same problem.

[akihikodaki]

Testing my FFI implementation for CLD3 on my instance, o.kagucho.net. It looks fine.

See 8116a031105ce2fdf09ce7aee211fc7b90e8fc24 for a change to use the gem in Mastodon. I will make a pull request soon.

CLD3 has some concerns when it comes to use in the real world; it requires C++11, Google's protocol buffers, and the FFI is young and may have potential problems; see the commit log for details.

[lexpierce]

CLD3 seems to work on my instance as well, with the above referenced changes.

[thurloat]

I've upgraded to cld3 on mastodon.club as well using @akihikodaki 's patch. Looks like it's running smoothly. Installed additional Dockerfile deps in alpine as well where g++ is already 6.3 in edge https://pkgs.alpinelinux.org/packages?name=g%2B%2B&branch=edge&repo=&arch=x86_64&maintainer=

Dockerfile additions are here: https://github.com/Moosetodon/mastodon/blob/master/Dockerfile#L32

[Wonderfall]

@thurloat g++ is already included in build-base (which is a metapackage). If you want to try with 6.3 you should put g++@edge instead.

[akihikodaki]

I made pull request #2949, which also includes a change for Dockerfile.

[thurloat]

@Wonderfall thanks for that, a bit of an alpine noobie. I was under the impression on first glance that we were using edge packages by pushing the edge repo into /etc/apk/repositories.

I'll end up rolling my changes out and using @akihikodaki 's official patch anyhow.

Thanks for wrapping this up, everyone 👍