Gentoolto: sys-boot/grub doesn't recognize sys-fs/fuse when built with LTO under GCC 8.2

[gcs-github]

GCC 8.2.0
binutils 2.30-r3
sys-fs/fuse-2.9.7-r1
sys-boot/grub-2.02-r2

• sys-boot/grub depends on sys-fs/fuse when built with the mount USE flag.
• if fuse is built with LTO with GCC 8.1, grub builds successfully
• if fuse is built without LTO with GCC 8.2, grub builds successfully
• if fuse is built with LTO with GCC 8.2, grub's src_configure crashes with the following:

checking for fuse_main_real in -lfuse... no
configure: error: grub-mount was explicitly requested but can't be compiled (need FUSE library)

In that case, grub's config.log mentions the following:

configure:27875: x86_64-pc-linux-gnu-gcc -o conftest   -D_FILE_OFFSET_BITS=64  conftest.c -lfuse   >&5
/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../lib64/libfuse.so: undefined reference to `__fuse_read_cmd'

implying that libfuse.so might be linked incorrectly.

Enforcing *FLAGS-=-flto* on sys-fs/fuse solves the build issue.

[InBetweenNames]

Just curious, did you do a full system rebuild with 8.2.0?

[gcs-github]

Yes. Ran emerge -ve1 @system @world as soon as I switched compilers.

(had to skip grub and debug that one after everything else got built)

[InBetweenNames]

I actually have not been able to reproduce this. I have sys-boot/grub with USE=mount enabled with sys-fs/fuse and sys-boot/grub both enabled with LTO. Is the issue still affecting you? Also, what is your binutils version and linker?

[gcs-github]

Just reproduced it again with binutils 2.31.1. I haven't changed my linker from the default, but another difference between your setup and mine might be the use of the default/linux/amd64/17.0/no-multilib/hardened profile vs your desktop plasma one.

full emerge --info for fuse below (nothing in my gcsventures overlays is a dependency for either fuse or grub, not even indirectly – _unless we're looking at a kernel issue, but I certainly don't see how that would be the case_ –) :

Portage 2.3.46 (python 2.7.15-final-0, default/linux/amd64/17.0/no-multilib/hardened, gcc-8.2.0, glibc-2.27-r6, 4.14.62-gcsventures x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.14.62-gcsventures-x86_64-Intel-R-_Xeon-R-_CPU_E5-2660_v3_@_2.60GHz-with-gentoo-2.6
KiB Mem:   264080368 total, 253439136 free
KiB Swap:          0 total,         0 free
Head commit of repository lto-overlay: 07a8ccc336595c7a049d9a1956fcce3ab3727117

Head commit of repository mv: 75d039db75b9e9752073310938ec559c8fc06d3e

Timestamp of repository gentoo: Thu, 16 Aug 2018 17:45:01 +0000
Head commit of repository gentoo: 75b453d94460484673f8b2d81ded9e37b233890d
Head commit of repository gcsventures-overlay: d9763170b2c9a821a6df26dfaf38084e34234a60

Head commit of repository gcsventures-private-overlay: f769a56af79541be5b76d33a5c0ab6b0df987c8b

sh bash 4.4_p23
ld GNU ld (Gentoo 2.31.1 p1) 2.31.1
app-shells/bash:          4.4_p23::gentoo
dev-lang/perl:            5.26.2::gentoo
dev-lang/python:          2.7.15-r1::lto-overlay, 3.6.6-r1::lto-overlay
dev-util/cmake:           3.12.1::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.38.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.31.1::gentoo
sys-devel/gcc:            8.2.0-r2::gentoo
sys-devel/gcc-config:     1.9.1::gentoo
sys-devel/libtool:        2.4.6-r5::gentoo
sys-devel/make:           4.2.1-r3::gentoo
sys-kernel/linux-headers: 4.14::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

lto-overlay
    location: /usr/local/portage/lto-overlay
    sync-type: git
    sync-uri: https://github.com/InBetweenNames/gentooLTO.git
    masters: gentoo mv
    priority: 50

mv
    location: /usr/local/portage/mv
    sync-type: git
    sync-uri: https://github.com/vaeth/mv-overlay.git
    masters: gentoo
    priority: 60

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage/
    priority: 100
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24

gcsventures-overlay
    location: /usr/local/portage/gcsventures_overlay
    sync-type: git
    sync-uri: https://github.com/GCsVentures/gcsventures-overlay.git
    masters: gentoo
    priority: 200

gcsventures-private-overlay
    location: /usr/local/portage/gcsventures_private
    sync-type: git
    sync-uri: [email protected]:GCsVentures/gcsventures-private-overlay.git
    masters: gentoo
    priority: 202

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask=n --jobs=30 --load-average=31.0"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg cgroup collision-protect config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync metadata-transfer multilib-strict network-sandbox news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
MAKEOPTS="-j30"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 audit bash-completion bzip2 caps cjk crypt cxx device-mapper git hardened iconv inotify ipv6 jemalloc libtirpc lvm lzma ncurses netlink nls nptl offensive openmp pam pax_kernel pcre pie readline seccomp ssl ssp subversion unicode vim-syntax xattr xtpax zfs zlib zsh-completion" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi efi-64 pc qemu" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="arm i386 x86_64" QEMU_USER_TARGETS="arm i386 x86_64" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

sys-fs/fuse-2.9.7-r1::gentoo was built with the following:
USE="-examples -static-libs"
CFLAGS="-O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
CXXFLAGS="-O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize"

[gcs-github]

Looking at the emerge --info above, I'm seeing that my CXXFLAGS are defective. Will re- emerge -ve @system @world after fixing that, and I'll report back.

[gcs-github]

Still occurring after fixing my CXXFLAGS to be equal to my CFLAGS and rebuilding the entire system.

Updated my emerge --info comment above with the updated result

[InBetweenNames]

Thanks for the detailed post!

This line sticks out to me:
~
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -O3 -pipe -march=haswell -flto=8 -fuse-linker-plugin -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize"
~

I've actually never heard of -Wl,-O1 until now, and I just looked it up -- it appears to optimize the final shared object library size. I'm curious if you take this out of your LDFLAGS if it changes anything (rebuilding sys-fs/fuse and then sys-boot/grub in order).

Otherwise, your setup looks great! The next thing I'll try to do is replicate your exact flags here for those two packages and see if I can reproduce the error, if the above approach doesn't work.

[gcs-github]

Thanks!

I tried what you recommended, without success. I event tried removing everything from my LDFLAGS in a second run just to be sure.

The presence or absence of -flto=8 in my CFLAGS for the compilation of sys-fs/fuse-2.9.7-r1 seem to be making the difference alone for me.

My emerge -pv output for fuse and grub both, in case you want to try and match me USE flag for USE flag:

[ebuild   R    ] sys-fs/fuse-2.9.7-r1::gentoo  USE="-examples -static-libs" 0 KiB
[ebuild   R    ] sys-boot/grub-2.02-r2:2/2.02-r2::gentoo  USE="device-mapper fonts libzfs mount nls themes -debug -doc -efiemu -multislot -sdl -static (-test) -truetype" GRUB_PLATFORMS="efi-64 pc qemu -coreboot -efi-32 -emu -ieee1275 -loongson -multiboot -qemu-mips -uboot -xen -xen-32" 0 KiB

[gcs-github]

Looking further, I found this Gentoo bug reporting the same missing symbol with libfuse.so when built with LTO: https://bugs.gentoo.org/663518

[gcs-github]

And of course, same symbol in libfuse.so at stake in issue #142

[gcs-github]

I'm able to get over the problem in libfuse.so by switching the linker to ld.gold, however, grub with libzfs enabled doesn't seem to play well with ld.gold, while ld.bfd causes no problem in that case.

Trading one problem for another. :)

Are you using ld.gold @InBetweenNames ? If you do, that might explain why you couldn't reproduce this problem and others related to libfuse.

[gcs-github]

One thing with ld.gold however: nm reports that the __fuse_read_cmd symbol is still undefined in libfuse.so in that case as well. The difference is that grub's src_configure doesn't crash because of it anymore, for some reason.

[InBetweenNames]

I am indeed using ld.gold! Good catch. I just tested out sys-boot/grub with USE=libzfs mount and it seemed to emerge fine using ld.gold. This is very interesting. Checking the config.log:
~
configure:27850: checking for fuse_main_real in -lfuse
configure:27875: x86_64-pc-linux-gnu-gcc -o conftest -D_FILE_OFFSET_BITS=64 conftest.c -lfuse >&5
configure:27875: $? = 0
~

I'm glad this has been reported upstream too. I think the best option for fuse right now is to leave it as no-lto, and once those issues get sorted out we'll bring it back.

[InBetweenNames]

Interestingly, my system does have this:
~
root@mainline:/t/p/s/g/w/grub-2.02-efi-64➜ objdump -T /usr/lib64/libfuse.so | grep read_cmd
0000000000000000 D UND 0000000000000000 Base __fuse_read_cmd
000000000001dbe0 g DF .text 0000000000000122 FUSE_2.2 fuse_read_cmd
~

It seems there is an fuse_read_cmd function which is present as expected.

[gcs-github]

I think your system is also marking __fuse_read_cmd as undefined., hence the *UND* marking in place of the .text, as well as the absence of the F flag.

My output for the same (with LTO disabled for fuse):

h0001b ~ # objdump -T /usr/lib64/libfuse.so|grep fuse_read_cmd
000000000000ce40 g    DF .text  000000000000013f (Base)       __fuse_read_cmd
000000000000ce40 g    DF .text  000000000000013f  FUSE_2.2    fuse_read_cmd

Quite interesting that you're still managing to compile grub. :)

[InBetweenNames]

By all accounts, it shouldn't be working. The only thing I can think of is this:
https://github.com/osxfuse/fuse/blob/master/include/fuse.h#L1116

~~~

define __fuse_read_cmd fuse_read_cmd

~~~

It seems to be a compatibility thing. And it also seems that __fuse_read_cmd is supposed to be an unversioned symbol that maps directly to the real fuse_read_cmd. For some reason in the LTO build, that mapping is left undefined and we get a symbol resolution error.

Only other thing I can think of is that I'm using binutils 2.31.1.

[gcs-github]

We're both on binutils 2.31.1 now.

If we go back to my original report, this could very well be a regression in GCC. Compiling the same fuse ebuild with GCC 8.1.0 with LTO seemed to work.

[InBetweenNames]

Indeed. One more tidbit, found at:
https://github.com/libfuse/libfuse/blob/fuse_2_9_bugfix/lib/fuse.c#L4953

~
FUSE_SYMVER(".symver fuse_read_cmd,__fuse_read_cmd@");
~

Notably, this symbol is completely absent in fuse 3.x. Probably why it's masked.

I went ahead and followed the breadcrumbs:
https://github.com/libfuse/libfuse/commit/918f0ad95b73e506d20488cb8ddd35d1a2524c7c
https://github.com/libfuse/libfuse/commit/f458b8c07b521cfb48152e5d3344cd22989aa4ab
https://github.com/libfuse/libfuse/commit/0b6a0adc4d44e5b9d0855e257091eec643f3a185

These seem to indicate these are in for backwards compatibility. For some reason, under LTO, these are being omitted.

Most recent documentation on symver is here:
https://sourceware.org/binutils/docs-2.31/as/Symver.html#Symver

[InBetweenNames]

Well, I'm glad that's over with. Going to leave the issue open for hopefully more constructive discussion.