Envoy: segmentation fault from 1.13.0 binary built from centos docker image
I got segmentation fault with recent 1.13.0 binary which built with envoy-build-centos docker image.
In case of using envoy-build-ubuntu, it all works well.
Basically centos version also works,
but if I trying to use below flags, it happens
- --version
- --help
Used below command to build the binary
$ ENVOY_DOCKER_BUILD_DIR=/tmp/envoy.build \
IMAGE_NAME=envoyproxy/envoy-build-centos \
IMAGE_ID=latest \
./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only
based on this commit
$ git log -1
commit bb7ceff4c3c5bd4555dff28b6e56d27f2f8be0a7
Author: Matt Klein <[email protected]>
Date: Mon Jan 20 14:06:02 2020 -0800
docs: final docs for 1.13.0 (#9745)
Including reorganizing the configuration overview page.
Signed-off-by: Matt Klein <[email protected]>
message I got
build_release/envoy --version
Segmentation fault
from gdb
(gdb) r --version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x00005555561e24b2 in endl<char, std::__1::char_traits<char> > (__os=...) at /opt/llvm/bin/../include/c++/v1/ostream:1002
1002 /opt/llvm/bin/../include/c++/v1/ostream: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.el7_2.1.x86_64
(gdb) bt
#0 0x00005555561e24b2 in endl<char, std::__1::char_traits<char> > (__os=...) at /opt/llvm/bin/../include/c++/v1/ostream:1002
#1 operator<< (this=0x5555578bfb28 <std::__1::cout>, __pf=<optimized out>) at /opt/llvm/bin/../include/c++/v1/ostream:196
#2 TCLAP::StdOutput::version (this=<optimized out>, _cmd=...) at external/com_github_mirror_tclap/include/tclap/StdOutput.h:113
#3 0x00005555561e5585 in TCLAP::VersionVisitor::visit (this=<optimized out>) at external/com_github_mirror_tclap/include/tclap/VersionVisitor.h:73
#4 0x00005555561d3e43 in TCLAP::SwitchArg::processArg (this=0x5555580d23f0, i=0x7fffffffcd40, args=...) at external/com_github_mirror_tclap/include/tclap/SwitchArg.h:231
#5 0x00005555561df0d0 in TCLAP::CmdLine::parse (this=0x7fffffffcdc8, args=...) at external/com_github_mirror_tclap/include/tclap/CmdLine.h:460
#6 0x00005555561d8e31 in Envoy::OptionsImpl::OptionsImpl(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::function<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > (bool)> const&
, spdlog::level::level_enum) (this=0x555558112358, args=..., hot_restart_version_cb=..., default_log_level=spdlog::level::info) at source/server/options_impl.cc:143
#7 0x00005555561d5e10 in Envoy::OptionsImpl::OptionsImpl(int, char const* const*, std::__1::function<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > (bool)> const&, spdlog::level::level_enum) (this=0x555558112358, argc=<optimized out>, argv=<optimized out>, hot_restart_version_cb=..., default_log_level=spdlog::level::info)
at source/server/options_impl.cc:36
#8 0x00005555561be69a in Envoy::MainCommon::MainCommon (this=0x555558112000, argc=2, argv=0x7fffffffea18) at source/exe/main_common.cc:139
#9 0x00005555561bd143 in make_unique<Envoy::MainCommon, int&, char**&> (__args=<optimized out>, __args=<optimized out>) at /opt/llvm/bin/../include/c++/v1/memory:3132
#10 main (argc=2, argv=0x7fffffffea18) at source/exe/main.cc:26
my system info
$ uname -r
4.14.5
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
keyolk
👍2
All 12 comments
cc @lizan
mattklein123
on 23 Jan 2020
I don't have any idea why std::endl can cause SIGSEGV, it could be the toolchain issue though. What's the output of readelf -p .comment <path_to_envoy> of your envoy? @keyolk
lizan
on 26 Jan 2020
@keyolk it has to do with the STDLIB that it is linked against. Set the envvar ENVOY_STDLIB=libstdc++ in the build stage and it will start working again.
pablosole
on 31 Jan 2020
@lizan here the elf section
$ readelf -p .comment ./envoy
String dump of section '.comment':
[ 0] GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-36)
[ 2d] Linker: LLD 8.0.0
[ 3f] DynASM 1.4.0
[ 4c] clang version 8.0.0 (tags/RELEASE_800/final)
[ 79] GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)
@pablosole It works : ) Thanks alot!
the env should be set during centos build process
Now I built it with below
IMAGE_NAME=envoyproxy/envoy-build-centos IMAGE_ID=latest ENVOY_STDLIB=libsdtdc++ ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'
How about
set default ENVOY_STDLIB
from ci/run_envoy_docker.sh ?
keyolk
on 3 Feb 2020
@keyolk I think you're on older image with clang-8, the latest one should be with clang-9. Note with libstdc++ on CentOS 7 you're taking the risk described here too.
lizan
on 3 Feb 2020
👍1
@lizan Thats another point : 0
seems latest tagged image in docker hub not updated yet
$ docker run -ti --rm envoyproxy/envoy-build-centos:latest clang --version
clang version 8.0.0 (tags/RELEASE_800/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/bin
$ docker run -ti envoyproxy/envoy-build-ubuntu:latest 'apt list | grep clang'
/naver/docker/bin/docker: Error response from daemon: Container command 'apt list | grep clang' not found or does not exist..
css1668:~$ docker run -ti envoyproxy/envoy-build-ubuntu:latest bash -c 'apt list | gr
'
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
clang-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-format-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-tidy-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-tools-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
libclang-common-8-dev/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
libclang1-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
with recent updated tag
$ docker run -ti envoyproxy/envoy-build-centos:a0fbb2626d4658d142a197ec7acdc3b5886ebf78-amd64 bash -c 'clang --version' clang version 9.0.0 (tags/RELEASE_900/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/bin
$ docker run -ti envoyproxy/envoy-build-ubuntu:a0fbb2626d4658d142a197ec7acdc3b5886ebf78-amd64 bash -c 'apt list | grep clang'
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libclang1-3.8/now 1:3.8-2ubuntu4 amd64 [installed,local]
@keyolk can you try envoyproxy/envoy-build-centos:b6c2bae5716a289fd68c45a6e5bc16979dbe69c5? that's the build image tag we build 1.13.0.
lizan
on 3 Feb 2020
👍1
@lizan looks good to me : )
$ IMAGE_NAME=envoyproxy/envoy-build-centos IMAGE_ID=b6c2bae5716a289fd68c45a6e5bc16979dbe69c5 ENVOY_STDLIB=libstdc++ ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'
$ readelf -p .comment ./envoy
String dump of section '.comment':
[ 0] GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-39)
[ 2d] clang version 9.0.0 (tags/RELEASE_900/final)
[ 5a] DynASM 1.4.0
[ 67] Linker: LLD 9.0.0
[ 79] GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)
$ ./envoy --version
./envoy version: c0ac21349bd0ad37383a9124a2e16cf2a790c63b/1.13.0/Clean/RELEASE/BoringSSL
@lizan Just built envoy 1.13.0 using the image envoyproxy/envoy-build-centos:b6c2bae5716a289fd68c45a6e5bc16979dbe69c5 and default stdlib (ie. without setting ENVOY_STDLIB=libsdtdc++) and it dies the same way.
Tried it on multiple docker centos containers (centos:7 which is a 7.1, centos:centos7.5.1804, centos:centos8) they all crash in the same way.
Update: running the built envoy binary on the envoy-build-centos container that built it also crashes in the same way as the others.
Output of readelf:
String dump of section '.comment':
[ 0] GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-39)
[ 2d] clang version 9.0.0 (tags/RELEASE_900/final)
[ 5a] DynASM 1.4.0
[ 67] Linker: LLD 9.0.0
[ 79] GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)
I also did an strace -f but didn't seem very helpful. It dies pretty early during the stdlib boostrapping.
# strace -f ./envoy --help
execve("./envoy", ["./envoy", "--help"], [/* 9 vars */]) = 0
brk(NULL) = 0x557b647b6000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c809000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/tls/x86_64/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/tls/x86_64", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/tls/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/tls", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/x86_64/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/x86_64", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=11488, ...}) = 0
mmap(NULL, 11488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff14c806000
close(3) = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pS\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1139680, ...}) = 0
mmap(NULL, 3150136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14c2e7000
mprotect(0x7ff14c3e8000, 2093056, PROT_NONE) = 0
mmap(0x7ff14c5e7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7ff14c5e7000
close(3) = 0
open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\"\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=44448, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c805000
mmap(NULL, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14c0df000
mprotect(0x7ff14c0e6000, 2093056, PROT_NONE) = 0
mmap(0x7ff14c2e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7ff14c2e5000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19776, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14bedb000
mprotect(0x7ff14bedd000, 2097152, PROT_NONE) = 0
mmap(0x7ff14c0dd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7ff14c0dd000
close(3) = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=144792, ...}) = 0
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14bcbf000
mprotect(0x7ff14bcd6000, 2093056, PROT_NONE) = 0
mmap(0x7ff14bed5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7ff14bed5000
mmap(0x7ff14bed7000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff14bed7000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P%\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2173512, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c804000
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14b8f2000
mprotect(0x7ff14bab5000, 2093056, PROT_NONE) = 0
mmap(0x7ff14bcb4000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7ff14bcb4000
mmap(0x7ff14bcba000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff14bcba000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c803000
mmap(NULL, 90112, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7ed000
arch_prctl(ARCH_SET_FS, 0x7ff14c801cc0) = 0
mprotect(0x7ff14bcb4000, 16384, PROT_READ) = 0
mprotect(0x7ff14bed5000, 4096, PROT_READ) = 0
mprotect(0x7ff14c0dd000, 4096, PROT_READ) = 0
mprotect(0x7ff14c2e5000, 4096, PROT_READ) = 0
mprotect(0x7ff14c5e7000, 4096, PROT_READ) = 0
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffe081561f0) = -1 ENOENT (No such file or directory)
mprotect(0x557b635af000, 1789952, PROT_READ) = 0
mprotect(0x7ff14c80a000, 4096, PROT_READ) = 0
munmap(0x7ff14c806000, 11488) = 0
set_tid_address(0x7ff14c801f90) = 62
set_robust_list(0x7ff14c801fa0, 24) = 0
rt_sigaction(SIGRTMIN, {0x7ff14bcc57e0, [], SA_RESTORER|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7ff14bcc5870, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
gettid() = 62
brk(NULL) = 0x557b647b6000
futex(0x557b637c8e38, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(NULL) = 0x557b647b6000
brk(0x557b64fb6000) = 0x557b64fb6000
brk(NULL) = 0x557b64fb6000
brk(NULL) = 0x557b64fb6000
brk(0x557b650b6000) = 0x557b650b6000
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192) = 4
close(3) = 0
open("/sys/devices/system/cpu/cpu0/tsc_freq_khz", O_RDONLY) = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 284302874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 285719474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 285782174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 286987174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 287259574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 287372374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 288474374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 288644574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289622474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289718874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289819974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 290627774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 290790274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292389174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292447874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292479274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292588974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 293781674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 293999674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 294354874}) = 0
nanosleep({0, 1000000}, 0x7ffe081572d8) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 297765374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 297830774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 299215674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 300332474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 300381974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 301383174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 301476474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 304280974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 305287074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 306430674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 307769274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 308596274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 308626674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309370674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309407574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309433974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310691374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310727874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310754774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313827174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313912274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313948774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313983574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314018174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314052774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314205274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314943274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 315008174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316311274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316347974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316424574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316484274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316513474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316540674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316590574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316643174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316702174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 317924174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 317977474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 330363074}) = 0
nanosleep({0, 2000000}, 0x7ffe081572d8) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333236374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333269074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333324074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 334996274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 335491774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336669774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336741974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336784374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336838474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336894574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336938174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337028374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337182874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337286374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337334974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337377574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337499474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337554874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337610974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337692974}) = 0
rt_sigaction(SIGPROF, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPROF, {0x557b63504420, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
futex(0x557b637c4720, FUTEX_WAKE_PRIVATE, 2147483647) = 0
rt_sigprocmask(SIG_BLOCK, [PROF], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
futex(0x7ffe08157464, FUTEX_WAKE, 1) = 0
futex(0x7ffe08157464, FUTEX_WAKE_PRIVATE, 1) = 0
mmap(NULL, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7ff14c7e7000
mprotect(0x7ff14c7e7000, 4096, PROT_NONE) = 0
mprotect(0x7ff14c7ec000, 4096, PROT_NONE) = 0
sigaltstack({ss_sp=0x7ff14c7e8000, ss_flags=0, ss_size=16384}, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
gettid() = 62
rt_sigaction(SIGABRT, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGBUS, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGFPE, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGILL, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSEGV, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xffffffffffffffe8} ---
futex(0x7ff14bcbac40, FUTEX_WAKE_PRIVATE, 2147483647) = 0
gettid() = 62
gettid() = 62
gettid() = 62
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7d7000
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7c7000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7b7000
clock_gettime(CLOCK_REALTIME, {1580855616, 213477500}) = 0
gettid() = 62
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c808000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 118
lseek(3, -62, SEEK_CUR) = 56
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 62
close(3) = 0
munmap(0x7ff14c808000, 4096) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xffffffffffffffe8} ---
+++ killed by SIGSEGV +++
Segmentation fault
Another weirdness I can say is that if you run it without arguments it will just hang without showing anything on screen and stay there forever. From the strace it seems it is able to move on a bit forward in the code (probably because it didn't need to print anything on screen yet), clone two threads and then the main thread dies with the same SIGSEGV but the children stay alive stuck in some futex loop.
pablosole
on 4 Feb 2020
@lizan some more debugging about this. I wrote a hello world and compiled it inside the envoy bazel workspace with -s to get the building commands and during linking it does something like this:
-o
bazel-out/k8-opt/bin/tools/hello/world
-fuse-ld=/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/../../../../bin/ld.gold
-Wl,-no-as-needed
-Wl,-z,relro,-z,now
-B/opt/llvm/bin
-lm
-Wl,--gc-sections
bazel-out/k8-opt/bin/tools/hello/_objs/world/world.o
bazel-out/k8-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build/lib/libtcmalloc_and_profiler.a
-pthread
-lrt
-ldl
-Wl,--hash-style=gnu
-pie
-static-libgcc
-lpthread
-L/opt/llvm/lib
-Wl,-rpath,/opt/llvm/lib
-l:libc++.a
-l:libc++abi.a
-lm
There are 2 -fuse-ld (one using gold and one using ldd). If I comment the line on .bazelrc that forces clang to use ldd (build:clang --linkopt=-fuse-ld=lld) the binary works again.
It sounds the ld.gold is hardcoded somewhere else and it's breaking linking.
pablosole
on 5 Feb 2020
👍1
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 6 Mar 2020
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.
stale[bot]
on 13 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
justConfused
路
3Comments
ghost
路
3Comments
anatolebeuzon
路
3Comments
alkov-ibm
路
3Comments
yanniszark
路
3Comments