Mingw-packages: C++ hello world is causing a segmentation fault

[DavidEGrayson]

I just updated all MSYS2 packages on one of my machines, then tried to recompile some of my C++ programs, and noticed that several of them are giving a segmentation fault. I was able to simplify the issue a lot. Here is a shell session in a MinGW-w64 64-bit Shell showing how I can recreate the problem by trying to compile a simple C++ program. Can anyone else reproduce this?

$ cat prog.cpp
#include <iostream>

int main(int argc, char ** argv)
{
    std::cout << "Hello world." << std::endl;
    return 1;
}

$ g++ prog.cpp -o prog.exe

$ ./prog.exe
Segmentation fault

$ gdb ./prog.exe
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./prog.exe...done.
(gdb) run
Starting program: C:\Users\David\Documents\scraps\test_c++\prog.exe
[New Thread 116908.0x1c420]

Program received signal SIGSEGV, Segmentation fault.
0x000000006fcae829 in ?? () from D:\msys64\mingw64\bin\libstdc++-6.dll
(gdb) where
#0  0x000000006fcae829 in ?? () from D:\msys64\mingw64\bin\libstdc++-6.dll
#1  0x000000006fcfb2a9 in ?? () from D:\msys64\mingw64\bin\libstdc++-6.dll
#2  0x000000006fd00317 in ?? () from D:\msys64\mingw64\bin\libstdc++-6.dll
#3  0x00000000004015e7 in main ()
(gdb)

If I comment out the line that writes to std::cout, the problem goes away and the executable works.

Some of my C++ programs were able to write to their output without having a segmentation fault, so I might look into this further to see what the difference is between a working program and a failing program.

[DavidEGrayson]

I downgraded by toolchain from 5.3.0-2 to 5.3.0-1 by running the following command, but the problem is _still_ happening, so I'm not sure what the deal is:

pacman -U /var/cache/pacman/pkg/mingw-w64-x86_64-{gcc,gcc-libs,gcc-objc,gcc-ada,gcc-fortran,gcc-libgfortran}-5.3.0-1-any.pkg.tar.xz

FYI, this is not an urgent problem for me.

[vpetrigo]

:+1:
I have the same issue.
Here is the gdb output from the simple "Hello world" program:

#include <iostream>

int main() {
  std::cout << "Hello world" << std::endl;

  return 0;
}

Output

Reading symbols from hello.exe...done.                                
(gdb) step                                                            
The program is not being run.                                         
(gdb) start                                                           
Temporary breakpoint 1 at 0x4015cd                                    
Starting program: C:\Users\Vladimir\Documents\Projects\hello.exe      
[New Thread 10728.0x21cc]                                             
[New Thread 10728.0x1fd0]                                             
[New Thread 10728.0x119c]                                             
[New Thread 10728.0x2810]                                             

Temporary breakpoint 1, 0x00000000004015cd in main ()                 
(gdb) step                                                            
Single stepping until exit from function main,                        
which has no line number information.                                 

Program received signal SIGSEGV, Segmentation fault.                  
0x000000006fcae829 in ?? () from C:\msys64\mingw64\bin\libstdc++-6.dll

And if it might be helpful the strace output:

create_child: hello.exe                                                                                      
--- Process 7876 created                                                                                     
--- Process 7876 loaded C:\Windows\System32\ntdll.dll at 00007FFB1EF10000                                    
--- Process 7876 loaded C:\Windows\System32\kernel32.dll at 00007FFB1D3B0000                                 
--- Process 7876 loaded C:\Windows\System32\KernelBase.dll at 00007FFB1C120000                               
--- Process 7876 loaded C:\Windows\System32\msvcrt.dll at 00007FFB1D460000                                   
--- Process 7876 thread 3480 created                                                                         
--- Process 7876 thread 10572 created                                                                        
--- Process 7876 loaded C:\msys64\mingw64\bin\libstdc++-6.dll at 000000006FC40000                            
--- Process 7876 thread 7796 created                                                                         
--- Process 7876 loaded C:\Windows\System32\user32.dll at 00007FFB1D240000                                   
--- Process 7876 loaded C:\Windows\System32\gdi32.dll at 00007FFB1CEF0000                                    
--- Process 7876 loaded C:\msys64\mingw64\bin\libwinpthread-1.dll at 0000000064940000                        
--- Process 7876 loaded C:\msys64\mingw64\bin\libgcc_s_seh-1.dll at 0000000061440000                         
--- Process 7876 loaded C:\Windows\System32\imm32.dll at 00007FFB1C600000                                    
--- Process 7876 loaded C:\Windows\System32\nvinitx.dll at 00007FFB1A670000                                  
--- Process 7876 loaded C:\Windows\System32\advapi32.dll at 00007FFB1EBB0000                                 
--- Process 7876 loaded C:\Windows\System32\version.dll at 00007FFB1A610000                                  
--- Process 7876 loaded C:\Windows\System32\sechost.dll at 00007FFB1C540000                                  
--- Process 7876 loaded C:\Windows\System32\rpcrt4.dll at 00007FFB1C6B0000                                   
--- Process 7876 loaded C:\Program Files\NVIDIA Corporation\coprocmanager\detoured.dll at 0000000064FD0000   
--- Process 7876 loaded C:\Program Files\NVIDIA Corporation\coprocmanager\nvd3d9wrapx.dll at 00007FFB19D10000
--- Process 7876 loaded C:\Windows\System32\setupapi.dll at 00007FFB1C7D0000                                 
--- Process 7876 loaded C:\Windows\System32\cfgmgr32.dll at 00007FFB1BD30000                                 
--- Process 7876 loaded C:\Program Files\NVIDIA Corporation\coprocmanager\nvdxgiwrapx.dll at 00007FFB18BB0000
--- Process 7876 loaded C:\Program Files\ConEmu\ConEmu\ConEmuHk64.dll at 000000007E110000                    
--- Process 7876, exception c0000005 at 000000006FCAE829                                                     
--- Process 7876 thread 7796 exited with status 0xc0000005                                                   
--- Process 7876 thread 10572 exited with status 0xc0000005                                                  
--- Process 7876 thread 3480 exited with status 0xc0000005                                                   
--- Process 7876 exited with status 0xc0000005                                                               

[DavidEGrayson]

@vpetrigo I am glad that I am not the only one!

Here is my strace output:

create_child: C:\Users\David\Documents\scraps\test_c++\prog.exe
--- Process 1446472 created
--- Process 1446472 loaded C:\Windows\System32\ntdll.dll at 00007FF8BB320000
--- Process 1446472 loaded C:\Windows\System32\kernel32.dll at 00007FF8B8F10000
--- Process 1446472 loaded C:\Windows\System32\KernelBase.dll at 00007FF8B8540000
--- Process 1446472 loaded C:\Windows\System32\msvcrt.dll at 00007FF8BAC70000
--- Process 1446472 loaded D:\msys64\mingw64\bin\libstdc++-6.dll at 000000006FC40000
--- Process 1446472 loaded D:\msys64\mingw64\bin\libwinpthread-1.dll at 0000000064940000
--- Process 1446472 loaded C:\Windows\System32\user32.dll at 00007FF8B8A70000
--- Process 1446472 loaded D:\msys64\mingw64\bin\libgcc_s_seh-1.dll at 0000000061440000
--- Process 1446472 loaded C:\Windows\System32\gdi32.dll at 00007FF8BAD20000
--- Process 1446472 loaded C:\Windows\System32\imm32.dll at 00007FF8B8CB0000
--- Process 1446472 loaded C:\Windows\System32\msctf.dll at 00007FF8B9240000
--- Process 1446472, exception c0000005 at 000000006FCAE829
--- Process 1446472 exited with status 0xff

I have an NVIDIA graphics card, but I don't see any NVIDIA DLLs getting loaded in the strace like you do. It really seems like "hello world" should not load NVIDIA DLLs. :confused:

In case it matters, I ran my strace inside a normal MSYS2 shell, based on MinTTY. It looks like you are using ConEmu.

[Alexpux]

As I mentioned on IRC:try to downgrade mingw-w64 crt and headers packages to see if it solve problem

[DavidEGrayson]

It took three hours, but I managed to compile a debug version of mingw-w64-x86_64-gcc-libs using the PKGBUILD in this repository. (I just changed the options array to have !strip and debug and the build went flawlessly.) This allowed me to get a real stack trace from GDB:

(gdb) run
Starting program: C:\Users\David\Documents\scraps\test_c++\prog.exe
[New Thread 1447000.0x161460]

Program received signal SIGSEGV, Segmentation fault.
0x000000006fcab6c6 in std::ostream::sentry::sentry (this=this@entry=0x23fd90,
    __os=...)
    at D:/builds/mingw-packages/mingw-w64-gcc/src/build-x86_64-w64-mingw32/x86_64-w64-mingw32/libstdc++-v3/include/bits/ostream.tcc:51
51            if (__os.tie() && __os.good())
(gdb) where
#0  0x000000006fcab6c6 in std::ostream::sentry::sentry (
    this=this@entry=0x23fd90, __os=...)
    at D:/builds/mingw-packages/mingw-w64-gcc/src/build-x86_64-w64-mingw32/x86_64-w64-mingw32/libstdc++-v3/include/bits/ostream.tcc:51
#1  0x000000006fcfd81b in std::__ostream_insert<char, std::char_traits<char> >
    (__out=..., __s=0x404030 "Hello world.", __n=12)
    at D:/builds/mingw-packages/mingw-w64-gcc/src/build-x86_64-w64-mingw32/x86_64-w64-mingw32/libstdc++-v3/include/bits/ostream_insert.h:82
#2  0x000000006fd029d5 in std::operator<< <std::char_traits<char> > (
    __out=..., __s=<optimized out>)
    at D:/builds/mingw-packages/mingw-w64-gcc/src/build-x86_64-w64-mingw32/x86_64-w64-mingw32/libstdc++-v3/include/ostream:561
#3  0x00000000004015e7 in main ()

Here is some more output from GDB where I attempt to figure out what __os is. It looks like this object might be messed up.

(gdb) p __os
$1 = (std::basic_ostream<char, std::char_traits<char> > &) @0x6fd0d360: {<std::basic_ios<char, std::char_traits<char> >> = <invalid address>,
  _vptr.basic_ostream = 0x0}
(gdb) p __os.tie()
Cannot access memory at address 0xffffffffffffffe8

@AlexPux: Thanks! I will try that next.

[DavidEGrayson]

@AlexPux @vpetrigo Good news! I can confirm that downgrading to the previous version of mingw-w64-x86_64-headers-git and mingw-w64-x86_64-crt-git seems to solve the problem, at least for the hello world program.

I ran this command to downgrade:

pacman -U /var/cache/pacman/pkg/mingw-w64-x86_64-{headers,crt}-git-5.0.0.4609.566d621-1-any.pkg.tar.xz

For reference, the version that doesn't work is 4621.a7e64f9, which is only twelve commits ahead. It shouldn't be too hard for me to do a binary search and find the commit that causes the problem. (However, it's about time to go to sleep in this time zone.)

After downgrading, I had to recompile the hello world program to get it to work.

[Alexpux]

@DavidEGrayson, can you locally build crt/headers with reverted last commit (scarf memory leak) to see if it cause issue
https://github.com/Alexpux/mingw-w64/commit/a7e64f9d5aa44809fe8959d9bf1eb9d0393db307

[DavidEGrayson]

@AlexPux OK, I tried that, and verison 4620 doesn't seem to work. I built the packages by adding git checkout 801b6110b51a63b01a91cb07fd88d0d6f6032ef4 to the top of the pkgver functions.

Here is a shell session in case you want to double check something:

$ pacman -Q | egrep 'x86_64-(headers|crt)'
mingw-w64-x86_64-crt-git 5.0.0.4620.801b611-1
mingw-w64-x86_64-headers-git 5.0.0.4620.801b611-1
$ make
g++ prog.cpp -o prog.exe
$ ./prog.exe
Segmentation fault
$

So here is a summary of what we know so far. (I'll update this as I try more commits.)

• 4621.a7e64f9 - bad
• 4620.801b611 - bad
• 4617.e0531bb - good
• 4614.a0cd5af - good
• 4609.566d621 - good

[Alexpux]

I'm rebuilding GCC ... Will see how it is. There are some other commits that can break it. For example:
https://github.com/Alexpux/mingw-w64/commit/ca451a7a45d4876065edc6755f8aab8095914b04

[DavidEGrayson]

Version 4614.a0cd5af looks good to me (and I updated my summary above). So that means there are just 6 commits that could have caused the problem. Thanks for the help, @AlexPux.

[mmicko]

Can confirm that pacman -U http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-crt-git-5.0.0.4609.566d621-1-any.pkg.tar.xz is fixing issue. Thanks for looking at this @Alexpux

[elieux]

Confirmed that reverting https://github.com/Alexpux/mingw-w64/commit/ca451a7a45d4876065edc6755f8aab8095914b04 fixes the segfault.

[mkae]

Yesterday it was discussed on IRC, that the above referenced issue's finding could also be caused by this. I didn't experience any segfaults in said project, but who knows, perhaps it's related?!

Update: Turns out that this is not related.

[mingwandroid]

Pinging @martell and also adding a link to the mailing list post that introduced this change:

https://www.mail-archive.com/[email protected]/msg11917.html

[martell]

Hey guys,
yeah it's quite possible that patch breaks it with our current mingw-w64 gcc setup.
There is a header setting for when building gcc or rather libgcc to specify that ctors and dtors are provided by the c runtime. This is usually enabled for some embedded targets.

Alternatively I could only enable that when we build the crt with clang and not gcc. This would then mean that we can't use compiler-rt with mingw-w64 crt built with gcc and that we can't use libgcc with a clang built mingw-w64 crt.

I can probably live with that.

The correct fix would be to upstream a patch into libgcc to assume it is external from now on. Might be hard to get that merged because of legacy support though.

@mindwandroid thoughts?

[Alexpux]

I'm upload fixed mingw-w64 runtime.
Fixed by https://github.com/Alexpux/MINGW-packages/commit/f631991389bcbe62f2965b03469d9f05616e9ed9

[mmicko]

updated and confirmed working fine

On Sun, Feb 28, 2016 at 6:59 AM, Алексей [email protected] wrote:

I'm upload fixed mingw-w64 runtime.

—
Reply to this email directly or view it on GitHub
https://github.com/Alexpux/MINGW-packages/issues/1104#issuecomment-189788092
.

[DavidEGrayson]

Thanks @AlexPux!

[ian-abbott]

The binaries in the repository still seem to be broken for some reason:

$ pacman -Qo /mingw64/x86-644-w64-mingw32/lib/libmingw32.a
/mingw64/x86_64-w64-mingw32/lib/libmingw32.a is owned by mingw-w64-x86_64-crt-git 5.0.0.4624.4c13e3f-1
$ nm /mingw64/x86_64-w64-mingw32/lib/libmingw32.a | grep TOR_LIST
0000000000000000 d __MINGW_CTOR_LIST__
0000000000000000 d __MINGW_DTOR_LIST__

I.e., it hasn't reverted back to using __CTOR_LIST__ and __DTOR_LIST__. The same is true for the i686 package. It looks like whatever built the package didn't apply the reverse patch for some reason.

[Alexpux]

Yeah something went wrong. Will fix it shortly

[ian-abbott]

Seems to be fixed now. Thanks Alexey!

[katahiromz]

Please update crt of MSYS2-Packages.

[Alexpux]

already updated

[Kwizatz]

This exact issue is happening to me right now (Jan 2018) on 2 out of 3 computers, if I remove cout lines there is no segfault.
CRT is mingw-w64-x86_64-crt-git-6.0.0.5100.739199f8-1, I've checked gcc version and they match, libstdc++6 matches as well on all 3 computers, the only difference on the failing computer with respect to the one running fine is that the 2 failing have AMD CPUs (one is a Phenom, the other a Ryzen), the one running correctly is an Intel i7. Very odd.

[mati865]

@Kwizatz I've just checked on Ryzen and it works fine.
Is your environment fully upgraded (pacman -Syuu)?

[Kwizatz]

@mati865 Thanks! that seems to have done the trick, I did not know that existed.
Just in case what it did was downgrade binutils:

(1/1) downgrading mingw-w64-x86_64-binutils

I take it this is a known issue?

[mati865]

There was breakage recently https://github.com/Alexpux/MINGW-packages/issues/3330

[nanochap]

I see the same crash issue(a simple hello world console program get crashed) after I install the i686 GCC 7.3, and I just run the command pacman -Syuu, and I see it downgrade the GCC 6.4 with other packages. Now I don't see the crash issue now, Thanks.