Rust: Rust stack backtrace filled with unknown on Windows 8.1

[skiwi2]

When running a piece of code I get the following stack backtrace:

' panicked at 'index out of bounds', ../src/libcoreoption.rs:700
stack backtrace:
0: 0x4707c3 -
1: 0x46fcb5 -
2: 0x433baf -
3: 0x43573b -
4: 0x462eab -
5: 0x484e05 -
6: 0x4a61c4 -
7: 0x41ffdb -
8: 0x41ff4c -
9: 0x402bf5 -
10: 0x46f2ed -
11: 0x46f11a -
12: 0x42ba1a -
13: 0x4013b4 -
14: 0x4014e7 -
15: 0x7ffa1a5613d1 -
error: Process didn't exit successfully: C:\Users\Frank\Dropbox\workspace\sieve_multithreaded\target\debug\sieve_multithreaded.exe (exit code: 101)

I'm running cargo build with RUST_BACKTRACE=1 and have the following %PATH%:

C:Program FilesMicrosoft MPIBin;C:Program FilesHaskellbin;C:Program FilesHaskell Platform7.10.2-alibextralibsbin;C:Program FilesHaskell Platform
7.10.2-abin;C:ProgramDataOracleJavajavapath;C:jet9.0-eval-amd64bin;C:Program FilesImageMagick-6.8.7-Q16;C:Windowssystem32;C:Windows;C:WindowsSyste
m32Wbem;C:WindowsSystem32WindowsPowerShellv1.0;C:Program FilesMicrosoftWeb Platform Installer;C:Program Files (x86)Microsoft ASP.NETASP.NET Web Pag
esv1.0;C:Program FilesMicrosoft SQL Server110ToolsBinn;C:Program Files (x86)CMake 2.8bin;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;C:W
INDOWSSystem32WindowsPowerShellv1.0;C:Program Files (x86)Windows LiveShared;C:Program FilesJavajdk1.7.0_25bin;C:Program Files (x86)QuickTimeQTSys
tem;C:Program Files (x86)Tesseract-OCR;C:Program FilesCalibre2;C:Program Files (x86)Windows Kits8.1Windows Performance Toolkit;C:Program Files (x86)
Microsoft SDKsTypeScript1.0;C:Program Files (x86)Subversionbin;C:texlive2014binwin32;C:Program Filesnodejs;C:HashiCorpVagrantbin;C:Program Fil
es (x86)Gourcecmd;C:Program Files (x86)NVIDIA CorporationPhysXCommon;C:Program FilesHaskell Platform7.10.2-amingwbin;C:Leksahbin;C:Program Files
(x86)SkypePhone;C:Program FilesRust stable GNU 1.9bin;C:UsersFrankAppDataRoamingcabalbin;C:Program Files (x86)NVIDIA CorporationPhysXCommon;C:P
rogramDataOracleJavajavapath;C:jet9.0-eval-amd64bin;C:Program FilesImageMagick-6.8.7-Q16;C:Windowssystem32;C:Windows;C:WindowsSystem32Wbem;C:Windo
wsSystem32WindowsPowerShellv1.0;C:Program FilesMicrosoftWeb Platform Installer;C:Program Files (x86)Microsoft ASP.NETASP.NET Web Pagesv1.0;C:Progr
am FilesMicrosoft SQL Server110ToolsBinn;C:Program Files (x86)CMake 2.8bin;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;C:WINDOWSSystem32W
indowsPowerShellv1.0;C:Program Files (x86)Windows LiveShared;C:Program FilesJavajdk1.7.0_25bin;C:Program Files (x86)QuickTimeQTSystem;C:Program F
iles (x86)Tesseract-OCR;C:Program FilesCalibre2;C:Program Files (x86)Windows Kits8.1Windows Performance Toolkit;C:Program Files (x86)Microsoft SDKsT
ypeScript1.0;C:Program Files (x86)Subversionbin;C:Program Files (x86)SkypePhone;C:python27;C:python27Scripts;D:texlive2014binwin32;C:UsersFran
kAppDataRoamingnpm;C:Program FilesOracleVirtualBox;C:grails-3.0.9bin;C:Program Filesmingw-w64x86_64-4.9.0-posix-seh-rt_v3-rev2mingw64bin;C:Program
Files (x86)Gitbin;C:cntk

I have no clue why it is not showing more information about the stack frames and haven't had any success in Googling it.

[petrochenkov]

It's a very recent regression (couple of weeks maybe), I observe it too.

[alexcrichton]

@skiwi2 do you have some sample code that could be used to reproduce this?

[skiwi2]

I'm running the following code: https://gist.github.com/skiwi2/ce188c7d3b13fe3e3dbdb5c5f0752cd1 (Please don't mind the quality)

[alexcrichton]

I was unfortunately unable to reproduce, @skiwi2 can you also gist the output of rustc -vV as well as cargo -V?

[alexcrichton]

Ah and also, the panic message (e.g. the full output of one compile) would also be useful

[skiwi2]

@alexcrichton What exactly do you mean by your last request? Do you want the program output (including logging statements), or some compilation output?

[skiwi2]

The rustc and Cargo info files are up at https://gist.github.com/skiwi2/be1a8bf2171f3c1114a5fa65933e6732

[alexcrichton]

@skiwi2 oh just along the lines of gisting the entire error (from the invocation of rustc/cargo all the way through to the next prompt). It looks like the snippet above may just be a subset? The panic message may be enough, but can't hurt to have more output!

[retep998]

https://github.com/rust-lang/rust/pull/33554 was merged on May 14, your rustc is from May 18, could that have caused the regression perhaps?

[alexcrichton]

Oh! I apologize, I thought this was about an ICE not for a normal panic (which this appears to be about). In that case, yes, it's likely closely related to #33554, which was implemented for security purposes.

[retep998]

In the meantime if you do want backtraces and don't mind installing VC++ you can always use the x86_64-pc-windows-msvc version of Rust.

[DoumanAsh]

Having the same issue on current stable gnu toolchain

[abonander]

This is still occurring on recent nightlies: https://www.reddit.com/r/rust/comments/4zgzfk/is_there_an_unwrap_in_macro_form/d6vrvuw

[ruabmbua]

I have the same problem too on newest gnu nightly and stable.
Could someone look into fixing that? It is quiet annoying, when I have to use gdb for my cargo test cases, to actually find the panicing code.

Edit:
msvc is not an option for me

[petrochenkov]

This is a stable -> stable regression, by the way, if it helps with prioritization.

[MoreAxes]

This issue affects me as well. It makes debugging really hard. Luckily I can switch to msvc for now, but I have to abandon most of my tooling (e.g. gdb, which AFAIK doesn't support msvc-built binaries; I also often link to mingw-built libraries).

[petrochenkov]

I've confirmed, https://github.com/rust-lang/rust/pull/33554 is indeed the PR that killed backtraces (cc @sfackler). I'll try to investigate further.

[petrochenkov]

@MoreAxes
This issue doesn't affect debugging with gdb, only backtraces produced by rustc.
So, you still can run the executable with gdb, break on rust_panic and obtain the backtrace with bt.

[petrochenkov]

The simplest solution is to revert https://github.com/rust-lang/rust/pull/33554, but use env::current_exe (aka GetModuleFileNameW) only on Windows.
Having backtraces seems more useful than guarding from unconfirmed vulnerability.

[petrochenkov]

One of concerns from https://github.com/rust-lang/rust/issues/21889 (in addition to malicious executable replacement) were race conditions in env::current_exe. GetModuleFileNameW seems to not be racy according to various people on the internet.

[sfackler]

If the windows implementation doesn't have those race issues then it definitely seems reasonable to just turn this back on on Windows.

[retep998]

@petrochenkov The point is not whether env::current_exe is safe (which it definitely is on Windows), but rather the fact that the file that env::current_exe points to may not be the original executable, but some other file (yes, this _can_ happen on Windows). This other file can be carefully crafted to exploit certain bugs in libbacktrace potentially allowing an exploit. Granted, this exploit requires the attacker to have a rather extreme amount of power in the first place to swap files like that, but that's the reasoning used for that PR.

[arielb1]

This vulnerability is hard to reach, but I don't want to be introducing random vulnerabilities to all Rust programs.

[petrochenkov]

You assume feeding an arbitrary ill-formed PE/COFF to libbacktrace is indeed a vulnerability, I haven't seen this confirmed so far. By default it sounds like "feeding an arbitrary text to rustc is vulnerability".
Anyway, I'll try to investigate why exactly libbacktrace needs these paths to produce backtraces and maybe find another solution before submitting a PR reverting https://github.com/rust-lang/rust/pull/33554.

[petrochenkov]

Update:

Prelude

DWARF debug info is used on Windows/GNU, as opposed to PDB debug info on Windows/MSVC.
WinAPI has tools for reading PDB, but can't read DWARF. libbacktrace can read DWARF, that's why libbacktrace is used on Windows/GNU but not on Windows/MSVC.
Debug info enriches backtraces with file names, line numbers, inlined functions etc in addition to the minimal backtrace - non-inlined function names and addresses, that can be obtained with WinAPI alone without using libbacktrace.

The problem

libbacktrace needs to the read the executable file to extract DWARF debug info from it.
If filename is not provided to libbacktrace in backtrace_create_state (this is what https://github.com/rust-lang/rust/pull/33554 did), then libbacktrace will try to retrieve it itself using getexecname, /proc/self/exe or /proc/curproc/file, whatever works first. Neither works on Windows/GNU, therefore backtraces are broken.

In any case, the executable file is read and interpreted by libbacktrace whether we provide the filename to backtrace_create_state or not, unless some error happens. There's still a window between the file name is retrieved and the file is read, so the hypothetical vulnerability discussed above is inherent to libbacktrace, but the attack window is much smaller because libbactrace falls back to getexecname etc almost immediately before reading the file and don't cache the obtained filename for later use.

So, why don't we patch libbacktrace like this commit does and use GetModuleFileName (instead of /proc/self/exe etc) to get the fallback name on Windows? This fixes backtraces for sure.
The answer is that GetModuleFileName returns the original executable name with which it was launched. If executable is renamed and replaced with a malicious file, then GetModuleFileName will refer to the malicious file even if it's called after renaming.

Possible solutions

• Try to use QueryFullProcessImageName instead of GetModuleFileName, it should support renaming (this is what https://github.com/rust-lang/rust/pull/37359 does).
• Use minimal backtraces on Windows/GNU without additional goodies like line numbers, they doesn't require libbacktrace at all.
• Use GetModuleFileName and enjoy the hypothetical vulnerability (i.e. revert #33554 for Windows).
• Rewrite libbacktrace in Rust and harden it against ill-formed malicious DWARVes.

[retep998]

Use minimal backtraces on Windows/GNU without additional goodies like line numbers, they doen't require libbacktrace at all.

Note that the names of statically linked functions are also only specified in the debuginfo. Without debuginfo all you get is the names of functions that are exported from a DLL, and because Rust statically links everything by default, this means all Rust functions end up as unnamed, not even a mangled name.

[petrochenkov]

Regarding @alexcrichton 's alternatives from https://github.com/rust-lang/rust/pull/37359#issuecomment-256182219

Figure out another method of giving libbacktrace dwarf data

Unless there's a way to somehow automatically read DWARF into a process' memory with OS' help, I don't see a way to avoid reading it from a file.

Figure out another method of backtraces on MinGW

This means either even more invasive changes in libbacktraces to make it secure enough (how to determine if it's secure enough?), or find a readily available more secure version of libbacktrace (any concrete variants? how to determine if the alternative is secure enough?), or rewrite libbacktrace by ourselves (how to determine if our rewrite is secure enough?). Why do we think libbacktrace is not secure enough? I haven't seen any proofs that unintentional code-execution is possible yet.
Meanwhile, a tier1 platform doesn't have backtraces for almost half a year.

[petrochenkov]

Fixed by https://github.com/rust-lang/rust/pull/39234

[dpelevin]

@petrochenkov, with current implementation application crashes in the case of panic, if debug info is stripped out with (strip -s helloworld.exe) and RUST_BACKTRACE is set to 1.

[petrochenkov]

@dpelevin
It's better to create a new issue for this problem, so it doesn't get lost.