Runtime: [R2R] corssgen2 with singlefilehost runs into SIGABRT

[am11]

Description

On Windows x64 or Linux x64 system, execute the following command with .NET 5 executable in PATH:

# Bash on Linux and PowerShell Windows
# for windows, replace linux-x64 with win-x64 in the steps below

# create a console application
$ dotnet new console -n r2rtest; cd r2rtest

# publish it as a .NET 5 single file and R2R deployment with crossgen2
$ dotnet publish -r linux-x64 -p:'PublishReadyToRun=true,PublishReadyToRunUseCrossgen2=true,PublishSingleFile=true,IncludeAllContentForSelfExtract=false'

# run
$  bin/Debug/net5.0/linux-x64/publish/r2rtest

Expected

Prints Hello World!

Actual

• Linux: Aborted (core dumped) with exit code 134 (SIGSEGVSIGABRT)
  
  
  
  • strace output https://gist.github.com/am11/cf42aa17b7b10a3a115ada3c110fbf5c (lldb with SOS is unable to debug .NET5 singlefile in general: https://github.com/dotnet/diagnostics/issues/1764)
  
  
• Windows: exit code -1073740286
  
  
  
  • windbg logs https://gist.github.com/am11/818f0e70895f3ae7419ab92e492aacd0
  
  

Workaround

Change IncludeAllContentForSelfExtract=false part as IncludeAllContentForSelfExtract=true (uses 3.x bundle), rebuild and execute; prints Hello World! as expected.

Version info

# on both, Windows and Linux
$ dotnet --version
5.0.100

cc @AntonLapounov

[msftbot[bot]]

Tagging subscribers to this area: @agocke, @vitek-karas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

On Windows x64 or Linux x64 system, execute the following command with .NET 5 executable in PATH:

# Bash on Linux and PowerShell Windows
# for windows, replace linux-x64 with win-x64 in the steps below

# create a console application
$ dotnet new console -n r2rtest; cd r2rtest

# publish it as a .NET 5 single file and R2R deployment with crossgen2
$ dotnet publish -r linux-x64 -p:'PublishReadyToRun=true,PublishReadyToRunUseCrossgen2=true,PublishSingleFile=true,IncludeAllContentForSelfExtract=false'

# run
$  bin/Debug/net5.0/linux-x64/publish/r2rtest

Expected

Prints Hello World!

Actual

• Linux: Aborted (core dumped) with exit code 134 (SIGSEGV)
  
  
  
  • strace output https://gist.github.com/am11/cf42aa17b7b10a3a115ada3c110fbf5c (lldb with SOS is unable to debug crossgen2'd singlefile https://github.com/dotnet/diagnostics/issues/1764)
  
  
• Windows: exit code -1073740286
  
  
  
  • windbg logs https://gist.github.com/am11/818f0e70895f3ae7419ab92e492aacd0
  
  

Workaround

Change IncludeAllContentForSelfExtract=false part as IncludeAllContentForSelfExtract=true (uses 3.x bundle), rebuild and execute; prints Hello World! as expected.

Version info

# on both, Windows and Linux
$ dotnet --version
5.0.100

cc @AntonLapounov

Author:	am11
Assignees:	-
Labels:	`area-Single-File`, `untriaged`
Milestone:	-

[vitek-karas]

Somebody with understanding of R2R & crossgen2 should take a look and at least roughly determine what's failing.

[janvorli]

• Linux: Aborted (core dumped) with exit code 134 (SIGSEGV)

Exit code 134 is not a SIGSEGV, it is SIGABRT, which means that runtime called abort. It most likely means an unhandled exception or runtime failfast.

[am11]

Ah true. I was running into SIGSEGV with a diagnostic tool. LLDB was unable to load symbols from singlefile, although i have copied libcoreclr.so libcoreclrtraceptprovider.so libmscordaccore.so libmscordbi.so to publish directory as per the singlefile debugging docs. Here is how the backtrace looks like:

Process 15585 launched: '/home/vagrant/r2rtest/bin/Debug/net5.0/linux-x64/publish/r2rtest' (x86_64)
Process 15585 stopped
* thread #1, name = 'r2rtest', stop reason = signal SIGABRT
    frame #0: 0x00007ffff6650f47 libc.so.6`__GI_raise(sig=2) at raise.c:51
(lldb) bt
* thread #1, name = 'r2rtest', stop reason = signal SIGABRT
  * frame #0: 0x00007ffff6650f47 libc.so.6`__GI_raise(sig=2) at raise.c:51
    frame #1: 0x00007ffff66528b1 libc.so.6`__GI_abort at abort.c:79
    frame #2: 0x0000555555c12a3e r2rtest`___lldb_unnamed_symbol18222$$r2rtest + 30
    frame #3: 0x0000555555c129ae r2rtest`RaiseFailFastException + 14
    frame #4: 0x0000555555912215 r2rtest`___lldb_unnamed_symbol10217$$r2rtest + 1077
    frame #5: 0x00005555558ab411 r2rtest`___lldb_unnamed_symbol8880$$r2rtest + 65
    frame #6: 0x00005555556ec4e9 r2rtest`___lldb_unnamed_symbol2965$$r2rtest + 969
    frame #7: 0x00005555558b420e r2rtest`___lldb_unnamed_symbol8998$$r2rtest + 494
    frame #8: 0x00005555558b3224 r2rtest`___lldb_unnamed_symbol8984$$r2rtest + 468
    frame #9: 0x00005555558b4ed0 r2rtest`___lldb_unnamed_symbol9008$$r2rtest + 192
    frame #10: 0x00005555558a56a6 r2rtest`___lldb_unnamed_symbol8815$$r2rtest + 134
    frame #11: 0x00005555558a63ac r2rtest`___lldb_unnamed_symbol8827$$r2rtest + 556
    frame #12: 0x00005555558f6a13 r2rtest`___lldb_unnamed_symbol9829$$r2rtest + 131
    frame #13: 0x00005555558f5eca r2rtest`___lldb_unnamed_symbol9812$$r2rtest + 90
    frame #14: 0x000055555589a5cc r2rtest`___lldb_unnamed_symbol8712$$r2rtest + 236
    frame #15: 0x00005555558999c2 r2rtest`___lldb_unnamed_symbol8706$$r2rtest + 386
    frame #16: 0x0000555555899fdc r2rtest`___lldb_unnamed_symbol8708$$r2rtest + 1084
    frame #17: 0x000055555589623e r2rtest`___lldb_unnamed_symbol8654$$r2rtest + 46
    frame #18: 0x000055555589603d r2rtest`___lldb_unnamed_symbol8653$$r2rtest + 557
    frame #19: 0x00005555558916c0 r2rtest`___lldb_unnamed_symbol8585$$r2rtest + 1712
    frame #20: 0x0000555555890b09 r2rtest`___lldb_unnamed_symbol8578$$r2rtest + 361
    frame #21: 0x000055555589096d r2rtest`___lldb_unnamed_symbol8577$$r2rtest + 301
    frame #22: 0x000055555564a57e r2rtest`___lldb_unnamed_symbol626$$r2rtest + 110
    frame #23: 0x0000555555649b83 r2rtest`coreclr_initialize + 643
    frame #24: 0x000055555561e9a0 r2rtest`___lldb_unnamed_symbol360$$r2rtest + 800
    frame #25: 0x000055555562da35 r2rtest`___lldb_unnamed_symbol428$$r2rtest + 309
    frame #26: 0x000055555562d45b r2rtest`corehost_main + 155
    frame #27: 0x00005555556070a2 r2rtest`___lldb_unnamed_symbol234$$r2rtest + 1746
    frame #28: 0x00005555556057fb r2rtest`___lldb_unnamed_symbol232$$r2rtest + 667
    frame #29: 0x0000555555601d62 r2rtest`hostfxr_main_bundle_startupinfo + 210
    frame #30: 0x00005555555fce48 r2rtest`___lldb_unnamed_symbol200$$r2rtest + 760
    frame #31: 0x00005555555fd420 r2rtest`___lldb_unnamed_symbol201$$r2rtest + 144
    frame #32: 0x00007ffff6633b97 libc.so.6`__libc_start_main(main=(r2rtest`___lldb_unnamed_symbol201$$r2rtest), argc=1, argv=0x00007fffffffe488, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe478) at libc-start.c:310
    frame #33: 0x00005555555f0aaa r2rtest`_start + 41

and clrstack fails to load libcoreclr.so:

(lldb) clrstack
Failed to find runtime module (libcoreclr.so), 0x80004005
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
ClrStack  failed

[janvorli]

This is where the fail fast comes from. I've compiled it on Windows and replaced coreclr.dll and clrjit.dll with release build binaries built locally from the same commit as the ones I was replacing:
https://github.com/dotnet/runtime/blob/b79e4d3cd24f76dd1bcb722f589327c827e66632/src/coreclr/src/vm/nativeimage.cpp#L190-L195

Here is the call stack at the point of failure:

 # Child-SP          RetAddr           Call Site
00 00000054`2477d7d0 00007ffa`9bd0121a KERNELBASE!RaiseFailFastException+0x15c [minkernel\kernelbase\xcpt.c @ 1199] 
01 00000054`2477ddb0 00007ffa`9bcc4cc6 coreclr!NativeImage::Open+0x382 [F:\git\runtime6\src\coreclr\src\vm\nativeimage.cpp @ 198] 
02 00000054`2477e1f0 00007ffa`9bc55dfd coreclr!AssemblyLoadContext::LoadNativeImage+0x62 [F:\git\runtime6\src\coreclr\src\vm\assemblyloadcontext.cpp @ 27] 
03 (Inline Function) --------`-------- coreclr!AcquireCompositeImage+0x71 [F:\git\runtime6\src\coreclr\src\vm\readytoruninfo.cpp @ 506] 
04 00000054`2477e230 00007ffa`9bae104e coreclr!ReadyToRunInfo::Initialize+0x17482d [F:\git\runtime6\src\coreclr\src\vm\readytoruninfo.cpp @ 592] 
05 00000054`2477e2d0 00007ffa`9bb50316 coreclr!Module::Initialize+0xee [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 596] 
06 00000054`2477e300 00007ffa`9bb4fe1f coreclr!Module::DoInit+0xa2 [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 175] 
07 00000054`2477e3f0 00007ffa`9bb500c8 coreclr!Module::Create+0x97 [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 1164] 
08 00000054`2477e480 00007ffa`9bb4ffcd coreclr!Assembly::Init+0xb8 [F:\git\runtime6\src\coreclr\src\vm\assembly.cpp @ 180] 
09 00000054`2477e4e0 00007ffa`9bb506c3 coreclr!Assembly::Create+0x119 [F:\git\runtime6\src\coreclr\src\vm\assembly.cpp @ 384] 
0a 00000054`2477e5f0 00007ffa`9bb53ecd coreclr!DomainAssembly::Allocate+0x93 [F:\git\runtime6\src\coreclr\src\vm\domainfile.cpp @ 1438] 
0b 00000054`2477e850 00007ffa`9bb53cdc coreclr!DomainFile::DoIncrementalLoad+0x91 [F:\git\runtime6\src\coreclr\src\vm\domainfile.cpp @ 536] 
0c 00000054`2477e880 00007ffa`9bb53b89 coreclr!AppDomain::TryIncrementalLoad+0xa8 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3306] 
0d 00000054`2477e9a0 00007ffa`9bb03e5c coreclr!AppDomain::LoadDomainFile+0x115 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3240] 
0e 00000054`2477ea50 00007ffa`9bb03c05 coreclr!AppDomain::LoadDomainAssemblyInternal+0x248 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3105] 
0f 00000054`2477eb60 00007ffa`9bad2251 coreclr!AppDomain::LoadDomainAssembly+0xc1 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 2967] 
10 00000054`2477ee50 00007ffa`9bb3a025 coreclr!SystemDomain::LoadBaseSystemClasses+0x55 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 1425] 
11 00000054`2477eec0 00007ffa`9bba2361 coreclr!SystemDomain::Init+0x159 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 1299] 
12 00000054`2477ef30 00007ffa`9bc045db coreclr!EEStartupHelper+0x1325 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 983] 
13 00000054`2477f2c0 00007ffa`9bb59e22 coreclr!EEStartup+0x27 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 1140] 
14 00000054`2477f310 00007ffa`9bbfa678 coreclr!EnsureEEStarted+0x92 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 325] 
15 00000054`2477f360 00007ffa`9bb7a5d0 coreclr!CorHost2::Start+0x58 [F:\git\runtime6\src\coreclr\src\vm\corhost.cpp @ 102] 
16 00000054`2477f3a0 00007ff7`962645a5 coreclr!coreclr_initialize+0x140 [F:\git\runtime6\src\coreclr\src\dlls\mscoree\unixinterface.cpp @ 236] 
17 00000054`2477f470 00007ff7`9626d0fd singlefilehost!coreclr_t::create+0x189 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\coreclr.cpp @ 58] 
18 00000054`2477f5e0 00007ff7`9626eb39 singlefilehost!`anonymous namespace'::create_coreclr+0x14d [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 81] 
19 00000054`2477f640 00007ff7`96254d49 singlefilehost!corehost_main+0xe9 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 404] 
1a 00000054`2477f800 00007ff7`9625871c singlefilehost!execute_app+0x239 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 146] 
1b (Inline Function) --------`-------- singlefilehost!?A0x83a23e19::read_config_and_execute+0x10a [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 520] 
1c 00000054`2477f8f0 00007ff7`96256eab singlefilehost!fx_muxer_t::handle_exec_host_command+0x214 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 1001] 
1d 00000054`2477f9e0 00007ff7`96253285 singlefilehost!fx_muxer_t::execute+0x39b [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 566] 
1e 00000054`2477fb20 00007ff7`9624faac singlefilehost!hostfxr_main_bundle_startupinfo+0x155 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\hostfxr.cpp @ 40] 
1f 00000054`2477fc40 00007ff7`9624fdf4 singlefilehost!exe_start+0x658 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 207] 
20 00000054`2477fe20 00007ff7`962824e8 singlefilehost!wmain+0xa4 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 302] 
21 (Inline Function) --------`-------- singlefilehost!invoke_main+0x22 [D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
22 00000054`2477fe50 00007ffb`2b897034 singlefilehost!__scrt_common_main_seh+0x10c [D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
23 00000054`2477fe90 00007ffb`2c6dcec1 KERNEL32!BaseThreadInitThunk+0x14 [clientcore\base\win32\client\thread.c @ 64] 
24 00000054`2477fec0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [minkernel\ntdll\rtlstrt.c @ 1153] 

[trylek]

Thanks Jan for figuring this out. @vitek-karas I guess there are likely two problems with Crossgen2 composite + single file:

1) Crossgen2 compilation emits the extra composite native image with no source; I'm not super-familiar with the single file publishing but the build logic needs to make sure to embed the composite image into the single file in addition to the compiled input files;

2) The existing composite implementation doesn't account for single file in the sense that it uses path manipulation and always tries to open the composite image from disk.

For (1), double-checking and potentially fixing this would be easiest for someone from the appmodel team. For (2), I can easily make the change if you or someone from the appmodel teams explains to me how to modify

https://github.com/dotnet/runtime/blob/b79e4d3cd24f76dd1bcb722f589327c827e66632/src/coreclr/src/vm/nativeimage.cpp#L112

to properly cater for loading from memory.

[janvorli]

This is where we actually fail - the first location where we throw an exception that later leads to the fail fast. The NativeImageLayout constructor gets "r2rtest.r2r.dll" as the fullPath argument.

 # Child-SP          RetAddr           Call Site
00 000000e5`f8f7d7f0 00007ffa`9cd00080 KERNELBASE!RaiseException+0x69 [minkernel\kernelbase\xcpt.c @ 937] 
01 000000e5`f8f7d8d0 00007ffa`9cf1d5cb coreclr!_CxxThrowException+0x90 [d:\agent\_work\59\s\src\vctools\crt\vcruntime\src\eh\throw.cpp @ 75] 
02 000000e5`f8f7d930 00007ffa`9cf1d914 coreclr!ThrowHR+0x13f [F:\git\runtime6\src\coreclr\src\utilcode\ex.cpp @ 965] 
03 000000e5`f8f7d980 00007ffa`9cf1d8c1 coreclr!ThrowWin32+0x24 [F:\git\runtime6\src\coreclr\src\utilcode\ex.cpp @ 1016] 
04 000000e5`f8f7d9b0 00007ffa`9cdf4826 coreclr!ThrowLastError+0x11 [F:\git\runtime6\src\coreclr\src\utilcode\ex.cpp @ 1025] 
05 000000e5`f8f7d9e0 00007ffa`9cdf590e coreclr!NativeImageLayout::NativeImageLayout+0x9e [F:\git\runtime6\src\coreclr\src\vm\peimagelayout.cpp @ 763] 
06 000000e5`f8f7da20 00007ffa`9cdf1183 coreclr!PEImageLayout::LoadNative+0x7a [F:\git\runtime6\src\coreclr\src\vm\peimagelayout.cpp @ 81] 
07 000000e5`f8f7da50 00007ffa`9cdb4cc6 coreclr!NativeImage::Open+0x2eb [F:\git\runtime6\src\coreclr\src\vm\nativeimage.cpp @ 145] 
08 000000e5`f8f7de90 00007ffa`9cd45dfd coreclr!AssemblyLoadContext::LoadNativeImage+0x62 [F:\git\runtime6\src\coreclr\src\vm\assemblyloadcontext.cpp @ 27] 
09 (Inline Function) --------`-------- coreclr!AcquireCompositeImage+0x71 [F:\git\runtime6\src\coreclr\src\vm\readytoruninfo.cpp @ 506] 
0a 000000e5`f8f7ded0 00007ffa`9cbd104e coreclr!ReadyToRunInfo::Initialize+0x17482d [F:\git\runtime6\src\coreclr\src\vm\readytoruninfo.cpp @ 592] 
0b 000000e5`f8f7df70 00007ffa`9cc40316 coreclr!Module::Initialize+0xee [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 596] 
0c 000000e5`f8f7dfa0 00007ffa`9cc3fe1f coreclr!Module::DoInit+0xa2 [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 175] 
0d 000000e5`f8f7e090 00007ffa`9cc400c8 coreclr!Module::Create+0x97 [F:\git\runtime6\src\coreclr\src\vm\ceeload.cpp @ 1164] 
0e 000000e5`f8f7e120 00007ffa`9cc3ffcd coreclr!Assembly::Init+0xb8 [F:\git\runtime6\src\coreclr\src\vm\assembly.cpp @ 180] 
0f 000000e5`f8f7e180 00007ffa`9cc406c3 coreclr!Assembly::Create+0x119 [F:\git\runtime6\src\coreclr\src\vm\assembly.cpp @ 384] 
10 000000e5`f8f7e290 00007ffa`9cc43ecd coreclr!DomainAssembly::Allocate+0x93 [F:\git\runtime6\src\coreclr\src\vm\domainfile.cpp @ 1438] 
11 000000e5`f8f7e4f0 00007ffa`9cc43cdc coreclr!DomainFile::DoIncrementalLoad+0x91 [F:\git\runtime6\src\coreclr\src\vm\domainfile.cpp @ 536] 
12 000000e5`f8f7e520 00007ffa`9cc43b89 coreclr!AppDomain::TryIncrementalLoad+0xa8 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3306] 
13 000000e5`f8f7e640 00007ffa`9cbf3e5c coreclr!AppDomain::LoadDomainFile+0x115 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3240] 
14 000000e5`f8f7e6f0 00007ffa`9cbf3c05 coreclr!AppDomain::LoadDomainAssemblyInternal+0x248 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 3105] 
15 000000e5`f8f7e800 00007ffa`9cbc2251 coreclr!AppDomain::LoadDomainAssembly+0xc1 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 2967] 
16 000000e5`f8f7eaf0 00007ffa`9cc2a025 coreclr!SystemDomain::LoadBaseSystemClasses+0x55 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 1425] 
17 000000e5`f8f7eb60 00007ffa`9cc92361 coreclr!SystemDomain::Init+0x159 [F:\git\runtime6\src\coreclr\src\vm\appdomain.cpp @ 1299] 
18 000000e5`f8f7ebd0 00007ffa`9ccf45db coreclr!EEStartupHelper+0x1325 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 983] 
19 000000e5`f8f7ef60 00007ffa`9cc49e22 coreclr!EEStartup+0x27 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 1140] 
1a 000000e5`f8f7efb0 00007ffa`9ccea678 coreclr!EnsureEEStarted+0x92 [F:\git\runtime6\src\coreclr\src\vm\ceemain.cpp @ 325] 
1b 000000e5`f8f7f000 00007ffa`9cc6a5d0 coreclr!CorHost2::Start+0x58 [F:\git\runtime6\src\coreclr\src\vm\corhost.cpp @ 102] 
1c 000000e5`f8f7f040 00007ff7`962645a5 coreclr!coreclr_initialize+0x140 [F:\git\runtime6\src\coreclr\src\dlls\mscoree\unixinterface.cpp @ 236] 
1d 000000e5`f8f7f110 00007ff7`9626d0fd singlefilehost!coreclr_t::create+0x189 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\coreclr.cpp @ 58] 
1e 000000e5`f8f7f280 00007ff7`9626eb39 singlefilehost!`anonymous namespace'::create_coreclr+0x14d [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 81] 
1f 000000e5`f8f7f2e0 00007ff7`96254d49 singlefilehost!corehost_main+0xe9 [F:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 404] 
20 000000e5`f8f7f4a0 00007ff7`9625871c singlefilehost!execute_app+0x239 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 146] 
21 (Inline Function) --------`-------- singlefilehost!?A0x83a23e19::read_config_and_execute+0x10a [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 520] 
22 000000e5`f8f7f590 00007ff7`96256eab singlefilehost!fx_muxer_t::handle_exec_host_command+0x214 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 1001] 
23 000000e5`f8f7f680 00007ff7`96253285 singlefilehost!fx_muxer_t::execute+0x39b [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 566] 
24 000000e5`f8f7f7c0 00007ff7`9624faac singlefilehost!hostfxr_main_bundle_startupinfo+0x155 [F:\workspace\_work\1\s\src\installer\corehost\cli\fxr\hostfxr.cpp @ 40] 
25 000000e5`f8f7f8e0 00007ff7`9624fdf4 singlefilehost!exe_start+0x658 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 207] 
26 000000e5`f8f7fac0 00007ff7`962824e8 singlefilehost!wmain+0xa4 [F:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 302] 
27 (Inline Function) --------`-------- singlefilehost!invoke_main+0x22 [D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
28 000000e5`f8f7faf0 00007ffb`2b897034 singlefilehost!__scrt_common_main_seh+0x10c [D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
29 000000e5`f8f7fb30 00007ffb`2c6dcec1 KERNEL32!BaseThreadInitThunk+0x14 [clientcore\base\win32\client\thread.c @ 64] 
2a 000000e5`f8f7fb60 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [minkernel\ntdll\rtlstrt.c @ 1153] 

[janvorli]

It passes that to CLRLoadLibraryEx and that one fails.

[janvorli]

Just looking via a binary editor at the end of the single exe file, it seems that the r2rtest.r2r.dll is packaged in it, so I guess the problem is that in the code path that loads the composite image, we don't go through a helper that can find it in the current exe and we call LoadLibraryEx directly instead. I don't remember the details on where we handle the loading from the exe image at the moment.

[janvorli]

And one last comment - copying the r2rtest.r2r.dll from the obj\Debug\net5.0\win-x64\R2R next to the single exe makes it run successfully. So it seems this is the only problem.

[am11]

Thanks. I have tested on Windows, it works. However, on Linux, it seems like copying r2rtest.r2r.dll has no effect on sigabrt:

~/r2rtest$ cp ./obj/Debug/net5.0/linux-x64/R2R/r2rtest.r2r.dll bin/Debug/net5.0/linux-x64/publish/

~/r2rtest$ ls bin/Debug/net5.0/linux-x64/publish/
r2rtest  r2rtest.r2r.dll  # on Windows we have coreclr.dll and few more files here, but not in Linux

~/r2rtest$ bin/Debug/net5.0/linux-x64/publish/r2rtest
Aborted (core dumped)

[trylek]

I think that the main problem is that the single file + composite R2R combo is currently unsupported because NativeImage::Open doesn't perform the IsAppBundle check. As Vitek explained to me offline, in single-file mode the bundle components are represented by pseudo-paths and these probably wreak havoc in the Linux version. I'm working with Vitek on fixing this, I'll keep this thread posted with the development.

[janvorli]

@am11 on Linux, even the coreclr and jit are embedded in the single exe.

[janvorli]

Here is the call stack from Linux (without the r2rtest.r2r.dll being copied next to the executable). To get the symbols, the r2rtest needs to be renamed to singlefilehost. Then the dotnet-symbol tool can download the symbols. Then execute the renamed binary and lldb can see all the symbols:

* thread #1, name = 'singlefilehost', stop reason = signal SIGABRT
  * frame #0: 0x00007ffff6714428 libc.so.6`__GI_raise(sig=6) at raise.c:54
    frame #1: 0x00007ffff671602a libc.so.6`__GI_abort at abort.c:89
    frame #2: 0x0000555555c12a3e singlefilehost`::PROCAbort() at process.cpp:3474:5
    frame #3: 0x0000555555c129ae singlefilehost`::RaiseFailFastException(pExceptionRecord=<unavailable>, pContextRecord=<unavailable>, dwFlags=<unavailable>) at process.cpp:1395:5
    frame #4: 0x0000555555912215 singlefilehost`NativeImage::Open(componentModule=<unavailable>, nativeImageFileName=<unavailable>, pAssemblyLoadContext=<unavailable>, pLoaderAllocator=0x00005555561514f0) at nativeimage.cpp:195:9
    frame #5: 0x00005555558ab411 singlefilehost`AssemblyLoadContext::LoadNativeImage(this=<unavailable>, componentModule=0x00005554db994020, nativeImageName=<unavailable>) at assemblyloadcontext.cpp:27:12
    frame #6: 0x00005555556ec4e9 singlefilehost`ReadyToRunInfo::Initialize(Module*, AllocMemTracker*) at readytoruninfo.cpp:506:29
    frame #7: 0x00005555556ec4cc singlefilehost`ReadyToRunInfo::Initialize(pModule=0x00005554db994020, pamTracker=0x00007fffffffdbe0) at readytoruninfo.cpp:592
    frame #8: 0x00005555558b420e singlefilehost`Module::Initialize(this=0x00005554db994020, pamTracker=0x00007fffffffdbe0, szName=<unavailable>) at ceeload.cpp:596:34
    frame #9: 0x00005555558b3224 singlefilehost`Module::DoInit(this=0x00005554db994020, pamTracker=0x00007fffffffdbe0, szName=0x0000000000000000) at ceeload.cpp:175:9
    frame #10: 0x00005555558b4ed0 singlefilehost`Module::Create(pAssembly=0x000055555622dbb0, moduleRef=637534208, file=0x000055555622d670, pamTracker=0x00007fffffffdbe0) at ceeload.cpp:1162:18
    frame #11: 0x00005555558a56a6 singlefilehost`Assembly::Init(this=0x000055555622dbb0, pamTracker=0x00007fffffffdbe0, pLoaderAllocator=<unavailable>) at assembly.cpp:178:23
    frame #12: 0x00005555558a63ac singlefilehost`Assembly::Create(pDomain=<unavailable>, pFile=<unavailable>, debuggerFlags=<unavailable>, fIsCollectible=<unavailable>, pamTracker=0x00007fffffffdbe0, pLoaderAllocator=0x0000000000000000) at assembly.cpp:384:20
    frame #13: 0x00005555558f6a13 singlefilehost`DomainAssembly::Allocate(this=0x000055555622d970) at domainfile.cpp:1438:38
    frame #14: 0x00005555558f5eca singlefilehost`DomainFile::DoIncrementalLoad(this=0x000055555622d970, level=FILE_LOAD_ALLOCATE) at domainfile.cpp:491:9
    frame #15: 0x000055555589a5cc singlefilehost`AppDomain::TryIncrementalLoad(this=<unavailable>, pFile=0x000055555622d970, workLevel=FILE_LOAD_ALLOCATE, lockHolder=0x00007fffffffdf08)), &(FileLoadLock::HolderLeave(FileLoadLock*)), 0ul, &(int CompareDefault<FileLoadLock*>(FileLoadLock*, FileLoadLock*)), true>&) at appdomain.cpp:3306:31
    frame #16: 0x00005555558999c2 singlefilehost`AppDomain::LoadDomainFile(this=<unavailable>, pLock=0x000055555622da50, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at appdomain.cpp:3240:21
    frame #17: 0x0000555555899fdc singlefilehost`AppDomain::LoadDomainAssemblyInternal(this=0x00005555562044d0, pIdentity=0x0000000000000000, pFile=0x000055555622d670, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at appdomain.cpp:3105:40
    frame #18: 0x000055555589623e singlefilehost`SystemDomain::LoadBaseSystemClasses() [inlined] AppDomain::LoadDomainAssembly(this=<unavailable>, pSpec=<unavailable>, pFile=<unavailable>, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at appdomain.cpp:2967:16
    frame #19: 0x000055555589622f singlefilehost`SystemDomain::LoadBaseSystemClasses(this=0x0000555556150b10) at appdomain.cpp:1425
    frame #20: 0x000055555589603d singlefilehost`SystemDomain::Init(this=0x0000555556150b10) at appdomain.cpp:1291:5
    frame #21: 0x00005555558916c0 singlefilehost`EEStartupHelper() at ceemain.cpp:975:33
    frame #22: 0x0000555555890b09 singlefilehost`EEStartup() [inlined] EEStartup(this=<unavailable>, p=<unavailable>)::$_0::operator()(void*) const at ceemain.cpp:1139:9
    frame #23: 0x0000555555890a0c singlefilehost`EEStartup() at ceemain.cpp:1141
    frame #24: 0x000055555589096d singlefilehost`EnsureEEStarted() at ceemain.cpp:324:17
    frame #25: 0x000055555564a57e singlefilehost`CorHost2::Start(this=0x00005555561af060) at corhost.cpp:101:14
    frame #26: 0x0000555555649b83 singlefilehost`::coreclr_initialize(exePath="/home/janvorli/issues/45277/bin/Debug/net5.0/linux-x64/publish/singlefilehost", appDomainFriendlyName="clrhost", propertyCount=<unavailable>, propertyKeys=<unavailable>, propertyValues=0x00005555561a7ac0, hostHandle=0x00007fffffffe3a8, domainId=0x00007fffffffe3a4) at unixinterface.cpp:236:16
    frame #27: 0x000055555561e9a0 singlefilehost`coreclr_t::create(libcoreclr_path=<unavailable>, exe_path="/home/janvorli/issues/45277/bin/Debug/net5.0/linux-x64/publish/singlefilehost", app_domain_friendly_name=<unavailable>, properties=0x00005555561a3198, inst=nullptr) at coreclr.cpp:58:10
    frame #28: 0x000055555562da35 singlefilehost`(anonymous namespace)::create_coreclr() at hostpolicy.cpp:74:23
    frame #29: 0x000055555562d45b singlefilehost`::corehost_main(argc=<unavailable>, argv=0x00007fffffffe978) at hostpolicy.cpp:404:10
    frame #30: 0x00005555556070a2 singlefilehost`fx_muxer_t::handle_exec_host_command(std::string const&, host_startup_info_t const&, std::string const&, std::unordered_map<known_options, std::vector<std::string, std::allocator<std::string> >, known_options_hash, std::equal_to<known_options>, std::allocator<std::pair<known_options const, std::vector<std::string, std::allocator<std::string> > > > > const&, int, char const**, int, host_mode_t, char*, int, int*) at fx_muxer.cpp:146:20
    frame #31: 0x0000555555606fa2 singlefilehost`fx_muxer_t::handle_exec_host_command(std::string const&, host_startup_info_t const&, std::string const&, std::unordered_map<known_options, std::vector<std::string, std::allocator<std::string> >, known_options_hash, std::equal_to<known_options>, std::allocator<std::pair<known_options const, std::vector<std::string, std::allocator<std::string> > > > > const&, int, char const**, int, host_mode_t, char*, int, int*) [inlined] (anonymous namespace)::read_config_and_execute(host_command=<unavailable>, host_info=<unavailable>, app_candidate=error: summary string parsing error, opts=0x000055555562d3c0, new_argc=1, new_argv=0x00007fffffffe978, mode=<unavailable>, out_buffer=<unavailable>, buffer_size=<unavailable>, required_buffer_size=<unavailable>) at fx_muxer.cpp:520
    frame #32: 0x0000555555606d11 singlefilehost`fx_muxer_t::handle_exec_host_command(host_command=<unavailable>, host_info=<unavailable>, app_candidate=<unavailable>, opts=<unavailable>, argc=<unavailable>, argv=0x00007fffffffe978, argoff=1, mode=apphost, result_buffer=0x0000000000000000, buffer_size=0, required_buffer_size=0x0000000000000000) at fx_muxer.cpp:1001
    frame #33: 0x00005555556057fb singlefilehost`fx_muxer_t::execute(host_command=error: summary string parsing error, argc=1, argv=0x00007fffffffe978, host_info=0x00007fffffffe750, result_buffer=0x0000000000000000, buffer_size=0, required_buffer_size=0x0000000000000000) at fx_muxer.cpp:566:18
    frame #34: 0x0000555555601d62 singlefilehost`::hostfxr_main_bundle_startupinfo(argc=1, argv=0x00007fffffffe978, host_path="/home/janvorli/issues/45277/bin/Debug/net5.0/linux-x64/publish/singlefilehost", dotnet_root="/home/janvorli/issues/45277/bin/Debug/net5.0/linux-x64/publish/", app_path="/home/janvorli/issues/45277/bin/Debug/net5.0/linux-x64/publish/45277.dll", bundle_header_offset=<unavailable>) at hostfxr.cpp:40:12
    frame #35: 0x00005555555fce48 singlefilehost`exe_start(argc=1, argv=0x00007fffffffe978) at corehost.cpp:207:18
    frame #36: 0x00005555555fd420 singlefilehost`main(argc=1, argv=0x00007fffffffe978) at corehost.cpp:302:21
    frame #37: 0x00007ffff66ff830 libc.so.6`__libc_start_main(main=(singlefilehost`main at corehost.cpp:284), argc=1, argv=0x00007fffffffe978, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe968) at libc-start.c:291
    frame #38: 0x00005555555f0aaa singlefilehost`_start + 41

[janvorli]

And actually, I've found that copying the r2rtest.r2r.dll next to the host works on Linux too. The difference is that the current directory needs to be changed to the directory with the executable.

[vitek-karas]

If the native image resolution takes current directory into consideration I think that is a bug which should be fixed. If nothing else it makes the app unpredictable in behavior. I'm not sure but it might even be a security problem (loading code from "random" location).

[AntonLapounov]

Hmm, I believe this scenario worked on Windows in July when I tested different combinations and filed issues on Crossgen2 not working with PublishReadyToRunEmitSymbols and PublishTrimmed.

[am11]

Yup, it was working with CG1 after @VSadov's PR https://github.com/dotnet/runtime/pull/40104 was merged. I guess for CG2, we may need to handle if (Bundle::AppIsBundle()) case in PEImageLayout::LoadNative.