Runtime: Support Single-File Apps in .NET 5

[swaroop-sridhar]

The goal of this effort is enable .Net-Core apps to be published and distributed as a single executable.

Goals

The .Net 5.0 single file solution should be:

• Widely compatible: Apps containing IL assemblies, ready-to-run assemblies, composite assemblies, native binaries, configuration files, etc. can be packaged into one executable.
• Can run managed components of the app directly from bundle, without need for extraction to disk.
• Usable with debuggers and tools.

User Experience

Here's the overall experience for publishing a HelloWorld single-file app in .net 5:

• Framework-dependent

  • Publish command: dotnet publish -r win-x64 --self-contained=false /p:PublishSingleFile=true
  • Published files: HelloWorld.exe, HelloWorld.pdb

• Self-contained (Linux)

  • Publish command: dotnet publish -r linux-x64 /p:PublishSingleFile=true
  • Published files: HelloWorld, HelloWorld.pdb

• Self-contained (Windows):

  • Publish command: dotnet publish -r win-x64 /p:PublishSingleFile=true
  • Published files: HelloWorld.exe, HelloWorld.pdb, coreclr.dll, clrjit.dll, clrcompression.dll, mscordaccore.dll

• Self-contained (Windows) with bundled native components:

  • Publish command: dotnet publish -r win-x64 /p:PublishSingleFile=true /p:IncludeNativeLibrariesInSingleFile=true
  • Published files: HelloWorld.exe, HelloWorld.pdb

Design Document

A detailed discussion of the goals, non-goals, related-work, options, design decisions, and implementation details of supporting single-file apps is available in this design document

Tracking Progress

• [x] Initial proposal and poll for customer response:
  
  
  
  • [x] Poll on GitHub: #11201
  
  
  • [x] Enterprise customer poll
  
  
• [x] Propose staged implementation of the feature.
• [x] Support single-file apps through extraction (.net core 3)
  
  
  
  • [x] Self-Extractor Design.
  
  
  • [x] SDK support for publishing apps as a single-file. https://github.com/dotnet/sdk/pull/3132
  
  
  • [x] AppHost support for extracting contents of a single-file bundle at startup. https://github.com/dotnet/core-setup/pull/5742
  
  
• [x] Support single-file apps running from bundle (.net 5)
  
  
  
  • [x] HostModel library
    
    
    
    
    
    • [x] Implement bundle format version 2. https://github.com/dotnet/core-setup/pull/5742
    
    
    
  
  
  • [x] SDK
    
    
    
    
    
    • [x] Implement support for publishing a few files when necessary for PublishSingleFile. https://github.com/dotnet/sdk/pull/11586
    
    
    
    • [x] Implement optional settings for single-file publishing. https://github.com/dotnet/sdk/pull/11797
    
    
    
    • [x] Use SingleFileHost when publishing self-contained single-file apps. https://github.com/dotnet/sdk/issues/11567 https://github.com/dotnet/sdk/pull/11797
    
    
    
    • [x] Trim the set of native dependencies used for single-file publish. https://github.com/dotnet/runtime/pull/36578 https://github.com/dotnet/sdk/pull/11797
    
    
    
  
  
  • [x] Host
    
    
    
    
    
    • [x] Move bundle-processing from the host to the framework . https://github.com/dotnet/runtime/pull/34274
    
    
    
    • [x] Process deps.json and runtimeconfig.json files directly from bundle. https://github.com/dotnet/runtime/pull/34274
    
    
    
    • [x] Implement a callback for the runtime to probe the contents of the bundle. https://github.com/dotnet/runtime/pull/34845
    
    
    
    • [x] Host Builds
    
    
    
    • [x] Windows: Implement SingleFileHost with AppHost, HostFxr, and HostPolicy statically linked. #32823 #36230
    
    
    
    • [x] Linux: Implement SingleFileHost with host and runtime components statically linked.
      
      
      
      
      
      
      
      • [x] Statically link CoreCLR and ClrJIT with SingleFileHost #37119 #36847
      
      
      
      
      • [x] Statically link core native libraries with SingleFileHost #38304
      
      
      
      
    
    
    
    • [x] Runtime
      
      
      
      
      
      
      
      • [x] Load assemblies directly from bundle. #32822 #36052
      
      
      
      
      • [x] Dependency resolution: Resolve assemblies from bundle. #36052
      
      
      
      
    
    
    
  
  
  • [x] APIs
    
    
    
    
    
    • [x] AppContext.BaseDirectory should return the location of the single-file executable https://github.com/dotnet/runtime/pull/34274
    
    
    
  
  

[msftbot[bot]]

Tagging subscribers to this area: @swaroop-sridhar
Notify danmosemsft if you want to be subscribed.

[YZahringer]

Single-File deployment support for IIS and Azure Web Apps is planned? Related to https://github.com/dotnet/aspnetcore/issues/14473

[psmulovics]

Any chance of implementing this to be single file app for windows?

[swaroop-sridhar]

@YZahringer Loading single-file apps out-of-proc by IIS should be supported. However, loading single-file apps in-proc is not planned for .net 5.

In a single-file app, since host-components (hostfxr. hostpolicy) don't exist as real files on disk, in-process loading of such apps require a new build configuration. That is:

• ASP.NET would need to produce a unmanaged .dll that is a combination of the IIS module and the single file host.
• SDK would need to allow customization of the single file host to use.

[swaroop-sridhar]

Here's the Table of supported host configuration, thanks to @vitek-karas

FD = Framework Dependent
SD = Self-Contained

App  | Normal FD | Normal SC | Single-File 3.1 FD | Single-File 3.1 SC | Single-File 5.0 FD | Single-File 5.0 SC
-- | -- | -- | -- | -- | -- | --
.exe app | Yes | Yes | Yes | Yes | Yes | Yes
Native hosting app (IIS scenario) | Yes | Yes | Not Planned | Not Planned (complex) | Not Planned | Not Planned (needs new build config)
Native hosting component (assembly) (COM server, plugins, …) | Yes | No (desire exists) | Not Planned| Not Planned | Not Planned | Not Planned
Load a plugin | Yes | Yes | Yes | Yes | Yes | Yes

[swaroop-sridhar]

@psmulovics: Are you asking about Asp.net/IIS issue, or Windows single-file apps in general?
Yes this feature is planned for Windows. Please see the notes in the "User Experience" section, and this design doc. Thanks.

[YZahringer]

@swaroop-sridhar thank you for the answer and the details. This is not blocking, but the structure of the deployed files is not very "clean" at the moment.

When deployed in Self-contained mode, I have 313 .NET Framework files at root folder of my AzureWebApp/IIS website. It's a bit messy and it can cause human errors during updates or complicate the editing of configuration files like appsettings.json and web.config. Previously with ASP.NET 4, these files are in a bin folder, no binary at the root and more clean.

It might be cleaner/magic with a Single-File deployed 😉

[davidfowl]

@YZahringer unfortunately IIS in process doesn't support single file. Out of process does.

[Lonli-Lokli]

Is it supposed that all runtimes (Alpine as well) will support PublishReadyToRun flag?

[swaroop-sridhar]

@Lonli-Lokli did you mean to ask about PublishSingleFile or PublishReadyToRun property?
PublishSingleFile is expected to work on all platforms, including Alpine.

[Lonli-Lokli]

@swaroop-sridhar yep, you are right, it was a typo.
Actually, given the description

Widely compatible: Apps containing IL assemblies, ready-to-run assemblies, composite assemblies, native binaries, configuration files, etc. can be packaged into one executable.

PublishReadyToRun should be supported as well for all runtimes, otherwise SFA cannot be verified for all runtimes.

[Serentty]

I would personally like to see a single-file distribution option that uses a file format that can be associated with the dotnet loader instead of each individual platform's native executable loader, similar to JAR files in the JVM world, so that there can be an option for easy-to-open portable binaries that don't depend on providing a loader for each platform.

[Serentty]

Obviously not suggesting this for .NET 5 or anything. Just for the future in general, it would be nice to have a file format that any user with .NET installed can open, without worrying about having a matching native loader.

[Serentty]

Judging by the reactions I guess that isn't a popular idea. To be honest I would be happy even if it weren't a single file, but just a small text file or something that points to an entry point in a DLL, and maybe holds some dependency information and stuff like that. Just anything nicer than a shell script as a portable solution for launching a program. If it's not single-file it starts to be a bit off-topic for this issue though, so let me know if I should open something somewhere else.

EDIT: Reading my previous comment I also realize that it made it sound like I was saying I wanted that _instead_ of single native executables. That's not what I meant and I see why ready-to-run self-contained binaries are useful.

[swaroop-sridhar]

@Serentty Platform independent single-file apps is tracked by https://github.com/dotnet/runtime/issues/13677. As you've noted in your comments, this is beyod the scope for .net5. We'll keep this in mind for .net6 planning. Thanks.

[Serentty]

Oh wow, looks like I've found and commented in that issue before, and then forgotten about it. Alright, thanks, I'll be on my way over there.

[ImVexed]

Is there anything special that needs to be done to also package hostfxr/hostpolicy inside the single executable?

I'm using: dotnet publish -r linux-x64 /p:PublishSingleFile=true

Not sure if I'm missing something, but using .Net 5 Preview 5 on linux, and building for both linux/win gives me an executable then 2 dll/so's. Which kinda defeats the purpose of a single executable :/

[swaroop-sridhar]

Is there anything special that needs to be done to also package hostfxr/hostpolicy inside the single executable?

I'm using: dotnet publish -r linux-x64 /p:PublishSingleFile=true

Not sure if I'm missing something, but using .Net 5 Preview 5 on linux, and building for both linux/win gives me an executable then 2 dll/so's. Which kinda defeats the purpose of a single executable :/

@ImVexed. This is a known issue in Preview 5, and will be fixed in Preview 6. Please see: https://github.com/dotnet/core/blob/master/release-notes/5.0/5.0-known-issues.md#preview-5-1

[swaroop-sridhar]

Until Preview 4, all files bundled in a single-file app were extracted to the disk.
In Preview 5, the application config files were no longer extracted, but used directly form the bundle.
In Preview 6 assemblies embedded in the bundle are loaded directly without need for extraction to disk.

By default, the Preview6 SDK only bundles files that can be run directly from bundle into the single-file app. That is, PublishSingleFile could generate "a few files" (especially for self-contained apps) but will run without need to extract contents to disk. Specifically:

• The managed app, runtime configuration files (deps.json, runtimeconfig.json), and managed binary dependencies to be embedded within the single-file app, and used directly from the bundle. The bundled assemblies are loaded as-if they were loaded from a memory-stream. The Assembly.Location for such assemblies will be an empty string, and certain apps may need to adapt to this behavior.
• The native host components (hostfxt and hostpolicy) are statically linked to the single-file application host.
• Any other native runtime binaries (including runtime components like coreclr and clrjit) are left unbundled beside the single-file app. There is an optional setting to IncludeNativeLibrariesInSingleFile, and extract them to disk on startup.
• Any other published content is also left unbundled beside the single-file app. There is an optional setting to IncludeAllContent, and extract the additional files to disk on startup.

These settings as explained in the design doc.

There are two main limitations in this preview for Unix systems (Linux/macOS):

• The singlefilehost with runtime components linked in is still to be implemented. Therefore, the runtime native binaries will be published as separate files (similar to Windows experience). #37119 , #38304
• Ready-to-run assemblies embedded in a bundle are loaded like IL assemblies, which may result in slower startup. #38061

[Herohtar]

When using PublishSingleFile with Preview 6 on a Windows 10 system with .NET Core 2.1 installed, I get the following error message:

The required library C:\Program Files\dotnet\host\fxr\2.0.7\hostfxr.dll does not support single-file apps.
  _ To run this application, you need to install a newer version of .NET Core.

  - https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=win10-x64&apphost_version=5.0.0-preview.6.20305.6

If I change the PATH environment variable to point to a folder containing .NET Core 3.1 I get the same message, but the path shown is the path to the 3.1.0 hostfxr.dll. So it seems the generated executable is trying to use hostfxr.dll from the system instead of the one that is bundled.

[swaroop-sridhar]

@Herohtar can you please file a separate issue with a repro of the app you're trying to run?
Please give the details whether you are publishing a framework-dependent or self-contained app.
What is the target framework for the app? Thanks.

[gjaw]

It appears that self-contained single-file executable with native libraries embedded doesn't work for WPF apps currently. Attempting to launch the executable silently fails, but Windows Event Viewer is able to show that the error happened because of WPF DLL consistency check not allowing loading of PresentationNative_cor3.dll from the TEMP location where it was extracted from the bundle, expecting the file to be available at the location the .exe was originally started from. Is this a known limitation currently?

[vitek-karas]

@gjaw Can you please create a new issue for the problem you're running into? In it, please include:

• The version of SDK you use to build the app
• Ideally the repro app (or if you can repro it with a template like dotnet new wpf even better)
• The target framework for the app (netcoreapp3.1 for example)

[gjaw]

@gjaw Can you please create a new issue for the problem you're running into? In it, please include:

• The version of SDK you use to build the app
• Ideally the repro app (or if you can repro it with a template like dotnet new wpf even better)
• The target framework for the app (netcoreapp3.1 for example)

Thanks, I submitted bug #38636

[swaroop-sridhar]

@gjaw Thanks for reporting the issue. I ran into a similar problem trying to test NugetPackageExplorer with PublishSingleFile=true and IncludeNativeLibrariesInSingleFile=true. I'm looking into it.
The issue likely needs an update from WPF side.

[Herohtar]

@swaroop-sridhar In putting together a repo for the issue, I realized that I had forgotten to update one of the NuGet packages in my project to the preview6 version. After updating that the issue is resolved, so it was just me being dumb.

[swaroop-sridhar]

Thanks for getting back @Herohtar. The Preview6 SDK uses a different host for single-file apps (than Preview 5 and before). So, mixing builds with Pre-preview6 SDKs will not work. But a clean build with the Preview6 SDK should work OK. Thanks.

[scamille]

The Assembly.Location for such assemblies will be an empty string, and certain apps may need to adapt to this behavior.

We currently use FileVersionInfo.GetVersionInfo(assembly.Location) to display versioning info in certain locations of our application. One particular reason why we use the file version ProductVersion is that it contains the Semver version string and not just the 4 numbers of a assembly version.

Will there be any way to still access these informations, or rather what would be the correct way to obtain the file name of the published SingleFile executable, which the user has launched?

Edit: I have found an answer to my problem elsewhere. Process.GetCurrentProcess().MainModule?.FileName seems to get the correct file in all cases.

[vitek-karas]

Your solution should definitely work. I just wanted to add that currently we don't plan to have an API which would provide access to the assemblies in the single-file directly (as byte-streams).

[jnm2]

@scamille I usually use assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>().InformationalVersion for that since we also use semver prerelease and/or build metadata. It seems less indirect than going to the win32 version resource block.

[giggio]

@agocke is this ready? Should we expect it in the next preview?

[agocke]

Yup, single-file publishing with managed assembly embedding should be widely available in 5.0

[sttz]

@giggio See the section on single file applications in the .NET 5.0 Preview 8 blog post, especially:

We were not able to implement this approach on Windows or macOS, due to various operating system constraints. We do not have a superhost on Windows or macOS. On those operating systems, the native runtime binaries (~3 of them) sit beside the single file app. We will revisit this situation in .NET 6.0, however, we expect the problems we ran into to remain challenging.

[agocke]

Yes, at the moment we can only fully embed managed assemblies . The runtime itself consists of ~4 files on Mac/Windows. We'll work to reduce that to 1 for .NET 6.

[d3c0d3d]

The Assembly.Location for such assemblies will be an empty string, and certain apps may need to adapt to this behavior.

Because of this problem the code below no longer works in a Self-Contained Single File application:

var domainAssemblys = AppDomain.CurrentDomain.GetAssemblies();
var metadataReferenceList = new List<MetadataReference>();

foreach (var assembl in domainAssemblys)
{   
    // Not work in NET 5.0 Self-Contained Single File, Cause 'Path' Exception
    var assemblyMetadata = AssemblyMetadata.CreateFromFile(assembl.Location);
    var metadataReference = assemblyMetadata.GetReference();
    metadataReferenceList.Add(metadataReference);
}

// Add extra refs | Not work in NET 5.0 Self-Contained Single File, Cause 'Path' Exception
metadataReferenceList.Add(MetadataReference.CreateFromFile(typeof(System.Linq.Expressions.Expression).Assembly.Location));

Any workarounds to solve this problem?

[jkotas]

Use AssemblyExtensions.TryGetRawMetadata to get the metadata of the assembly loaded from memory.

[d3c0d3d]

@jkotas awesome! Thank you for pointing me the way forward, solved my problem.
Thanks, i have a good job!

[davidfowl]

Can you post the complete code for other people to see 😬?

[d3c0d3d]

@davidfowl Yes with pleasure!

var domainAssemblys = AppDomain.CurrentDomain.GetAssemblies();
var metadataReferenceList = new List<MetadataReference>();

foreach (var assembl in domainAssemblys)
{   
    unsafe
    {
        assembl.TryGetRawMetadata(out byte* blob, out int length);                    
        var moduleMetadata = ModuleMetadata.CreateFromMetadata((IntPtr)blob, length);
        var assemblyMetadata = AssemblyMetadata.Create(moduleMetadata);
        var metadataReference = assemblyMetadata.GetReference();
        metadataReferenceList.Add(metadataReference);
    }
}
unsafe
{
    // Add extra refs
    typeof(System.Linq.Expressions.Expression).Assembly.TryGetRawMetadata(out byte* blob, out int length);
    metadataReferenceList.Add(AssemblyMetadata.Create(ModuleMetadata.CreateFromMetadata((IntPtr)blob, length)).GetReference());
}

For the publishing/compilation process to work perfectly, it is necessary to allow unmanaged code in all projects.

<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
   <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
   <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>

Thank you, I hope this information is useful to someone.

[tmds]

@swaroop-sridhar @richlander @KathleenDollard it would be nice if this gets exposed through the dotnet cli.
You can't figure out how to do this from dotnet publish --help.

$ dotnet publish --help
Usage: dotnet publish [options] <PROJECT | SOLUTION>

Arguments:
  <PROJECT | SOLUTION>   The project or solution file to operate on. If a file is not specified, the command will search the current directory for one.

Options:
  -h, --help                            Show command line help.
  -o, --output <OUTPUT_DIR>             The output directory to place the published artifacts in.
  -f, --framework <FRAMEWORK>           The target framework to publish for. The target framework has to be specified in the project file.
  -r, --runtime <RUNTIME_IDENTIFIER>    The target runtime to publish for. This is used when creating a self-contained deployment.
                                        The default is to publish a framework-dependent application.
  -c, --configuration <CONFIGURATION>   The configuration to publish for. The default for most projects is 'Debug'.
  --version-suffix <VERSION_SUFFIX>     Set the value of the $(VersionSuffix) property to use when building the project.
  --manifest <MANIFEST>                 The path to a target manifest file that contains the list of packages to be excluded from the publish step.
  --no-build                            Do not build the project before publishing. Implies --no-restore.
  --self-contained                      Publish the .NET runtime with your application so the runtime doesn't need to be installed on the target machine.
                                        The default is 'true' if a runtime identifier is specified.
  --no-self-contained                   Publish your application as a framework dependent application without the .NET runtime. A supported .NET runtime must be installed to run your application.
  --nologo                              Do not display the startup banner or the copyright message.
  --interactive                         Allows the command to stop and wait for user input or action (for example to complete authentication).
  --no-restore                          Do not restore the project before building.
  -v, --verbosity <LEVEL>               Set the MSBuild verbosity level. Allowed values are q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic].
  --no-dependencies                     Do not restore project-to-project references and only restore the specified project.
  --force                               Force all dependencies to be resolved even if the last restore was successful.
                                        This is equivalent to deleting project.assets.json.

[jjxtra]

In the case of multiple single file exe's embedding native binaries will they all share the same temporary native extracted dlls (assuming all exe were published with same runtime version)?

[vitek-karas]

@jjxtra No - each application extracts into its own unique directory, there's no sharing.

[jjxtra]

Publishing with the VS release from Sept. 15 and matching SDK for .NET 5.

I'm seeing extra files that do not appear to be part of the spec. Here is my publish command line:

-f net5.0 -o package/win-x64 -c Release -r win-x64 /p:PublishSingleFile=true /p:IncludeNativeLibrariesInSingleFile=true

I would expect my exe, pdb and my appsettings.json files. Instead I also have:

clrcompression.dll
clrjit.dll
coreclr.dll
e_sqlite3.dll
mscordaccore.dll
sni.dll
ReferencedProject.pdb

The Linux publish has

libe_sqlite3.so
System.IO.Ports.Native.so

These could be bugs, or more likely user error:

• Why do these files appear in the publish folder and not embed in the exe?
• Is the pdb of referenced projects unable to be embedded in the main exe pdb? This should be called out in the docs if it isn't already.
• When publishing from VS 2019 preview from Sept. 15 I select the single file option but it appears to be ignored and all the files are published separately to the output folder.

[vitek-karas]

Per the design: https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/design.md#user-experience

Single-file publish Windows: dotnet publish -r win-x64 /p:PublishSingleFile=true
Published files: HelloWorld.exe, HelloWorld.pdb, coreclr.dll, clrjit.dll, clrcompression.dll, mscordaccore.dll

On Windows we were not able to create the so called "superhost" (that is include coreclr in the exe) because it would break debuggers. We plan to improve this in future releases (get it closer to the Linux behavior).

You can bundle everything, but then it requires extraction on startup:

Single-file publish Windows with Extraction: dotnet publish -r win-x64 /p:PublishSingleFile=true /p:IncludeNativeLibrariesForSelfExtract=true
Published files: HelloWorld.exe, HelloWorld.pdb

Note that only the native dependencies will be extracted (so basically the files left beside the exe in the previous case), all managed code will still be loaded from the exe directly.

PDB: Embedding PDBs is technically possible, but it would not help with the debugging as debuggers would not know how to get to it. Thanks for pointing out the need to document this clearly.

VS: Can you please grab a screenshot of the publish settings dialog? Or ideally file a separate issue for this.

[jjxtra]

I am using the embed native binaries true parameter but the publish folder still contains the native binaries even before I run the exe. Please see my above post. I will get you a vs screenshot.

[vitek-karas]

Sorry - I missed that. The property has been renamed to IncludeNativeLibrariesForSelfExtract. The design doc actually contains the right property name now.

[jjxtra]

I will try the new property name and let you know!

[jjxtra]

Here is the VS screenshot. I think the issue is that there is no checkbox to embed the native binaries.

[vitek-karas]

Yes - you're right, there's no checkbox for the native libraries. You can do that either:

• In your .csproj file simply set the property to true -this would apply to all single-file publish operations on that project
• Edit the publish.xml and add the property there (it's an msbuild project file which is included during publish for that profile).

[jjxtra]

New property name is working! If it's possible to merge all the pdb into one that'd be awesome, if not no big deal :)

[jjxtra]

One final piece of feedback - compress the assemblies (with max compression) embedded in the .exe - this will greatly reduce the .exe size especially in cases where trimming is not used or trimming fails with various errors such as

ILLink : error IL1005: Microsoft.EntityFrameworkCore.Scaffolding.Internal.SqlServerDesignTimeServices.ConfigureDesignTimeServices(IServiceCollection): Error processing method 'Microsoft.EntityFrameworkCore.Scaffolding.Internal.SqlServerDesignTimeServices.ConfigureDesignTimeServices(IServiceCollection)' in assembly 'Microsoft.EntityFrameworkCore.SqlServer.Design.dll'

[jjxtra]

Looks like single file .exe break when trying to use map end points, map controllers methods due to use of CodeBase property on assembly... Is there an alternative or do I need to file a bug?

[agocke]

This sounds like it's worth a bug on the code using CodeBase, since that's also a deprecated API.

[jjxtra]

Sounds good, where should I log it?

[vitek-karas]

Re compression - it's definitely on the list of things we want look into, but there are caveats:

• Decompression would likely slowdown startup - it's unclear by how much, but it could be undesirable in quite a few scenarios
• R2R images might be tricky in some cases - as that's native code (note that lot of size from framework assemblies comes from R2R) - dropping R2R on the other hand will impact startup perf a lot

But maybe as an opt-in feature... it might work.

[vitek-karas]

Re usage of CodeBase - which APIs do you see it using?

[jjxtra]

Here is one stack trace when I call MapControllerRoute from a single file exe:

Error Exception: System.NotSupportedException: CodeBase is not supported on assemblies loaded from a single-file bundle.
   at System.Reflection.RuntimeAssembly.get_CodeBase()
   at Microsoft.Extensions.DependencyModel.DependencyContextLoader.GetNormalizedCodeBasePath(Assembly assembly)
   at Microsoft.Extensions.DependencyModel.DependencyContextLoader.GetDepsJsonPath(Assembly assembly)
   at Microsoft.Extensions.DependencyModel.DependencyContextLoader.LoadAssemblyContext(Assembly assembly, IDependencyContextReader reader)
   at Microsoft.Extensions.DependencyModel.DependencyContextLoader.Load(Assembly assembly)
   at Microsoft.Extensions.DependencyModel.DependencyContext.Load(Assembly assembly)
   at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.CSharpCompiler.GetDependencyContextCompilationOptions()
   at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.CSharpCompiler.EnsureOptions()
   at Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation.CSharpCompiler.get_ParseOptions()
   at Microsoft.Extensions.DependencyInjection.RazorRuntimeCompilationMvcCoreBuilderExtensions.<>c__DisplayClass2_0.<AddServices>b__2(RazorProjectEngineBuilder builder)
   at Microsoft.AspNetCore.Razor.Language.RazorProjectEngine.Create(RazorConfiguration configuration, RazorProjectFileSystem fileSystem, Action`1 configure)
   at Microsoft.Extensions.DependencyInjection.RazorRuntimeCompilationMvcCoreBuilderExtensions.<>c.<AddServices>b__2_1(IServiceProvider s)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.AspNetCore.Builder.ControllerEndpointRouteBuilderExtensions.GetOrCreateDataSource(IEndpointRouteBuilder endpoints)
   at Microsoft.AspNetCore.Builder.ControllerEndpointRouteBuilderExtensions.MapControllerRoute(IEndpointRouteBuilder endpoints, String name, String pattern, Object defaults, Object constraints, Object dataTokens)

[jjxtra]

Even just compressing the CLR assemblies would result in big size savings. Thanks for looking into it, agreed that it should be opt-in.

[jjxtra]

Size benefits of compression:

I have a 94 MB assembly with trimming turned on.
Running deflate with max compression brings this down to 34 mb. Something like z-standard would likely bring this closer to 30 mb.

Not a huge deal since I zip my deliverables anyway, but in cases where the code is run on tiny Linux images/docker or there are many executables in a tools folder, the size savings are interesting.

[extesy]

• Decompression would likely slowdown startup - it's unclear by how much, but it could be undesirable in quite a few scenarios

I highly doubt that zstd decompression (1-2 GB/s) would noticeably slow down the startup. In the worst case you could use something like lz4 which is only twice slower than memcpy.

[davidfowl]

WRT https://github.com/dotnet/runtime/issues/36590#issuecomment-696416147

Seems like we didn't update the dependency model:

https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/libraries/Microsoft.Extensions.DependencyModel/src/DependencyContextLoader.cs#L137

https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/libraries/Microsoft.Extensions.DependencyModel/src/DependencyContextLoader.cs#L151-L161

cc @pranavkm

[jjxtra]

This could be because I was using the .NET core 3.x version of the razor compilation nuget, I will try with the 5.0 rc nuget see if error goes away...

[jjxtra]

Bummer, the .net 5 nuget has the same error. Will a fix for this be in the RC 2 or RTM release?

[Sewer56]

WRT #36590 (comment)

Seems like we didn't update the dependency model:

https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/libraries/Microsoft.Extensions.DependencyModel/src/DependencyContextLoader.cs#L137

https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/libraries/Microsoft.Extensions.DependencyModel/src/DependencyContextLoader.cs#L151-L161

cc @pranavkm

Semi-Related: https://github.com/dotnet/wpf/issues/3516

Although this is by design, WPF Suffers from the same underlying issue.

[jjxtra]

Another strange error: When publishing .NET 5 to Ubuntu with Linux-x64 and trimming ENABLED, I get this error on startup:

Error Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

File name: 'System.Runtime, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
   at Microsoft.Extensions.Hosting.GenericHostWebHostBuilderExtensions.ConfigureWebHost(IHostBuilder builder, Action`1 configure)

Turning off trimming fixes the error. The error does not happen on Windows with trimming ENABLED.

[vitek-karas]

Could you please file a new issue for the missing System.Runtime?
It's not that surprising that the behavior differs between windows and Linux - some of the framework functionality is implemented quite differently across the platforms, so it might have different dependencies. And there's apparently one which the trimmer doesn't understand here.

[vitek-karas]

@extesy - it's not just the decompression itself - currently pure-IL files are directly mapped from the bundle - decompression would require to make a copy - so more memory allocation.

Overall I do agree that it's likely to not be a big issue for startup perf. Given the relative sizes of things in typical apps, the main target must be framework assemblies - as they are large, there's lot of them, and they typical make up majority of app's size (for self-contained apps anyway). The problem is that they're R2R, so the first thing is to prototype how decompression would work on R2R images in the bundle.

[jjxtra]

The more compression ratio the better - using something like LZMA will give you even more compression than zstandard - I've found it to compress about 15-20% better than zstandard. Decompression speed is much faster than compression. Memory usage can be tuned.

[jjxtra]

@vitek-karas Can you use Assembly.Load on a read-to-run byte[]?

[jjxtra]

Is there a #define that is turned on when single file publishing is active?

[vitek-karas]

Re Assembly.Load on R2R images: (caveat: I didn't try this) I think it will work, but the image will be read as pure IL - the R2R part will be ignored. So you won't get the performance benefits of R2R.

Re define: There is no define, because single-file is a publish option - so post-build. I can publish the same app multiple times with different settings without rebuilding the app.

[jjxtra]

Thanks for the response. Is there any call at runtime I could make to reliably determine the exe is single file, short of a define of my own in the build process?

[agocke]

If you need to know about a specific assembly, I'd check if Assembly.Location returns empty string. That's a good check because it should work for regular scenarios where you're loading from memory and for single-file. You could probably check corelib (e.g.typeof(System.Object).Assembly) for single-file as a whole, but it's probably best to ask the narrowest question you need the answer to, because it may be more flexible for more scenarios.

[vitek-karas]

Mostly agree with @agocke - try to make your decisions based on local knowledge as much as possible.
I'm curious - what is the scenario where you think you will need to know this?

[Sewer56]

Not the original person to ask the question but I can think of a scenario.

Various 3rd party libraries and internal solutions for self-updating applications don't clean up unused files from previous updates; only overwrite existing ones and add new files. If the application were to update from for example self contained to single file (with native dependencies); you probably wouldn't want an entire unused second copy of the runtime (assuming no trimming was used).

Edit: I implemented this the other day in one of my applications; though FDD Single File WPF applications broke due to https://github.com/dotnet/wpf/issues/3516 in .NET 5 RC1 (this is a regression) so I'm holding out for a bit. I did it by checking if Assembly.Location is an empty string (same as suggestion above) after reading through the design doc.

[jjxtra]

This is one of my use cases - automatic software updates and cleaning out non self contained stuff. This code will all be in a shared library. If the new version is self contained, I want to clean out the non-self contained files.

[harold-b]

It seems like /p:IncludeNativeLibrariesInSingleFile=true no longer works with dotnet 5.0 rc1.

dotnet 5.0.100-rc.1.20452.10 produces clrcompression.dll, clrjit.dll, coreclr.dll and mscordaccore.dll, along with the executable and pdb. Whilst version 5.0.100-preview.8.20417.9 correctly produces just the executable and the pdb file.

[Sewer56]

It seems like /p:IncludeNativeLibrariesInSingleFile=true no longer works with dotnet 5.0 rc1.

dotnet 5.0.100-rc.1.20452.10 produces clrcompression.dll, clrjit.dll, coreclr.dll and mscordaccore.dll, along with the executable and pdb. Whilst version 5.0.100-preview.8.20417.9 correctly produces just the executable and the pdb file.

Per the design: https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/design.md#user-experience

The property was renamed to IncludeNativeLibrariesForSelfExtract as opposed to IncludeNativeLibrariesInSingleFile.

[harold-b]

Thank you, @Sewer56, I missed that. I was about to look to see if there was a rename when I see your response.

[alexb5dh]

While this is marked as closed I'm getting "Including symbols in a single file bundle is not supported when publishing for .NET5 or higher" while using the following publish file:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <Configuration>Release</Configuration>
    <Platform>Any CPU</Platform>
    <PublishDir>bin\Release\net5.0\publish\</PublishDir>
    <PublishProtocol>FileSystem</PublishProtocol>
    <TargetFramework>net5.0</TargetFramework>
    <RuntimeIdentifier>linux-x64</RuntimeIdentifier>
    <SelfContained>true</SelfContained>
    <PublishSingleFile>True</PublishSingleFile>
    <PublishTrimmed>False</PublishTrimmed>
  </PropertyGroup>
</Project>

Issue has no mention that publish files are not supported and I tried to exlude them via setting _Debugging Information_ to _None_ in advanced build settings to no avail. Could someone please clarify if this issue is really closed and if single-file publish is supported in .NET 5?

[vitek-karas]

@alexb5dh it's likely that somewhere in your project file you have IncludeSymbolsInSingleFile set to true. It doesn't have to be in the publish file, it can be in the .csproj for example.
You can also try to grab a binlog (just add /bl to the command line when publishing the app) and then open it in https://msbuildlog.com/. And search for IncludeSymbolsInSingleFile it should show you where it's set to true.

[alexb5dh]

Thanks @vitek-karas, that was the case.

For those looking for a way to embed pdb - you probably need to set _DebugType_ to _embedded_, as IncludeSymbolsInSingleFile won't be supported: https://github.com/dotnet/runtime/issues/42278#issuecomment-693036257.

[vitek-karas]

Bundling PDBs would be "weird" in .NET 5 since the assemblies (.dll) are never extracted out of the bundle onto disk. So having the pdbs on disk would make little sense. Runtime should be able to access the embedded pdbs in all cases it needs to (generating stack traces, attached debugger).

[michalhosala]

I just tried to migrate my windows service to .NET 5.0 (entity framework dependency too) and publish a single file exactly as I was doing it before with .NET core 3.1. However, my publish directory contains more than single file as it used to, most importantly Microsoft.Data.SqlClient.SNI.dll, why is that so?

[agocke]

We've changed single-file to do proper load-from-memory instead of extracting to a temporary directory. Unfortunately we didn't get the runtime all the way down to one file in .NET 5, and native libraries (Microsoft.Data.SqlClient.SNI) cannot be loaded from memory. We're aiming to get the runtime down to one file for 6.0.

If you need exactly one file in 5.0 you can extract only the native libraries using <IncludeNativeLibrariesForSelfExtract>true, although that will necessarily extract the native binaries to a temporary directory.

[michalhosala]

@agocke thank you for the useful info. The main reason for us to migrate to 5.0 was that binaries will no longer have to be extracted to temp location as we were experiencing a known issue where extracted files were getting deleted randomly from there (I know there is a workaround). So using <IncludeNativeLibrariesForSelfExtract>true would effectively be the same as 3.1 for us, will have to wait for 6.0.

[jkotas]

We do not plan to solve the single-file problem with native assets of 3rd party libraries in .NET 6. I expect that Microsoft.Data.SqlClient.SNI.dll is still going to be there either on disk or be extracted into temp directory.

[cincuranet]

We do not plan to solve the single-file problem with native assets of 3rd party libraries in .NET 6. I expect that Microsoft.Data.SqlClient.SNI.dll is still going to be there either on disk or be extracted into temp directory.

Out of curiosity. Is this a technical challenge (and if so what) or simple planning/resources reason?

[jkotas]

It is technical and ecosystem challenge. We would need to:

• Introduce a new way to distribute nuget package unmanaged assets that is based on either unmanaged C/C++ sources or unmanaged .obj files.
• Add runtime support and SDK tooling for this modality. E.g. Among other things, this woudl mean that .NET SDK would have dependency on the C/C++ toolset for this modality.
• Convince the NuGet package authors with unmanaged assert to adopt this distribution model, in addition to what they have. This may be particularly challenging for NuGet package like Microsoft.Data.SqlClient where the unmanaged assets are not open source.

[jkotas]

The alternative is for NuGet package authors to eliminate unmanaged dependencies by rewriting them in C#. In fact, it is where Microsoft.Data.SqlClient is heading with UseManagedNetworkingOnWindows configuration option: https://docs.microsoft.com/en-us/sql/connect/ado-net/appcontext-switches?view=sql-server-ver15#enabling-managed-networking-on-windows . This option is only supported for testing and debugging purposes only currently.

[agocke]

So using true would effectively be the same as 3.1 for us, will have to wait for 6.0.

Not quite. Only extracting the native files is significantly less than before, so it may still be a net improvement

[michalhosala]

@agocke I meant it is going to be the same because there are going to be more files in the temp folder than just one exe and these files can again get randomly purged, as it is happening. That's why it is "the same" for us, while I of course agree the process was overally improved quite a bit.

[agocke]

We've also implemented some fixes to validate that the proper files are present whenever the app is started, but you're right: there's still an inherent race condition risk with the file system, and these changes only reduce risk, not eliminate it.