Core: PublishSingleFile results in executble that can't decode

[daz10000]

Issue Title

A binary that previously published happily in 5.0-R1 now publishes a binary that can't decompress. I am moderately confident that it's a behaviour change between RC1 and RC2 (see later comments - I did manage to get a hello world reproduction of the problem)

General

we are using single file publishing to produce an executable from an F# .net core app that is consumed by the end user as a standalone executable. We were using 3.1.200 series but ran into problems with the temporary files getting deleted and so started using the 5.0 series which has this problem fixed. RC1 was successfully building a binary ( with the caveat that we had to switch off trimmed executables due to another separate bug). The most recent attempt to build a standalone binarysucceeds but when run, the binary reports it seems to be missing clrcompression.dll, which I can see in the publication directory, but I'm assuming should be also inside the single file executable.

Error:
  An assembly specified in the application dependencies manifest (OurApplicationName.deps.json) was not found:
    package: 'runtimepack.Microsoft.NETCore.App.Runtime.win-x64', version: '5.0.0-rc.2.20475.5'
    path: 'clrcompression.dll'

I have unfortunately uninstalled RC-1 but could also go back and be hundred percent sure it's just the change in the RC2 that is causing this.

[daz10000]

As as addendum, copying clrcompression.dll along with the "single file" executable helps it get further - it complaint next that clrjit.dll is missing, then when that is supplied coreclr.dll. Copying over all of these finally gets a running "standalone" binary

clrcompression.dll*
clrjit.dll*
coreclr.dll*
mscordaccore.dll*
SQLite.Interop.dll*

For what it's worth, this is the header for the project file, I think the settings are all fine

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <PublishSelfContained>true</PublishSelfContained>
    <PublishSingleFile>true</PublishSingleFile>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
  </PropertyGroup>

[daz10000]

Success! - here is a standlone simple repro

In an empty folder (e.g. tmp) run

dotnet new console -lang F#

Publish as a singlefile, self contained binary

dotnet publish -c Release --self-contained true -p:PublishTrimmed=true -p:PublishSingleFile=true -r win-x64

Copy the "standalone binary" into a new test folder

mkdir test
cp bin/Release/net5.0/win-x64/publish/tmp.exe test
cd test
./tmp.exe

Here is the full transcript in git bash

$ dotnet new console -lang F#
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on C:\tmp\tmp.fsproj...
  Determining projects to restore...
  Restored C:\tmp\tmp.fsproj (in 379 ms).
Restore succeeded.

$ dotnet publish -c Release --self-contained true -p:PublishTrimmed=true -p:PublishSingleFile=true -r win-x64
Microsoft (R) Build Engine version 16.8.0-preview-20475-05+aed5e7ed0 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored C:\tmp\tmp.fsproj (in 420 ms).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tmp -> C:\tmp\bin\Release\net5.0\win-x64\tmp.dll
  Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  tmp -> C:\tmp\bin\Release\net5.0\win-x64\publish\

$ mkdir test

$ cp bin/Release/net5.0/win-x64/publish/tmp.exe test

$ cd test
$ ./tmp.exe
Error:
  An assembly specified in the application dependencies manifest (tmp.deps.json) was not found:
    package: 'runtimepack.Microsoft.NETCore.App.Runtime.win-x64', version: '5.0.0-rc.2.20475.5'
    path: 'clrcompression.dll'

And for reference, installed versions

$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-rc.2.20479.15
 Commit:    da7dfa8840

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-rc.2.20479.15\

Host (useful for support):
  Version: 5.0.0-rc.2.20475.5
  Commit:  c5a3f49c88

.NET SDKs installed:
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100 [C:\Program Files\dotnet\sdk]
  3.1.100 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.403 [C:\Program Files\dotnet\sdk]
  5.0.100-rc.2.20479.15 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0-rc.2.20475.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0-rc.2.20475.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0-rc.2.20475.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

[daz10000]

Final comment - I went back to 3.1.403 to get this shipped to customer and the publish folder contains just an exe file, whereas publishing with RC2 was leaving the extra dlls separate, so I assumed RC2 somehow isn't taking the singlefile part seriously and leaving some dlls outside the package.

[vitek-karas]

In .NET 5 the single-file publish may produce more than one file and it's by design. See the expected behavior described here: https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/design.md#user-experience

In 3.1 single-file is basically just a self-extracting executable - it writes everything into temp and runs from there. As such for the running app nothing really changes (everything is a file, with a path and so on).

In 5 we don't write to disk by default, all managed assemblies are loaded directly from the executable. The problem is native libraries - it's technically VERY difficult to include a random .dll in an executable and run it from the executable directly. There are other reasons (like debugger support on Windows). For these reasons we leave some of the native libraries outside of the executable. On Linux we were able to get to just the executable for simple cases, on Windows we need several files next to the exe.

If you only take the .exe and run it, it won't work - it needs those additional files.

If you need .NET 5 but want the self-extracting behavior you can set IncludeAllContentForSelfExtract=true and it will behave basically the same as in 3.1.

There's also a "hybrid" mode IncludeNativeLibrariesForSelfExtract=true which will produce just one executable and that will write the native libraries to temp and then run.

[daz10000]

Thanks - appreciate the explanation. Did this change between RC1 and RC2? I swear we successfully deployed a single file executable off RC1 a couple of weeks ago. The whole single file experience has been pretty bumpy. It's super useful - we support a lot of users who want a simple artifact they can download or copy use (and that means an actual single file single file). We were burned by the older 3.1 extracted folders getting half cleaned up and not regenerating, switched to 5.0RC1 to get around that problem and ran into a bug with trimmed executable, switched that feature off and were good for a few weeks with 5.0.RC1 then upgraded to RC2 to stay current and single file now means multiple files. I understand the technical challenges doing a true single file and appreciate the pointer on docs - I think we can make this work. I guess we will have to decide if we want the self extraction or the multi file single file experience (or ask our users). Both are problematic - It is nice to be able to pick your poison. PublishSingleFile is not true though :(

[vitek-karas]

I know the name of the feature is wrong (it's actually unlikely we will ever be able to do true single-file for all apps, without extraction, it's just too hard) - unfortunately we're stuck with it (renaming would be even worse I think).

We did fix quite a few issues around the self-extract in 3.1.4 - try to get the latest 3.1 SDK and rebuild your app with it. It's not perfect, but it's better. One important thing we did was to check all of the files are present every time the app starts - so if something half-deleted the app, starting it again will "Fix it".
This is also in 5.0 and combined with the IncludeAllContentForSelfExtract will get you almost exactly the same behavior as 3.1 (with all the fixes).

For 6.0 we hope we will be able to do a true 1 file even on Windows - at least for basic apps - https://github.com/dotnet/runtime/issues/43071

For future issues with single-file - can you please create the issues in the dotnet/runtime repo? (We should be much faster responding there).

[daz10000]

Thanks! - will do and appreciate the thoughtful replies. I understand the challenges. The "singe file" support in the 3 series has been really useful for shipping artifacts to customers, even with all the imperfections. Thanks for all the hard work.