Runtime: Single file: Guard against partial cleanup of extracted files

[theimowski]

Steps to reproduce

Unfortunately no deterministic steps to reproduce here, but we've experienced multiple cases of files getting removed from %TEMP%.net on windows.
The files that were locked by main executable were still there, but "casual" dlls and other files got removed either by AntiVirus or other software.
As a result when our long-running process tried to load files from the temp directory it failed to do so, and we even observed high memory consumption when that was the case.

As a workaround we now set the DOTNET_BUNDLE_EXTRACT_BASE_DIR variable to custom directory.
That said I think the %TEMP% directory is not a good default.

Expected behavior

The default extraction directory for Single File Publish should be "safe", meaning there's low risk other programs might try to remove stuff from there

Actual behavior

Antivirus and other software might remove files from the extraction directory

Environment data

Various flavours of Windows OS

[vitek-karas]

/cc @swaroop-sridhar

[swaroop-sridhar]

@theimowski thanks for reporting the issue.

If parts of the extraction directory are deleted, then it is bad for the single-exe app, because:

• The current run may fail because of missing DLLs
• Subsequent runs may fail because of trying to use the extraction directory with partial contents.

Were the files removed soon after the extraction (ex: the first run of the app failed) or several hours/days after the extraction? Several apps rely on the use of temporary files (ex: some msbuild targets), which may fail if files in the temp directory are arbitrarily deleted.

The default extraction directory for Single File Publish should be "safe", meaning there's low risk other programs might try to remove stuff from there

I'll look into this is 3.1 release.
Related: https://github.com/dotnet/core-setup/issues/7940

we even observed high memory consumption when that was the case.

I don't have a good explanation to this ... the assembly load logic may page in a few components (like custom assembly resolver etc) when it doesn't find a DLL. But I wouldn't expect the memory load to change significantly.

[theimowski]

The console application in question is a long-running process managed by another win32 service and is run on many windows devices (mostly desktop).
We didn't manage to identify the moment at which the files got removed, however it wasn't immediately after starting the application.
Before .net core 3.0 we used Warp to bundle single file, which would use %LOCALAPPDATA% directory for extracting - the missing files issue didn't occur back then

[swaroop-sridhar]

@theimowski in your case, do you think setting the configuration DOTNET_BUNDLE_EXTRACT_BASE_DIR is a feasible alternative to handle the issue? Thanks.

[theimowski]

It seems to do the job yeah, will keep posted if we get reports of the issue again

[rohits79]

@swaroop-sridhar sorry i don't understand how this issue is related to the environment variable fix you suggested, we do not have anti virus on RedHat 7 and there is no cleanup script that is removing files either (unless the .net subsystem does some house keeping?). I saw the same issue few weeks ago on RedHat 7 and yesterday on windows.

Can you please correct me with environment variable set, how this will mitigate the issue? Not really sure who is doing the partial cleanup here?

Here is the issue i asked on stackoverflow, i should have reported here instead. The error i saw on linux was below, this issue is thus not limited to windows, it looks more like extraction issue.

The application to execute does not exist 'logs/slk/.net/AppName/5kp4eef5.q5/AppName.dll'

As this is a show stopper (i am using it for trading system), I've decided to not package the runtime.

[swaroop-sridhar]

I think what happened is that the files within %TEMP%\.net\<appName>\<id>\* were deleted by Windows, but the containing directory itself wasn't deleted yet. So, in this intermediate state, the .net host tries to reuse the extraction, but fails.

Similarly, in your Linux run, I believe $TMPDIR is logs/slk/, and the contents of logs/slk/.net/AppName/5kp4eef5.q5/ seem to be partially cleaned out.

We're considering a service fix to re-extract the contents of the bundle on failure within the host.

In the meantime, by setting DOTNET_BUNDLE_EXTRACT_BASE_DIR, you can extract to a non-temporary location where the OS doesn't cleanup the files, thus circumventing the failure.

[Mgamerz]

Is this going to be fixed for 3.1? I heard that this was planned to have a partial fix in a servicing early 2020.

While I can direct the installation for extraction, part of what is convenient is that it differentiates every build to it's own folder. I can just wrap it in an sfx to extract the archive and the user clicks it and runs. I can manually direct it but then the user can't simply double click my executable.

[swaroop-sridhar]

@Mgamerz fix for this issue is under development; I expect this issue be fixed in the Feb or March servicing release. Thanks.

[speed2048]

I also have this issue of files missing in the .net tmp directory. I have a question about if a new version of the application is run and it will create another .netApplicationRandomName Directory. What mechanism is in place to remove the old directory? As I have seen them them lingering around.

[swaroop-sridhar]

@speed2048 the extraction directory is different for each build, in order to avoid inadvertent overwrite across app releases. There is no explicit removal for these directories. They are removed over time by the cleanup of the temp-directories.

[speed2048]

Thanks swaroop-sridar.

In the current implementation what .net core process ( if it is a process ) removes these temp-directories over time?

I can see disk space becoming an issue if you are testing self contained apps that can take between 70 - 90 Mb per version.

[swaroop-sridhar]

@speed2048 there's no .net cleanup process. The cleanup is expected to be performed by the OS cleanup of temp-directories. Typically, the single-file publishing is not used during tight-loop development cycle; rather it is designed for publishing for deployment.

[swaroop-sridhar]

Here is the proposed algorithm to solve this partial-cleanup problem:
(thanks @davidwrighton)

1. ExtractionDir = $DOTNET_BUNDLE_EXTRACT_BASE_DIR/<app>/<bundle-id>
WorkingDir = $DOTNET_BUNDLE_EXTRACT_BASE_DIR/<app>/<process-id>

2. If ExtractionDir does not exist, then

   • Create WorkingDir, and extract bundle contents within it
   • Attempt to rename WorkingDir as ExtractionDir
     
     
     
     • If the rename succeeds, continue execution of the app
     
     
     • If not, another process completed extraction; Verify and reuse this extraction (as in step 2).
     
     

3. If ExtractionDir exists, then verify that it contains all files listed in the manifest.

   • If all contents are available,
     
     
     
     • Remove WorkingDir
     
     
     • Continue execution of the app, reusing the contents.
     
     
   • If certain (or all) files are missing within ExtractionDir, then
     
     
     
     • For each missing file, do the following individually
       
       
       
       
       
       • Extract the files within WorkingDir
       
       
       
       • Create sub-paths within ExtractionDir if necessary
       
       
       
       • Rename the file from WorkingDir/path/<file> to ExtractionDir/path/<file> unless ExtractionDir/path/<file> exists (extracted there by another process in the meantime)
       
       
       
     
     
     • Remove WorkingDir
     
     
     • Continue execution of the app.
     
     

All of the renames above should be done with appropriate retries to circumvent interference from anti-virus apps.

CC: @vitek-karas @lpereira

[vitek-karas]

• What is the $DOTNET_BUNDLE_EXTRACT_BASE_DIR default value?
• I would not do the partial extract if there are missing files in ExtractionDir. In that case i would simply not trust the content of that directory - delete it and start over. There is also a problem with concurrency. If there are 2 processes trying to "fixup" the ExtractionDir at the same time it will be a mess of failures to untangle. That said trying to delete the folder is also not "easy" if there are multiple processes trying to do that at the same time.

[swaroop-sridhar]

@vitek-karas

What is the $DOTNET_BUNDLE_EXTRACT_BASE_DIR default value?

By default, DOTNET_BUNDLE_EXTRACT_BASE_DIR will be %Temp%\.net on Windows, or $TMPDIR/.net/<userid> on Unix, as noted here.

I would not do the partial extract if there are missing files in ExtractionDir. In that case i would simply not trust the content of that directory - delete it and start over.

The extraction of individual files to ExtractionDir is not designed as an optimization -- but rather to side-step race-conditions and associated complexities with removing/recreating the ExtractionDir once it is commissioned. For example, after verification failed for ExtractionDir, another process may have recreated ExtractionDir and may be using it.

That said trying to delete the folder is also not "easy" if there are multiple processes trying to do that at the same time.

Yes, removing/re-extracting to ExtractionDir as a whole will involve locking protocols something like system-wide semaphores, or using monotonically increasing ExtractionDir name, etc.

. There is also a problem with concurrency. If there are 2 processes trying to "fixup" the ExtractionDir at the same time it will be a mess of failures to untangle

I don't see the problem if multuple processes are trying to fixup ExtractionDir. A file is only written to ExtractionDir via atomic rename system call. The rename must be done with retries; where each time we check if the file already exists.

There is no evidence that the temp-directory cleanup removes partial contents of files; so if a file exists in ExtractionDir we assume that it is correct.

[vitek-karas]

@swaroop-sridhar OK, makes sense. I do agree that "delete whole dir" is also problematic. And now reading it again, it's probably worse then updating it piece by piece. One note on that though:

We need to document somewhere that the extraction process may add files to the folder. Basically if the app carries a file with it, which is later on deleted by the app - it will be recreated next time I run it. That behavior will not be observable on "debug builds", but only on the single-file builds. Vast majority of users won't run into this, but some might. It's basically the scenario where the app carries a config file with it and writes to it as user modifies settings. If that app can also delete such file... it would hit this problem (I'm in no way suggesting apps should have files like this as part of their install BTW).

[swaroop-sridhar]

Thanks @vitek-karas. Yes these are good points to note in the documentation.

They are not specific to the current fix -- if ExtractionDir is removed, the files get re-extracted to that directory; so, the problem is already exists with single-files. However, this fix exacerbates the issue for file deletions, because re-extraction will be immediate on subsequent runs.

[blakepell]

Has there been any discussion about a switch the developer could use that would tell the extraction to replace the previous extraction (e.g. remove the previous version(s) if they exist and use the new one/current one that's being executed). In cases where this is appropriate the developer could toggle this setting on publish and it would alleviate the issue of having 10 old versions (at 150MB a version) stored when only the newest one is wanted/needed. It wouldn't stop a user from using old versions but switching versions back and forth (rare case when people are typically using the most updated bundle) would take the extraction hit.

[swaroop-sridhar]

Currently, there is no plan to support explicit removal of extraction directories. Removing extractions is tricky, because another process may be using them. It is very unlikely that the feature will be added to .net core 3.1 in servicing releases. So, the only cleanup currently is via tmp-directory cleanup by the OS.

In .net 5, the amount of extracted files will be minimal (or none, depending on the app). So, there we can consider the design where every execution extracts to a temp-directory, and attempts to cleanup after its completion. However, due to various failure-modes in the app, the runtime may not always get a chance to cleanup the extraction.

[speed2048]

A couple of things:

• To confirm, If someone overrides the DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable, the directories and files will never be pruned.
• Time Frame for fix?
• In some threads there has been discussion that Dotnet 5.0 will handle it differently, can anyone share what will change?

Thanks.

[swaroop-sridhar]

@speed2048

To confirm, If someone overrides the DOTNET_BUNDLE_EXTRACT_BASE_DIR environment variable, the directories and files will never be pruned.

Dotnet doesn't delete any extracted files. The default extraction directory is within the TEMP directory, from which the OS removes files from time to time. So this fix aims to recover those files.
If DOTNET_BUNDLE_EXTRACT_BASE_DIR is set to a non-temp directory, the files are not expected to be lost. The extracted files are verified/recovered regardless of whether DOTNET_BUNDLE_EXTRACT_BASE_DIR is set or not.

Time Frame for fix?
This fix is checked in to 5.0 branch; I'll port it to CoreCLR 3.1 branch, and the fix will likely make it into the April servicing release.

In some threads there has been discussion that Dotnet 5.0 will handle it differently, can anyone share what will change?

In .net 5, the assemblies bundled in single file apps will be loaded directly from the bundle. The design for this feature is explained in this document.

[theimowski]

Thanks @swaroop-sridhar for adding the fix - do I understand correctly that the validation mechanism is run only upon start of the application? I.e. for long running processes, if some of the extracted files are deleted, one needs to restart the app to restore those files right?

[swaroop-sridhar]

Yes, @theimowski, the extraction is only validated at startup. This is a potential risk for long running processes, but the files extracted in the temp directory are typically not lost for several months.
.net 5 will reduce/eliminate the dependency on extraction to help resolve this problem.

[hillin]

Just to make sure, this is not fixed in .net core 3.1 right? Some of our clients are running into this problem, it seems Windows or some other software clears their Temp folder in a weekly routine.

[swaroop-sridhar]

@hillin This bug is fixed in last week's 3.1.4 release. For the fix to work, apps targetting netcoreapp3.1 need to be published using the SDK in this release and re-deployed.

[yhnbgfd]

3.1.4 has been updated, but there seems to be a new problem. If PublishReadyToRun is set to true, the program will report errors. When PublishReadyToRun is set to false, the program can run normally

Err1:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Data.Sqlite, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
File name: 'Microsoft.Data.Sqlite, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'

Err2:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Authentication.JwtBearer, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
File name: 'Microsoft.AspNetCore.Authentication.JwtBearer, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
at MIQS.WebApiServer.Startup.ConfigureServices(IServiceCollection services)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.g__Startup|0(IServiceCollection serviceCollection)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.b__0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.b__0(HostBuilderContext context, IServiceCollection services)
at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
at Microsoft.Extensions.Hosting.HostBuilder.Build()
at MIQS.WebApiServer.Program.Main(String[] args)

[danmosemsft]

This issue is closed. Can you please open new issues?

[swaroop-sridhar]

@yhnbgfd I think the failure you report is not related to this issue. Please file a different issue with more details about how to repro the problem. Thanks.