Umbraco-cms: The process cannot access the file NuCache.Content.db because it is being used by another process

[robertjf]

PR: https://github.com/umbraco/Umbraco-CMS/pull/5924

Something is happening in Azure WebApps where the NuCache.Content.db file is locked causing the site to hang. I've attached the log file for reference.

The exception is as follows:

Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
   at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
   at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNormal..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
   at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(Options options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IUmbracoContextAccessor umbracoContextAccessor, ILogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, ISiteDomainHelper siteDomainHelper, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders)

...

This is on Umbraco 8.0.1 which was upgraded from Umbraco 8.0.0. (attachment will need to be changed from .txt to.json)

UmbracoTraceLog.RD2818786B7D96.20190319.txt

[zpqrtbnk]

Is this on Umbraco Cloud, or on pure Azure? Where is D:\home\site\wwwroot\App_Data\TEMP - is it on a shared drive, or on a local drive? Is it possible that another instance of the site, running on another server, is trying to access these files? These files can only be accessed by 1 instance at a time - we have mechanism in place to ensure this on one server but not accross servers.

[robertjf]

This is in standard Azure on a basic plan with single instance only, there’s no special configuration at all, pretty vanilla Umbraco 8 with hacked up starter kit

[Shazwazza]

The same config principles apply to v8 as v7 for running on Azure. We defo need to get the docs updated (PRs are welcome!) The current azure docs are here https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/azure-web-apps

for v8 you will need these in appSettings:

• <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
• <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
  
  
  
  • Alternatively depending on your requirements there is also Examine.LuceneEngine.Directories.TempEnvDirectoryFactory
  
  

[robertjf]

interestingly enough, there is no examine config files in this website - I hadn't noticed that before... is that normal? Out of the box shouldn't this just work? Do I need to add the files?

[Shazwazza]

There is no examine config files in v8. OOTB, just like v7, you need to adjust some config to make umbraco work with azure. So the above 2 config values are needed. These are equivalent to the v7 appSetting umbracoLocalTempStorage and the examine config directoryFactory switch.

[Shazwazza]

The 2 config values are just in your web.config, these are just appSetting key values in v8

[robertjf]

ah, right - good to know :)

[robertjf]

so is it right to assume these are going to be automatically added in future, or are they only in certain situations?

[Shazwazza]

No you cannot assume that, Just like deploying v7 to Azure you need to setup specific config for that, it's no different in v8.

If however, you create an Umbraco website from the Azure portal, then yes these should be pre-configured for you but I don't think we have a v8 build on the Azure portal yet.

[JoseMarcenaro]

Thanks for explaining the required config settings.

But in v8, even with the above settings, the exact error happens when you enable Slot Swap in the Azure App Service: before the swap both slots are working ok, but after the Swap the new "Production" slot throws the error every time. A full App Service restart is needed to make things working again.

If you work with a single slot everything is Ok,

[Shazwazza]

That's interesting, will assume the same problem exists in v7 too since it's the same paradigm.

I think the only way around such behavior is to have an option to not have a persisted cache file, or name the cache file based on the AppDomainAppId + MachineName (should be unique among processes).

@zpqrtbnk what do you think here?

[JoseMarcenaro]

Note: We have a site on Umbraco 7.13.2 running on Azure with slot swapping (and the "LocalTempStorage" ="EnvironmentTemp" app setting) and we never run into this problem with the XML cache file.
Maybe because it's not open all the time?

[zpqrtbnk]

NuCache stores its file in Path.Combine(_globalSettings.LocalTempPath, "NuCache") and therefore respects the <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" /> setting. The temp path combines the application id and the site name, but not the machine name, as the local temp is supposed to be local, ie per-server.

I don't know much about "slot swaps" in Azure. But, if they are running on the same machine, same application id, same site name, same temp dir then... there might be a collision? That would require some troubleshooting to get it right. But then, as you mention, it's not only NuCache but other things too.

Now indeed, the difference is that v8 locks the NuCache files for as long as it's running where v7 was "using" the Xml file from time to time = the two sites may cohabit (but that was a bad idea).

So... to make it short, I'd like to hear more about Slot Swaps and, on all slots, get the value of HostingEnvironment.SiteName, HostingEnvironment.ApplicationID, and %temp%.

[JoseMarcenaro]

@zpqrtbnk thanks for looking into this. I will post today the requested information about the Slot Swaps and those values.

[JoseMarcenaro]

This is what I got on each slot, before and after the swap:

<!-- STAGE slot, before swap

    HostingEnvironment.SiteName = 'terniumcomdev__e951'
    HostingEnvironment.ApplicationID = '/LM/W3SVC/1464357220/ROOT'
    %temp% = 'D:\local\Temp'
    -->
<!-- PRODUCTION slot, before swap
    HostingEnvironment.SiteName = 'terniumcomdev__dd91'
    HostingEnvironment.ApplicationID = '/LM/W3SVC/1742369212/ROOT'
    %temp% = 'D:\local\Temp'
    -->

<!-- STAGE slot, after swap
    HostingEnvironment.SiteName = 'terniumcomdev__dd91'
    HostingEnvironment.ApplicationID = '/LM/W3SVC/1742369212/ROOT'
    %temp% = 'D:\local\Temp'
    -->

<!-- PRODUCTION slot, after swap 
    HostingEnvironment.SiteName = 'terniumcomdev__e951'
    HostingEnvironment.ApplicationID = '/LM/W3SVC/1464357220/ROOT'
    %temp% = 'D:\local\Temp'
    -->

And this time the error was not thrown.
In a previous execution (with different slots) I got the following:

System.IO.IOException: The process cannot access the file 'D:\local\Temp\UmbracoData\854d45b396372eb551323cc722f33136\NuCache\NuCache.Content.db' because it is being used by another process.

I am positively sure that this setting was in effect when the error was thrown:

<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />

but this one was not (may this be the issue?)

<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />

In my latest test (no error) both settings were in effect.

[zpqrtbnk]

The Lucene setting has no impact on NuCache. The fact that you are seeing an error such as The process cannot access the file 'D:\local\Temp\UmbracoData\854...136\NuCache\NuCache.Content.db' indicates that you are indeed using the proper local temp storage setting.

NuCache uses the built-in MainDom mechanism to ensure that only one app domain at a time can own the cache (and the associated files). MainDom uses a machine-wide named lock; the name is built by combining the application id and the application physical path.

(so obviously I should also have asked you the HostingEnvironment.ApplicationPhysicalPath values...)

A site cannot, in theory, even try to access the NuCache file until it has aquired the MainDom lock. Therefore, for the error to happen... the site must own the machine-wide lock on (app.id, app.path) and yet someone else must lock the files at (app.id, site.name), meaning

• either there is something weird with app.path, or
• "machine-wide" is the key here, the two slots don't run on the same physical server yet share the same diskspace

Any chance you can get the ApplicationPhysicalPath values?

And... I don't know enough about "slots" to figure out the physical server thing. Ideas?

[JoseMarcenaro]

Just tested it. The ApplicationPhysicalPath value is always D:\home\site\wwwroot\ in both slots, before and after the swap.

Regarding the physical server... it is not fully documented, but many posts assume it is the same because all slots share the same App Service Plan = the same set of resources (i.e. 3.75 Gb RAM).

[zpqrtbnk]

So... assuming that all sites run on the same physical server (for now), that leaves us with app.path.

D:\home\site\wwwroot\ is a kinda virtual path that Azure uses, so basically all apps are always under that path, and that can be annoying in some cases... but in our case, it should indeed make things even more safer.

Did not realize you posted a log file with the original issue - now looking at that file.

[JoseMarcenaro]

@zpqrtbnk - the log file is not mine, so it does not apply to the slot issue. Thanks.

[JoseMarcenaro]

In any case, for some mysterious reason the "process cannot access the file NuCache.Content.db" error has not appeared lately - in all slots swaps during the last three days.
I will post more information - including a log - if it does happen again. Thanks for your help!

[zpqrtbnk]

Thanks for the update.

Even though it's not your log... the significant lines are:
~~
2019-03-19 03:16:49,695 [P9488/D3/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.0 on RD2818786B7D96. [Timing e8bc0a7]
2019-03-19 03:16:50,759 [P9488/D3/T1] INFO Umbraco.Core.MainDom - Acquired.
2019-03-19 03:17:39,322 [P9488/D3/T17] INFO Umbraco.Core.MainDom - Released (environment)
2019-03-19 03:25:35,501 [P9488/D4/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.0 on RD2818786B7D96. [Timing 761ae5c]
2019-03-19 03:25:36,074 [P9488/D4/T1] INFO Umbraco.Core.MainDom - Acquired.
2019-03-19 03:27:45,813 [P9488/D4/T16] INFO Umbraco.Core.MainDom - Released (environment)
2019-03-19 03:27:49,388 [P9488/D5/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.0 on RD2818786B7D96. [Timing 9ce9254]
2019-03-19 03:27:50,970 [P9488/D5/T1] INFO Umbraco.Core.MainDom - Acquired.
2019-03-19 03:28:24,123 [P9488/D5/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booted. (34755ms) [Timing 9ce9254]
2019-03-19 03:28:59,434 [P9472/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.0 on RD2818786B7D96. [Timing 56405ae]
2019-03-19 03:29:06,148 [P9472/D2/T1] INFO Umbraco.Core.MainDom - Acquired.
2019-03-19 03:29:31,320 [P9472/D2/T1] ERROR Umbraco.Core.Runtime.CoreRuntime - Boot failed. (32637ms) [Timing 56405ae]
Umbraco.Core.Exceptions.BootFailedException: Boot failed. ---> System.IO.IOException: ...
2019-03-19 03:29:46,191 [P9488/D5/T13] INFO Umbraco.Core.MainDom - Released (environment)
~~

Where we see that a new process (9472) starts while the previous one (9488) is running, and acquires the MainDom lock before the previous process releases it, thus hitting an exception when trying to read the cache. That should not be possible...

If anything happens again, thanks for reporting. Meanwhile, I've made a few changes so that in 8.0.2 we log all the important infos (app.path, app.id, etc) when the site boots.

[JoseMarcenaro]

Hi @zpqrtbnk
Some more info (the exception is being thrown again in my Azure environment).
A log is attached, the same behavior you described happens in the last lines of the log.

Note that in this case:

• This is not some kind of "race condition": twelve minutes passed between the first lock acquire and the following attempt.
• There is no lock release nor application shutdown between the first acquisition (at 18:45) and the following attempt (at 18:59)

UmbracoTraceLog.RD2818784FF824.20190409.zip

I'm running v8.0.1 installed via the nuget.org package.

Hope it helps.

[JoseMarcenaro]

@zpqrtbnk - I could further isolate the problem.

I performed the following steps:

• at 19:33 GMT stopped the "stage" slot and deleted the log file
• then I started the app (on the stage slot) and got into Umbraco - that is the start of the attached log file, at 19:35
• then I "went to play outside" and did nothing with the app for twenty minutes (I am the only user, for now). I checked the log, and nothing had happen during that time.
• at 19:55 I performed a slot swap (line 16 of the log)
  ... and _voilà!_ the app successfully acquired a new lock, with a different PID, without shutting down the previous instance.

So it looks like this is related to the Azure Slot swap - although it is the same machine, for some reason the process starts over without having shut down the previous one.

UmbracoTraceLog.RD2818784FF824.20190409.zip

[JoseMarcenaro]

One last comment about the slot swap process:
the "swapped" Web app remains the same, in the same machine .. but the URLs that hit it change (i.e. from contoso-stage.azurewebsites.net to contoso.azurewebsites.net). I don't know if immediately after the switch the Azure infrastructure sends some kind of wakeup call.

[zpqrtbnk]

Hey - thanks for the details - just FYI I am away at the Barcelona meetup, with little time for this issue - will resume work on it at the end of the week (so don't feel bad if I don't reply).

[zpqrtbnk]

~Have looked at your logfile and... here is the thing: the new process (P564) is starting for a different application (since you swapped). So P8108 was for the other application. It does not have to release its lock for P564 to acquire it... OTOH the lock has to be released... but that should happen on the other slot. And so... I would love to see the other log. Is this possible?~

~Note that it still does not explain what is going on with the cache. I am currently running some experiments with Azure AppServices to try to get a better understanding of it all.~

Or maybe not - further experimenting.

[zpqrtbnk]

I have created an Azure App Service hosting a simple MVC web app, with two slots. Had to have one per-slot appSetting in order to force slot-swaps to trigger a restart. Then, have our MainDom system and some temp file lock to the app... and I can swap slots, and see the MainDom lock being properly released, and swaps happening without issues.

I have to accept that something is wrong, considering your log, but for now I am running out of idea.

During the swap, on my test, the old process terminates and then a new process starts. In your log, the new process starts before the old process terminates, but that should not be a problem: this is precisely why we have the MainDom lock.

Is this happening every time you swap, or only from time to time? Is this on a production / live site? To which extend would it be possible for you to run some custom DLLs which would include more tracing / logging?

[JoseMarcenaro]

@zpqrtbnk ,
The error is no longer happening.
By the time I reported this and send you the log, it was happening on every swap.

I did not change anything in Umbraco since then - the same package 8.0.1 is used.

I'm almost sure the difference is that, while trying to workaround the error, I completely disabled Application Insights on Azure - it was enabled by default when I first created the App Service, and I did not opt out of it as I usually do.

To verify if this was the cause I enabled again Application Insights in both slots and something strange happened:

• On both slots, immediately after installing Application Insights, I went into the site and the same NuCache locking error was thrown - no swap involved, just installing Application Insights which restarts the app. I'm attaching this log, just in case. UmbracoTraceLog.RD2818784FF824.20190413.zip. Error is at the end of the log.
• I temporarily fixed the error by stopping and starting the App Service, the site started normally.
• _Further swaps between slots worked fine with no errors_

So looking in retrospect, now my thoughts are:

• the error is not related to the swap itself, but to the restart after Application Insights is installed or removed.
• the reason why this happened to me on every swap is that when creating a stage slot for an app service that has Application Insights installed, some App Settings are duplicated but the feature is not correctly installed in the new slot - so each swap means activating / deactivating Application Insights

Current situation:

• I can run the site normally (no errors) with or without Application Insights. So it's not a priority for me that you spend more time on the issue.
• On the other hand, if you want to get to the bone of what happens, I can gladly help by installing any DLL that collects additional information. This is not a production environment yet. And I found a way to reproduce the issue: Enable Application Insights on an App Service that doesn't have it.

Thanks for your help!

[zpqrtbnk]

Thanks for keeping up with the detective work ;-) I have tried to experiment with enabling/disabling AppliCation Insights on my test app, but still cannot reproduce the out-of-order lock management that you see.

Happy that it works for you, but it annoys me.

Will lower the priority of this issue, but still, will try to send you a DLL to test, that would log way more details about what is going on. Stay tuned!

[zpqrtbnk]

This: Debug.zip contains a patched DLL that just logs infos when acquiring the main lock (process id, but also user name, app id and maindom hash) - if you have a moment to try it...

[JoseMarcenaro]

Done.
I could reproduce the error. No slot swapping involved, I just enabled Application Insights.

Initial state:

• App Service with the patched debug version of Umbraco.Core.dll
• Umbraco working normally

Clean restart to clean up logs

• stop App Service
• delete log files
• start App Service, open Umbraco

Enable Application Insights

• enable from Azure portal (use default configuration)
  
  
  
  • test Umbraco again: NuCache.db locking error, boot fails.
  
  
• collected this log: error-log.zip

Let me know if I can perform any additional tests

[zpqrtbnk]

From the log:
~~
2019-04-15 21:07:33,806 [P9404/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.1. [Timing f0400f6]
2019-04-15 21:07:33,933 [P9404/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting
site 'terniumcomqa__482f',
app '/LM/W3SVC/653304847/ROOT',
path 'D:\home\site\wwwroot\',
server 'RD2818785491E5'.
2019-04-15 21:07:34,696 [P9404/D2/T1] INFO Umbraco.Core.MainDom - DEBUG
server RD2818785491E5
site terniumcomqa__482f
process 9404
user IIS APPPOOL\terniumcomqa__482f
domain 2
appId /LM/W3SVC/653304847/ROOT
appPath D:\home\site\wwwroot\
temp D:\local\Temp
local D:\local\Temp\UmbracoData\935c1c6dfba987b17a7c67a60ce950cb
mdhash d04962ddd4ea323ea555b5a47c5349c8f613ab9e
2019-04-15 21:07:34,714 [P9404/D2/T1] INFO Umbraco.Core.MainDom - Acquiring.
2019-04-15 21:07:34,714 [P9404/D2/T1] INFO Umbraco.Core.MainDom - Acquired.
[...]
2019-04-15 21:09:54,491 [P9404/D2/T15] INFO Umbraco.Core.MainDom - Stopping (environment)
2019-04-15 21:09:54,647 [P9404/D2/T15] INFO Umbraco.Core.MainDom - Released (environment)
[...]
2019-04-15 21:10:02,321 [P9404/D3/T24] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.1. [Timing c4c5f6d]
2019-04-15 21:10:02,446 [P9404/D3/T24] INFO Umbraco.Core.Runtime.CoreRuntime - Booting
site 'terniumcomqa__482f',
app '/LM/W3SVC/653304847/ROOT',
path 'D:\home\site\wwwroot\',
server 'RD2818785491E5'.
2019-04-15 21:10:02,758 [P9404/D3/T24] INFO Umbraco.Core.MainDom - DEBUG
server RD2818785491E5
site terniumcomqa__482f
process 9404
user IIS APPPOOL\terniumcomqa__482f
domain 3
appId /LM/W3SVC/653304847/ROOT
appPath D:\home\site\wwwroot\
temp D:\local\Temp
local D:\local\Temp\UmbracoData\935c1c6dfba987b17a7c67a60ce950cb
mdhash d04962ddd4ea323ea555b5a47c5349c8f613ab9e
2019-04-15 21:10:02,778 [P9404/D3/T24] INFO Umbraco.Core.MainDom - Acquiring.
2019-04-15 21:10:02,778 [P9404/D3/T24] INFO Umbraco.Core.MainDom - Acquired.
[...]
2019-04-15 21:10:12,305 [P9404/D3/T24] INFO Umbraco.Core.Runtime.CoreRuntime - Booted. (10011ms) [Timing c4c5f6d]
[...]
2019-04-15 21:10:16,369 [P1480/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting Umbraco 8.0.1. [Timing aa57fab]
2019-04-15 21:10:16,727 [P1480/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Booting
site 'terniumcomqa__482f',
app '/LM/W3SVC/653304847/ROOT',
path 'D:\home\site\wwwroot\',
server 'RD2818785491E5'.
2019-04-15 21:10:18,852 [P1480/D2/T1] INFO Umbraco.Core.MainDom - DEBUG
server RD2818785491E5
site terniumcomqa__482f
process 1480
user IIS APPPOOL\terniumcomqa__482f
domain 2
appId /LM/W3SVC/653304847/ROOT
appPath D:\home\site\wwwroot\
temp D:\local\Temp
local D:\local\Temp\UmbracoData\935c1c6dfba987b17a7c67a60ce950cb
mdhash d04962ddd4ea323ea555b5a47c5349c8f613ab9e
2019-04-15 21:10:18,852 [P1480/D2/T1] INFO Umbraco.Core.MainDom - Acquiring.
2019-04-15 21:10:18,962 [P1480/D2/T1] INFO Umbraco.Core.MainDom - Acquired.
[...]
2019-04-15 21:10:24,822 [P9404/D2/T20] INFO Umbraco.Web.UmbracoApplicationBase - Application shutdown. Details: ConfigurationChange
[...]
2019-04-15 21:10:31,793 [P1480/D2/T1] INFO Umbraco.Core.Runtime.CoreRuntime - Resolved. (10046ms) [Timing 0ba631e]
2019-04-15 21:10:38,462 [P1480/D2/T1] ERROR Umbraco.Core.Runtime.CoreRuntime - Boot failed. (22127ms) [Timing aa57fab]
Umbraco.Core.Exceptions.BootFailedException: Boot failed. ---> System.IO.IOException: The process cannot access the file
'D:\local\Temp\UmbracoData\935c1c6dfba987b17a7c67a60ce950cb\NuCacheNuCache.Content.db'
because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
[...]
2019-04-15 21:10:58,311 [P9404/D3/T19] INFO Umbraco.Core.MainDom - Stopping (environment)
2019-04-15 21:10:58,686 [P9404/D3/T19] INFO Umbraco.Core.MainDom - Released (environment)
~~

We see P9404/D2 booting, acquiring the MainDom lock, running, then releasing the MainDom lock, because it has been notified by the environment that it had to shut down. And _then_ P9404/D3 boots, acquires the MainDom lock, and runs.

And _then_, P1480/D2 starts and... acquires the MainDom lock, even though P9404/D3 has not been notified of its shutdown and has not released the lock (it does, 20 or 30 seconds later).

The important line here is mdhash d04962ddd4ea323ea555b5a47c5349c8f613ab9e - that hash is directly used as a name for the named Semaphore we use to handle the MainDom lock - and these semaphores are supposed to be system-wide. Since the same name is used everywhere... I fail to understand how P1480/D2 can acquire a lock that has not been released.

There has to be something I am not seeing here. Thanks for reporting... will think about it all and try to have an idea!

[zpqrtbnk]

Note to self: have run some tests of the MainDom class on local machine, with two console apps, and it all works as expected. Note that the app that releases the lock is Signaled (signal) whereas in the above log, it's always (environment) meaning... P9404/D3 is not even signaled to stop?!

[zpqrtbnk]

If I can bother you with another Debug.zip... MANY THANKS for your time here!

[Jeavon]

@JoseMarcenaro just wondering, do you have any applicationInitialization configuration?

[JoseMarcenaro]

Done. Same secuence as before, same error result.
error2-log.zip

[JoseMarcenaro]

@Jeavon you are right. I do have applicationInitialization configuration.

[zpqrtbnk]

Thanks for the log. What it contains is:

• P3488/D7 starts, acquires the lock, runs...
• P3488/D7 is signaled by the hosting environment that it needs to release the lock... and releases
• P3488/D8 starts, acquires the (now released) lock, runs...
• when P3488/D8 requires the lock, it signals P3488/D7 - which does report it
• P7468/D2 starts, and wants the lock

At that point... P7468/D2 just gets the lock immediately. It signals the current lock holder, P3488/D8, but P3488/D8 does not report this. It just reports, later on, that it is signaled by the hosting environment, and releases the lock.

So

• there is a period of time when the shared semaphore is owned by two owners
• at that time, the shared event used for signaling seems to not be working
• and of course, it all crashes when trying to open the files

And then again, this all seems to happen on the very same hosting environment / server. Shared semaphores and events are at system-level so... they should work here. Just can't figure out what is going on :(

[Jeavon]

@JoseMarcenaro I think that might be a clue for @zpqrtbnk
I only asked as I only saw the error once I had added applicationInitialization, could be coincidence but with applicationInitialization IIS is internally within the module starting up a new site

[zpqrtbnk]

but with applicationInitialization IIS is internally within the module starting up a new site

afraid I don't understand ;-) can you be more... something?

[Jeavon]

And it's starting internally before shutting down the currently running one, once it gets the 200 then it makes it public to serve requests

[Jeavon]

@zpqrtbnk to enable locally add within system.webServer element of web.config

    <applicationInitialization doAppInitAfterRestart="true">
      <add initializationPage="/"/>
    </applicationInitialization>

[Jeavon]

Trying to find out how it works exactly, this seems to be the most info https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization

[JoseMarcenaro]

I thought ApplicationInitialization configuration was used by the Azure infrastructure only when launching a swap, to warm up the stage slot before performing the swap and starting to send requests to it.
But that "doAppInitAfterRestart" attribute shows me it might not be the case.
In my app, I do not have the attribute set.

    <applicationInitialization>
      <add initializationPage="/es/index" hostName="www.txdev.ternium.com"/>
    </applicationInitialization>

[Jeavon]

@JoseMarcenaro it is used whenever IIS needs to start/restart. For Web Apps it's correct you are using it. It will be used for things such as Slot Swapping, Scale (cold boots) and server transition (when your site is moved to a new server) and app recycle. Without it, you will not get continuous operation.

[JoseMarcenaro]

From https://docs.microsoft.com/en-us/iis/configuration/system.webserver/applicationinitialization/:
doAppInitAfterRestart

Optional Boolean attribute.Specifies that the initialization process is initiated automatically whenever an application restart occurs. Note that this is different than the preLoadEnabled attribute in the application element, which specifies that the initialization process is started after a restart of the application pool.
The default value is false.

[Jeavon]

@JoseMarcenaro have you tried setting it to true?

[zpqrtbnk]

So... the reason why the new domain starts before the previous one is down, would be because IIS is warming up the new domain, then swaps them, then terminates the old domain. Makes sense.

Nevertheless, in theory, the new domain should signal the old domain, and then wait for the old domain to release the lock, before acquiring it. Whereas... it all looks like the new domain is running in isolation and not seeing the rest of the world...

Going to try and repro by enabling app initialization...

[JoseMarcenaro]

@Jeavon no, but I don't think it will add any useful information.
I can run my site without these errors - it now happens only when setting on Application Insights and then everything goes right.

So avoiding the error it is not the priority. Finding more information about how it happens is.

[zpqrtbnk]

Odd, on my test app it always shuts an app pool before starting another - either I fail at configuring app initialization, or ...

[JoseMarcenaro]

@zpqrtbnk - would it be possible that one of the composable components we coded in our app, or third-party packages we use (i.e. UmbracoFileSystemProvider.Azure) are delaying the release of the app domain, and causing the collision? That would explain why you cannot reproduce the error.

[stodolos]

We're having the same issue very frequently! Several times a day... Here are the settings we have in our App Service's configuration (not in the config file itself, in case that has any bearing on this):

• Umbraco.Examine.LuceneDirectoryFactory = Examine.LuceneEngine.Directories.TempEnvDirectoryFactory, Examine
• Umbraco.Core.LocalTempStorage = EnvironmentTemp

Seems similar to this issue regarding Examine locks

Please help!

[HannesWinston]

@stodolos We had this exact problem on two of our Umbraco 8 sites running in Azure Web App. However the issue seems to be fixed now (3 days without a single crash). Here is what we did:

1. Upgraded Umbraco to 8.0.2
2. Disabled the Application Insights on the web apps (both production and stage deployment slots)
3. Completely removed the Application Insights resource in the resource group
4. Removed all app settings that were automatically applied in the configuration tab on all our web apps (most of them regarding application insights)

Then it worked and no crashes during swaps. I think the 4th step is the most important. See if you have any left over configurations you don't want.

[stodolos]

@HannesWinston Thanks for the info. Once we upgraded to the new version of Examine, we only saw this once in 2 weeks (vs every day). I also saw how 8.0.2 generates the cache file folder/name, and it's way better b/c it goes off of the process id. You would think that should avoid any collision on the same file going forward. Application Insights had no bearing for us FYI.

[brreisner]

Any update on this issue? Does 8.0.2 seem to have resolved the issues for people?

[stodolos]

I still came across this issue in 8.0.2. The folder naming is better (b/c it uses the process ID), but the same process must be trying to access that file more than once. I wonder if something like an epoch date value or a GUID would be possible as to avoid file name collision.

[brreisner]

@zpqrtbnk any details from your side? Should this issue be resolved now or in an upcoming release. Are we even certain on the cause?

[brreisner]

This issue occurred for us with a single node app service plan, no slots, with app insights and appinit enabled. Running on 8.0.2 as well.

This course focuses on running Umbraco 8 load balanced in Azure Web apps and calls out using app insights: https://umbraco.com/blog/introducing-a-brand-new-umbraco-training-course-load-balancing/

Seems like its something that should work. Very concerning, hoping someone can provide some insights as to whether this is something that is being actively worked on, with a fix perhaps by 8.1?

[JoseMarcenaro]

@brreisner - I had a similar situation in a fresh environment I set up this week, and I found a workaround: disabling app preload at the IIS level - yes, I know we are talking about Azure App Services.

The reasoning for trying this is that the conflict is somehow related to the early initialization of the app pool before the previous lock has been released. Then I found this attribute in IIS config true in Azure Web Apps.

Setting it to false stopped my NuCache lock conflict from happening on each new deploy / swap.

The thing is: because this is set at the IIS level (applicationHost.config file) you cannot edit it directly on an Azure Web App. But you can modify it by adding a transformation file, as explained here: https://blogs.msdn.microsoft.com/benjaminperkins/2018/01/09/how-to-disable-preloadenabled-on-an-azure-app-service/

NOTES:

• If the lock conflict is happening in a multi-slot App Service, you need to repeat this procedure for each of the slots, because they are separate applications on IIS.
• If you are using Slot Swap and you are concerned about the warm-up time, you can still use an <applicationInitialization> block in web.config. It does not trigger a NuCache lock conflict, at least in my case.

[brreisner]

@JoseMarcenaro thank you for the update. Has this completely resolved your issues? no further crashes? Have you been provided any official guidance or followup from someone on the Umbraco team. @zpqrtbnk any update here? Will this be addressed in 8.1?

[JoseMarcenaro]

@brreisner

• It has completely resolved my issues for now (fingers crossed).
• I received no followup about this.

[zpqrtbnk]

We need first, to reproduce and understand. Happy to see that there is a workaround, but we'd love to understand what is really going on. We do have locks in place to ensure that an app restart happens in smooth condition. But the App Preload feature seems to break our locks in very odd ways.

We are trying to prioritize this issue. Will update here.

[brreisner]

Thank you for the follow up as this is concerning for many customers who have already made investments in web apps and have v8 projects moving forward. Are you unable to reproduce still? Seems to be a rather common issue. Jose were you/are you able to provide steps that cause the issue reliably? I would be playing catchup but would be willing to help push this forward.

[brreisner]

Zpqrtbnk please let me know if any of us can help expedite this if you do not yet have the information to properly troubleshoot and isolate the issue. We have several v8 projects in the works scheduled to go live end of October. Do you have any concept of when this might be resolved or when you will be able to accurately describe what is occuring? We just passed 3months since the bug was opened. Not exactly sure what your release cycle looks like for v8, but again I would hope this makes it into 8.1?

[azure-devops-sync[bot]]

This item has been added to our backlog AB#1528

[brreisner]

@Shazwazza Seems as though the backlog is not publicly visible and I'm not sure what that means in terms of priority or timing. Any further insights you can provide?

[Shazwazza]

@brreisner we use a separate proj mgmt tool to help us manage public and internal issues - this is not public. Investigating this has been set to start at the end of this sprint so probably later next week. We'll report back on findings but can't give you an answer because this has never been seen before. Whatever this Azure setting is doing is changing how normal locking at the OS level works.

[Shazwazza]

Hi all, here's a recap - (there's a boat load of information in this thread...)

The problem

When hosting in Azure web apps the locking mechanism in Umbraco that manages overlapping appdomains (called MainDom) is being bypassed in some scenarios which causes file(s) to be accessed by multiple appdomains at one time causing file locking exceptions. In theory this should not be possible because a named Semaphore is a systemwide lock.

These scenarios seem to be:

• Using Azure slot swap functionality with the default preloadEnabled IIS setting enabled
• Single Azure Web App with Application Insights enabled and with applicationInitialization enabled

Potentially Application Insights may affect this problem but most likely enabling/disabling application insights is probably toggling the preloadEnabled, applicationInitialization or some other IIS feature.

The workarounds

1. Disabling preloadEnabled IIS setting.
2. Don't use a persisted local cache file. The content cache is always in memory, the purpose of these persisted files is purely to for startup time performance. That said, the performance of loading in the content cache directly from the DB is also very fast, sometimes negligible. Also considering this is generally a slot swap issue or pre-warming issue, startup time isn't really a problem. To disable persisted file cache you can drop this class in to your app:
   
   
   
   • public class MyComposer : IUserComposer
     
     {
     
     public void Compose(Composition composition)
     
     {
     
     composition.Register(factory => new PublishedSnapshotService.Options
     
     {
     
     IgnoreLocalDb = true
     
     });
     
     }
     
     }
   
   

The questions

I need to try to replicate this locally and hopefully the key is in this preloadEnabled setting in IIS. If that is possible then I can try to start figuring out how a named Semaphore stops behaving the way it should be.

Then need to determine if that 2nd scenario listed above (i.e. "Single Azure Web App with Application Insights enabled and with applicationInitialization enabled") can be replicated and then try to determine if the underlying cause is preloadEnabled and/or a combination of applicationInitialization or something else

---

It would be very helpful if @brreisner and @JoseMarcenaro can verify that both of these workarounds work in your scenarios?

---

Braindump

If preloadEnabled is confirmed to be the (or part) of the problem, I need to investigate what it's doing.
I haven't looked much but there's a bit of info on the web like this which explains:

When you set the startMode property of your application pool to AlwaysRunning a worker process is spawned as soon as IIS starts up and does not wait for the first user request. But this does not mean the web application is initialized.
When you set preloadEnabled to true, IIS will simulate a user request to the default page (can be changed with initializationPage metabase setting) of the website/virdir so that the application initializes. The request is not logged in the IIS logs.
But you can trace this with FREB. Every time you restart your application pool you will see a FREB trace file for the dummy request. You can identify this request by analysing the GENERAL_REQUEST_HEADERS and looking at the User-Agent string.

which is interesting and if for some crazy reason we cannot figure out why a Semaphore cannot be a Semaphore then we could consider utilizing this information and auto-disabling the persisted file cache.

[brreisner]

I will review these options with my Dev team. I'm a little confused here as I've never (not once) had this issue with v7. This would lead me to believe the logic in v7 is being respected, is something different being done in v8?

Do you have any contacts within Microsoft on the App Service team? I have a few who may be interested in assisting with this.

[Shazwazza]

@brreisner yes this is described above: https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-479942871 . V7 would have it's own issues with this too but they would be hidden from you so you wouldn't know that 2x app domains are writing/reading to the same xml file at the same time until you data is either corrupted or some other exception would cause the site to fail. v7 and v8 are very different :)

I need to replicate locally and gather specific information about what is going on to provide an accurate report before we start involving MS.

In the meantime if you can please try those workarounds that will be very helpful. Both workarounds will still allow slot swap to work and startup time won't 'be an issue anyways.

[Shazwazza]

@brreisner I am trying to replicate the problem you had/have

This issue occurred for us with a single node app service plan, no slots, with app insights and appinit enabled. Running on 8.0.2 as well.

I have configured exactly this with the settings <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />, <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" /> and cannot reproduce.

Can you elaborate on how you reproduce this file lock error with this scenario along with where and when you see the error? The site I'm testing on is https://v8preload.azurewebsites.net/ . I have tried restarting the web app in various different ways, bumping web.config, global.asax, restarting on the portal, changing the /bin contents, exceeding quotas, etc... and I do not see this issue. Any additional info would be appreciated.

[Shazwazza]

Aha! ok well i can replicate the issue once in a while by turning on/off Application Insights (no slot swap involved) though i cannot get this error to happen with a single web app by doing anything else.

I'll test site swapping next to see what I can/can't replicate replicate with/without App Insights.

And locally i cannot replicate with the preloadEnabled, applicationInitialization combos enabled/disabled with overlapping recycling enabled (basically enabling a lot of this article https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization) , but i'll keep testing.

[Shazwazza]

What is different in my tests is that instead of getting this file lock issue, i consistently get the error 502 - Web server received an invalid response while acting as a gateway or proxy server. when toggling Application Insights and when doing a site swap. I've only seen the file lock issue a handful of times and thats mostly in the logs, not as a YSOD. I've seen the YSOD a couple of times during site swap and all of this is with Application Insights fully disabled on both slots.

---

As mentioned testing locally with the above settings (including preloadEnabled), i cannot break it. I'm load testing and restarting the appdomain concurrently. I've managed to get 5x overlapping concurrent w3wp.exe processes running for the same site and get no file locking issues.

---

I've discovered a potentially related bug in that when there is overlapping appdomains and when one has been started up and shutdown very quickly, we end up with null reference exceptions in the nucache layer. You can see the rev change I'm making here: https://github.com/umbraco/Umbraco-CMS/commit/3f73bf5489c1a94b4a56c6e52b5b75358a5fdbd7#diff-a7f4263c248ded990a4eac4c9943276f

During testing on Azure, I received this null reference exception in my logs a few times, but i don't see this log entry in the above posted logs so although it might be related i don't think it's a cause.

---

I've discovered a separate issue with slot swap as well regarding the Application Url so will need to get that fixed. It is related to this pre-warming of a site and Azure behind the scenes makes a request to the site on localhost:80 which means the ApplicationUrl is updated to use this and during pre-warming will send a keep alive request which will fail.

---

Further testing tomorrow....

[JoseMarcenaro]

@Shazwazza ,
Workaround # 1 works for us ... most of the times (!?). It fails from time to time (same exception)

Workaround # 2 ... couldn't test it, I don't know how to get past this:

Is there a way to access a class that is not public in Umbraco.Web ?
Or do I need to upgrade to 8.1.0 first? (I'm still on 8.0.2)

[KraftwerkWedl]

We are also experiencing the same error: "The process cannot access the file 'D:\local\Temp..."

Running currently Umbraco 8.0.1 on Azure. After deploying to a deployment slot (staging) the application can not start on staging because it runs into that problem, no swap included. After swap, the production runs into same problem. Restart helps to release the lock of course.

I can not add much more for now, what I wanted to test also is what @Shazwazza suggested, to not create persisted files, that would be the ideal solution for us.

But as @JoseMarcenaro mentioned, that class PublishedSnapshotService is internal as far as I see.

[Shazwazza]

Sorry folks, i didn't notice the internal bits, will have to update in 8.1.1. In the meantime, you can use this ugly reflection to disable persisted cache files inside your Compose method:

var optionsType = typeof(IPublishedSnapshotService).Assembly.GetType("Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService").GetNestedType("Options");
var ctor = optionsType.GetConstructor(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic, null, Array.Empty<Type>(), null);
var options = ctor.Invoke(Array.Empty<object>());
var ignoreLocalDbProp = optionsType.GetField("IgnoreLocalDb", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
ignoreLocalDbProp.SetValue(options, true);
composition.Register(optionsType, options);

[Shazwazza]

Good news - the latest changes I've made have resolved the issue. The issue i mentioned here https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-509966706 is the reason why the files get locked, it is a race condition when booting the nucache system when there's multiple appdomains starting up at once. The other good news is that i can confirm the above work around does work so you can use that if you want to for now - but do please test it out and report back!

With this patch, I've tested multiple times:

• Site swap = OK
• Toggling App Insights = OK
• Site swap where one slot has App Insights and not the other = OK

Have made a draft PR here https://github.com/umbraco/Umbraco-CMS/pull/5854

Turns out @zpqrtbnk mentioned this race condition in another document last week, I guess it's good that 2 people discover the same thing separate research ;)

I cannot replicate the problem locally (naturally) no matter how many appdomains i can manage to startup at the same time so it's odd that Azure is so affected by this.

Although this solves the file lock issue, it doesn't explain why the Semaphore lock logging output doesn't match up. I will investigate more of that tomorrow.

[JoseMarcenaro]

@Shazwazza - great news!
Just to be sure: the workaround we should test is the IPublishedSnapshotService / ignoreLocalDb code?
Thx

[Shazwazza]

@JoseMarcenaro indeed, that would be great.

[brreisner]

Will work with our dev team to put this fix in today and test. This is great news, really appreciate the work here @shazwazza. I know you mentioned the below:

"Don't use a persisted local cache file. The content cache is always in memory, the purpose of these persisted files is purely to for startup time performance. That said, the performance of loading in the content cache directly from the DB is also very fast, sometimes negligible. Also considering this is generally a slot swap issue or pre-warming issue, startup time isn't really a problem."

But can you think of any unintended consequences of making this change? Perhaps scenarios where it would be more impactful than others?

Very much look forward to hearing more on your findings.

[JoseMarcenaro]

@Shazwazza I tested your reflection code to ignoreLocalDb, with total success

I'm running Umbraco v8.0.2

This is the secuence I followed.
1) Re-enabled the IIS preloadEnabled setting, which I had previously set to false (Workaround #1 above) in both slots (production / stage)

2) Performed a swap - with the old code, before applying workaround #2 ==> site broke with the same error "NuCache.Content.db"

3) After stopping / restarting the site, I repeated the swap for further confirmation ==> site broke again.

4) Deployed the new code with the ignoreLocalDb setting and swap ==> site works!

5) Deployed the new code again to the other slot and swapped again ==> site works!

Regarding the performance difference, if there is any it does not affect the production site because we are using the <applicationInitialization> block in web.config, so when the swap is complete the site is up and running.

I will keep this code in place for now, and let you know if it fails at any time. Thanks!

[Shazwazza]

@brreisner regarding

But can you think of any unintended consequences of making this change? Perhaps scenarios where it would be more impactful than others?

If you are using slot swapping with applicationInitialization like @JoseMarcenaro mentions then i don't see unintended consequences of not having a persisted cache file. The only consequence outside of slot swapping is potentially slower startup but this depends largely on how much content you have. For a site with a couple thousand nodes the startup time would be probably be negligible anyways.

@JoseMarcenaro great news! In 8.1.1 I'll make these options to have a public API so the reflection code will probably stop working so just a heads up! That said, it will be easier to set the options but also 8.1.1 will have the race condition fixed anyways so you prob don't need to worry about anything anyways.

[JoseMarcenaro]

@Shazwazza , thanks for the heads up. I'm eager to test how it works in 8.1.1, meanwhile my problem is gone for now :-)

[brreisner]

@Shazwazza this has fixed our issue as well. Can you clarify on what the ultimate fix is that will be in 8.1.1 and is there any rough time frames on when this will be available? Will we be able to have a persisted cache file once the fix is in? If we can't, does this prevent the use of caching as detailed here: https://our.umbraco.com/documentation/Reference/Cache/?

You mentioned also the issue is present in v7 but not detected in the same manner. Are there fixes that will make there way into v7 or anything we need to do for v7 deployments?

Thanks for all your work to get this addressed!

[Shazwazza]

The 'ultimate fix' ( 😂 ) is that a race condition is fixed so that an appdomain that is started and immediately shutdown because another appdomain comes online moments later does not try to access the nucache persisted files.

Will we be able to have a persisted cache file once the fix is in

yes

If we can't, does this prevent the use of caching as detailed here:

This doesn't prevent you from doing anything. As mentioned before - the persisted cache __files__ are purely there for startup performance times. __All__ cache is in memory regardless of whether it loads from the DB on startup of the perstisted cache files. The usage of Umbraco is identical.

You mentioned also the issue is present in v7 but not detected in the same manner. Are there fixes that will make there way into v7 or anything we need to do for v7 deployments?

This is not in the same context as this issue. Like i said, v7 and v8 are very different. The XML cache in v7 has it's own issues which is one reason why it doesn't exist in v8. These issues have always been there - for example, there is a reason why the right click context menu in the very root 'content' node in the tree has a "refresh all" which doesn't exist in v8. It's because the v7 xml cache in some cases can be "corrupted" due to various different reasons ranging from the file not being exclusively locked and allowing multiple appdomains to read/write to it at the same time. These issues cannot be fixed in v7 - a big reason why v8 exists is so that we could fix these issues.

[Shazwazza]

The next tests that i would love for @JoseMarcenaro and @brreisner to run is to try a nightly of the 8.1.1 Umbraco.Core/Umbraco.Web DLLs. I will upload these here shortly. We would like to release an 8.1.1 asap (probably in a couple weeks) but I cannot close this issue until i know it works for you guys.

DLLs coming soon....

[JoseMarcenaro]

@Shazwazza sure, just post them here and we will test them ASAP.

[brreisner]

Will do, Thanks!

[Shazwazza]

Please see attached the 8.1.1 binaries zip file.

NOTE _These are NOT the final release binaries of 8.1.1 and they don't have a pre-release alias on the build. This build was made from rev 36030904212d321af75f96bfc894742e86629ec3_

To test:

• Take a backup of Umbraco.Core.dll and Umbraco.Web.dll and then replace these 2 DLLs with the ones in the zip file
• Remove the work around(s)
  
  
  
  • On another note - the reflection work around will no longer work with these DLLs since you don't need to use reflection anymore since we've made this class public: https://github.com/umbraco/Umbraco-CMS/pull/5924/files#diff-8ebcd4b5d06971ef8152c6628b5c9a6aR6 . The work around will still work but you don't need to use ugly reflection to do it
  
  
• Change your version in your web.config to 8.1.1 instead of 8.1.0 - or you can let the upgrader run
• Check if everything works

To rollback:

• replace these 2 DLLs with your backup copies
• change the version in your web.config back to 8.1.0
• re-instate work around

UmbracoCms.AllBinaries.8.1.1.zip

Please let me know how you go, we won't be shipping this release without knowing it works for you.

[JoseMarcenaro]

@Shazwazza - quick question
We should do this on top of an existing 8.1.0 installation, right?

The problem I have is that my production site is still 8.0.2, and I cannot upgrade it to 8.1.0 until I get issue #5886 solved (Variant Children of non-variant node are not listed) because it breaks the site in multiple places.

[Shazwazza]

@JoseMarcenaro yeah, let me make a build of 8.0.x with this fix for you to test and i'll also look into the children issue.

[Shazwazza]

@JoseMarcenaro here's an 8.0.2 build with the fix in these 2x DLLs. FYI these are Debug versions of these DLLs (not Release). I've just built this in VS based on the 8.0.2 tag and cherry picked the commit rev 36030904212d321af75f96bfc894742e86629ec3 ... i think that should work for you, please let me know

8.0.2_CoreAndWeb.zip

[JoseMarcenaro]

@Shazwazza thanks for building this DLL set for me.

Good news: On an existing 8.0.2 Azure web app, built without the reflection code, I dropped these DLLs in both slots (stage - production) and swapped back and forth without issues (no NuCache blocking)

Then I completed the setup of a 8.1.0 separate environment, and dropped the other DLL zip you provided for 8.1.1 (in both slots) and swapped back and forth - no incidents! Yay!
Of course, on the first run it upgraded the DB to 8.1.1 as expected.

It looks like the fix in the initialization code solves the lock issues. Great work!
Thanks,
Jose

[Shazwazza]

@JoseMarcenaro many thanks for testing! I'll go ahead and close this issue. @brreisner will still be great if you can let me know the outcome of your testing too. Cheers!

[warrenbuckley]

Closing this issue as this got merged in from this PR
https://github.com/umbraco/Umbraco-CMS/pull/5924

[ealse]

I have updated Umbraco from version 8.1.0 -> 8.1.1

When I was using version 8.1.0 with reflection to disable persisted cache files inside my Compose method (As mentioned above) no errors were thrown.

After updating to version 8.1.1 and removing the workaround the following error came back when swapping my web app:
Umbraco.Core.Exceptions.BootFailedException: Boot failed.
-> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process

UmbracoTraceLog.txt

[jsheard02]

@ealse @Shazwazza

We have tried this on 8.1.0, 8.1.1 and 8.1.2, and we experience the same NuCache.Content.Db error as you point out. For some additional context, we add the umbracoLocalTempStorage into the Application Settings of the Web App on azure, and not directly into the web app itself. I will try and add this directly into the web.config file now, and let you know if the issue persists.

[Shazwazza]

@jsheard02 umbracoLocalTempStorage isn't the correct setting, that is for v7, see https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-476477802

[anthonydotnet]

This doesnt seem to work in v8.1.2. We still need to restart the app after a slot swap :(

[JoseMarcenaro]

We have three similar Azure environments with v8.1.1
In one of them we get the error every time - the other two are good. And I can't figure out what the difference is. Any suggestions for further testing?

[nul800sebastiaan]

This sounds like a configuration problem, at least for @JoseMarcenaro - it might be good to triple-check all differences between those 3 servers.

Other than that, if it can be reproduced with a clean install then we can have a look at additional fixes.

[kjac]

@nul800sebastiaan This also happened to me with 8.1.2 when using deployment slots on Azure. Here are my findings.

Symptoms

Whenever a deployment slot swap was executed, the target (production) slot would crash with the The process cannot access the file error. The source slot would still boot fine when requested after the swap.

This happened with manual swaps from the Azure portal as well as with automated ones from DevOps.

Workarounds

Setting Umbraco.Core.LocalTempStorage = EnvironmentTemp did not help. Only difference was a new file path in the The process cannot access the file error (as one would expect given the new LocalTempStorage setting value).

However! Applying @Shazwazza's fix did the trick. The code looks like this in 8.1.2:

using Umbraco.Core.Composing;
using Umbraco.Web.PublishedCache.NuCache;

namespace Your.Namespace.Here
{
    public class IgnoreLocalDbComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Register(factory => new PublishedSnapshotServiceOptions
            {
                IgnoreLocalDb = true
            });
        }
    }
}

Since we can do warm-ups of deployment slots before swapping, any startup performance impact is of no concern.

[Shazwazza]

My 'fix' will be fine if that's what you choose to use but i recommend reading this whole thread to understand. Setting IgnoreLocalDb = true just means there is no persisted cache file on disk which can affect startup time, but with slot swapping you probably don't need to worry about that either.

What I would like to understand though is what the actual new problem is. It's unfortunate when people say things like "the problem still exists" or "I have the same problem" which isn't entirely helpful since the first problem was definitely fixed and was caused by a race condition. As it seems that there is a new/similar issue then please open up a new issue with steps to replicate and full details. Thanks!

[brreisner]

I can confirm that IgnoreLocalDb = true avoids the issue for us. I will say that we have experienced the issue without slot swapping, even occuring (rarely) when we deploying directly to the primary slot of our dev/qa instances. Will try to identify the issue further but it is difficult to track down.

[Shazwazza]

There's 2 workarounds - one is IgnoreLocalDb, the other is to set WEBSITE_DISABLE_OVERLAPPED_RECYCLING in appSettings to `, see https://github.com/umbraco/Umbraco-CMS/issues/6546#issuecomment-545224926

I will update the hosting on azure docs to include this setting, this is preferable to use over IgnoreLocalDb if you aren't using slot swapping.

[Shazwazza]

docs PR https://github.com/umbraco/UmbracoDocs/pull/2042

[stodolos]

Is there an equivalent for PublishedSnapshotServiceOptions in v8.0.2? We will upgrade to the latest soon, but we have to do some testing for that first.

Update: I just looked at the code, and it looks like you can't. Plus, this version of PublishedSnapshotService does not have the same locking, but please let me know if there is a way to do this in 8.0.2.

Thanks

[deMD]

Time to reopen this issue. I get it on Umbraco 8.4 with the following message:

Exception Details: Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\local\Temp\UmbracoData\2d6f8e550e7c1266526ef85adfa0edb1\NuCache\NuCache.Content.db' because it is being used by another process.
  at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
  at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
  at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
  at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
  at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
  at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\DataSource\BTree.cs:line 27
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.MainDomRegister() in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 175
  at Umbraco.Core.MainDom.Register(Action install, Action release, Int32 weight) in D:\a\1\s\src\Umbraco.Core\MainDom.cs:line 102
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(PublishedSnapshotServiceOptions options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IProfilingLogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 124
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__0(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4646
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItem(IFactory factory, Type itemType) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 33
  at Umbraco.Core.Composing.CollectionBuilderBase`3.<>c__DisplayClass10_0.<CreateItems>b__0(Type x) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 100
  at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
  at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
  at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 99
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 25
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateCollection(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 117
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 172
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory) in D:\a\1\s\src\Umbraco.Core\FactoryExtensions.cs:line 22
  at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\a\1\s\src\Umbraco.Core\Runtime\CoreRuntime.cs:line 158

I have the following keys in my appsettings:

    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

And I have the workaround still in the application:

public class UmbracoPersistedLocalCacheBugWorkaroundComposer
    {
        public void Compose(Composition composition)
        {
            composition.Register(factory => new PublishedSnapshotServiceOptions { IgnoreLocalDb = true });
        }
    }

I was also able to make it reproduceable for better or worse. The issue always occurs when I manually update the configuration is Azure.

[Shazwazza]

@deMD so to confirm - if you setup a vanilla 8.4 site on azure with

    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

you can replicate this problem just by changing an app setting?

[deMD]

@Shazwazza I have not tried it with a vanilla 8.4 yet, the version I have been able to replicate it with was an updated Umbraco install. I'll see what a vanilla instance does.

[kjac]

@deMD I just upgraded a rather complex client site from 8.1.4 to 8.4 in Azure and I can't seem to replicate.

The site has previously experienced this issue. To solve it we added this to the app settings:

<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory,Examine" />
<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />

...and this in a composition:

composition.Register(factory => new PublishedSnapshotServiceOptions
{
    IgnoreLocalDb = true
});

I'm pretty sure the Umbraco.Examine.LuceneDirectoryFactory app setting has nothing to do with the error, but it's included here for good measure - it should be there anyway on Azure.

[JoseMarcenaro]

@deMD,
If you want to use the IgnoreLocalDb workaround your class UmbracoPersistedLocalCacheBugWorkaroundComposer should implement IUserComposer as described here in order to be picked up by the initialization pipeline.

[deMD]

@deMD,
If you want to use the IgnoreLocalDb workaround your class UmbracoPersistedLocalCacheBugWorkaroundComposer should implement IUserComposer as described here in order to be picked up by the initialization pipeline.

Wow good catch, it seems that indeed our other dev forget that and I completely missed it. Though you would think that workaround shouldn't be necessary anymore, with all the other changes.

@kjac I have the following settings in my app settings:

 <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />

So even without the workaround I would've expected this to work. If the issue is caused by something else (I'll see what an empty clean install of Umbraco does on the same azure resource) then we'll need to dig down into the specific configuration of the Azure environment to see if there is anything there that might cause it.

[JoseMarcenaro]

@deMD you are right - it should work without the IgnoreLocalDB. However, in several Azure / Umbraco 8.3 installations I still got the same error in the past few months so I'm routinely using the workaround.

[deMD]

@Shazwazza I created an empty Umbraco 8.4 website and released it to azure, and I was able to replicate the issue when having added the keys.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\local\Temp\UmbracoData\2c21e6f8b472ef713ed2c473e5aee301\NuCache\NuCache.Content.db' because it is being used by another process.
  at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
  at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
  at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
  at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
  at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
  at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)
  at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
  at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\DataSource\BTree.cs:line 27
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.MainDomRegister() in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 175
  at Umbraco.Core.MainDom.Register(Action install, Action release, Int32 weight) in D:\a\1\s\src\Umbraco.Core\MainDom.cs:line 102
  at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(PublishedSnapshotServiceOptions options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IProfilingLogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders) in D:\a\1\s\src\Umbraco.Web\PublishedCache\NuCache\PublishedSnapshotService.cs:line 124
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__2(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.EmitConstructorDependency(IEmitter emitter, Dependency dependency) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4158
  at LightInject.ServiceContainer.EmitConstructorDependencies(ConstructionInfo constructionInfo, IEmitter emitter, Action`1 decoratorTargetEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4120
  at LightInject.ServiceContainer.EmitNewInstanceUsingImplementingType(IEmitter emitter, ConstructionInfo constructionInfo, Action`1 decoratorTargetEmitMethod) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4080
  at LightInject.ServiceContainer.EmitNewInstance(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4034
  at LightInject.ServiceContainer.EmitNewInstanceWithDecorators(ServiceRegistration serviceRegistration, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3929
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__0(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4646
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItem(IFactory factory, Type itemType) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 33
  at Umbraco.Core.Composing.CollectionBuilderBase`3.<>c__DisplayClass10_0.<CreateItems>b__0(Type x) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 100
  at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
  at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
  at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 99
  at Umbraco.Core.Composing.ComponentCollectionBuilder.CreateItems(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\ComponentCollectionBuilder.cs:line 25
  at Umbraco.Core.Composing.CollectionBuilderBase`3.CreateCollection(IFactory factory) in D:\a\1\s\src\Umbraco.Core\Composing\CollectionBuilderBase.cs:line 117
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 172
  at DynamicMethod(Object[] )
  at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
  at LightInject.ServiceContainer.<>c__DisplayClass198_0.<EmitLifetime>b__1() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4657
  at LightInject.PerContainerLifetime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6169
  at LightInject.ServiceContainer.EmitLifetime(ServiceRegistration serviceRegistration, Action`1 emitMethod, IEmitter emitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4656
  at LightInject.ServiceContainer.<>c__DisplayClass197_0.<ResolveEmitMethod>b__1(IEmitter methodSkeleton) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4649
  at LightInject.ServiceContainer.<>c__DisplayClass153_0.<CreateEmitMethodWrapper>b__0(IEmitter ms) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3856
  at LightInject.ServiceContainer.CreateDynamicMethodDelegate(Action`1 serviceEmitter) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3776
  at LightInject.ServiceContainer.CreateDelegate(Type serviceType, String serviceName, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4743
  at LightInject.ServiceContainer.CreateDefaultDelegate(Type serviceType, Boolean throwError) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 4705
  at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3437
  at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type) in D:\a\1\s\src\Umbraco.Core\Composing\LightInject\LightInjectContainer.cs:line 111
  at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory) in D:\a\1\s\src\Umbraco.Core\FactoryExtensions.cs:line 22
  at Umbraco.Core.Runtime.CoreRuntime.Boot(IRegister register, DisposableTimer timer) in D:\a\1\s\src\Umbraco.Core\Runtime\CoreRuntime.cs:line 158

Here is the ARM template of the azure environment I used: ExportedTemplate-UMBU-REGR-WEU.zip

As for the source I just created an umbraco website without a template, added the keys, and deployed it to Azure.

Then I went into the Azure portal to the Web App, went into the configuration section and added a new app setting and saved. Then it took ~30 seconds to 1 minute for the application to update and crash.

[Shazwazza]

Oh joy, the delights of working with Azure's custom kernels that aren't documented and don't abide by the normal .NET rules. I must have tested this countless times last time we discovered the WEBSITE_DISABLE_OVERLAPPED_RECYCLING setting, looks like I'll need to spend more time re-testing this.

In the meantime, there is some good news on the horizon - I've been working on a new implementation of IMainDom and we've created a new abstraction for this too that can more easily be replaced than the entire MainDom. The PR is here https://github.com/umbraco/Umbraco-CMS/pull/7306 and load testing is still pending on this along with more Azure testing. I'll follow up with more info about this later.

In the meantime, I suggest just using IgnoreLocalDb = true until the new MainDom testing is done and until I have time to re-test why the WEBSITE_DISABLE_OVERLAPPED_RECYCLING is not working again

[vaags]

I can confirm that this is happening in v 8.4 with the keys added. Could this issue be reopened?

[Shazwazza]

As mentioned above we are working on a real solution, not a work around. For now please use IgnoreLocalDb.

I would like to ask if anyone can test if this issue occurs in the latest 8.3.x? - perhaps there's some strange regression in 8.4.

[JoseMarcenaro]

@Shazwazza - not sure about 8.3.0 (I didn't test it) but I did get the error on 8.2.1.
As always, the IgnoreLocalDb workaround "fixed" the problem.

[Shazwazza]

I've been testing on Azure with the latest 8.6 codebase. This build includes many of the changes completed and contributed to by @John-Blair which greatly improves the handling of the shutdown sequence along with more robust handling of resource disposal. It also includes a new optional IMainDom implementation that doesn't rely on Semaphores.

But I cannot reproduce this locking issue with this version even with the Semaphore lock!

I have these settings:

• WEBSITE_DISABLE_OVERLAPPED_RECYCLING = 1
• Umbraco.Core.LocalTempStorage = "EnvironmentTemp"
• Umbraco.Examine.LuceneDirectoryFactory = "Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine"

Although i still cannot replicate it even without WEBSITE_DISABLE_OVERLAPPED_RECYCLING.

If you would like to test this out, here's a zip https://umbraco.visualstudio.com/_apis/resources/Containers/7645011?itemPath=zips%2FUmbracoCms.8.6.0-alpha.7394.zip (I'm hoping that zip is available to any microsoft login but let me know if not... it's too big for GH)

I've also tested the other various ways to replicate this issue - togging on/off app insights, adding/removing/changing appSettings via the portal and I cannot get this issue to occur with this build.

To me this indicates that the shutdown changes introduced are very beneficial and we no longer trigger race conditions so we don't see the problem. That said, this doesn't 'fix' the problem because the Semaphore issue is still there and there will be cases where the race condition is triggered.

I've also been testing out the non-semaphore locking mechanism and you can with this build as well by adding this setting:

• Umbraco.Core.MainDom.Lock = "SqlMainDomLock"

This will incur a very simple/fast DB query each second but this shouldn't affect any performance.

If you want to see detailed log outputs for this mechanism add this to your serilog.config file:

<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.MainDom" value="Debug" />
<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.SqlMainDomLock" value="Debug" />

[deMD]

Hey @Shazwazza just came back from vacation so I had no earlier opportunity to check the link out, but sadly it does not appears to be avaiable from my MS login.

[Shazwazza]

@deMD no worries, I've been working on getting to the bottom of this and it turns out that 8.4 has unfortunately included a deadlock issue which I believe is the cause for this.

The issue is here https://github.com/umbraco/Umbraco-CMS/issues/7404

There is a PR now and we'll get an 8.4.1 released with that patched and see how it goes. In the meantime I'm still testing the 8.6 changes with the Sql locking mechanism.

[DanDiplo]

I just encountered this issue on Umbraco 8.4 switching an Azure site from default HTTP/1 to use HTTP/2. Brought down the whole site because the NuCache.Content.db was locked by another process. Just adding this as it might be a useful way of testing this issue.

[Shazwazza]

@DanDiplo yes please try 8.4.1 which contains the #7404 deadlock issue fix https://our.umbraco.com/download/releases/841 also be sure your config is correct.

[danielibanez]

I'm using 8.4.1 and just started getting this issue last night, 02/24/20. Will try upgrading and IgnoreLocalDb = true

Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file D:\home\site\wwwroot\App_Data\TEMP\NuCacheNuCache.Content.db because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

[dalemccutcheon]

Hey @Shazwazza we are still seeing the issue happening on 8.5.2 at the moment. Any suggestions?

[Shazwazza]

@dalemccutcheon i can only really try to help if you have steps to replicate. I don't know your setup but folks tend to miss a configuration step or be doing something outside of the norm that can be part of the problem. Else you can try 8.6 RC (or wait until final release) with this setting https://github.com/umbraco/UmbracoDocs/pull/2330 which is there to be an official working setting for Umbraco on Azure websites.

[danielibanez]

@dalemccutcheon I had this issue pop up on 8.4.1 and 8.6.0-rc. Specifically, I noticed it seemed to happen after switching slots in Azure. But I also had my web.config misconfigured. Umbraco has an Azure guide that, imo, is not very clear:

https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/azure-web-apps

Anyone who knows otherwise please correct me, but I think as far as configuration goes just make sure these three lines are in your web.config.

<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp"/>
<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine"/>
<add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1"/>

I am still on 8.6.0-rc, now with a hopefully properly configured web.config. I will be doing more slot switches and daily log checking.

[Shazwazza]

If you are on 8.6 rc please follow the docs i posted above, here's the link: https://github.com/umbraco/UmbracoDocs/pull/2330, this PR is to update the docs, currently the page will look like this for 8.6 https://github.com/umbraco/UmbracoDocs/blob/4fc755ea1eb67a616e3268c8777fc106a3cb8128/Getting-Started/Setup/Server-Setup/azure-web-apps.md

you need

<add
    key="Umbraco.Core.MainDom.Lock"
    value="SqlMainDomLock" />

[yaplex]

hi, I have the same issue with Umbraco 8.5.3 on Azure and I have the following in web.config

    <!-- Azure hosting settings -->
    <add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
    <add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
    <add key="WEBSITE_DISABLE_OVERLAPPED_RECYCLING" value="1" />
    <!-- End Azure hosting settings -->

Is it any logs or something I can provide to help fix this?

I reading it is because Azure doing something with a website. I have multiple 8.5.x websites, and they are on the same application. The issue happening for all of them at the same time, so this theory seems correct.

[dalemccutcheon]

Hey,

I was still getting the issue however i have added this :

[RuntimeLevel(MinLevel = RuntimeLevel.Run)]
public class UmbracoCacheWorkaround : IUserComposer
{
public void Compose(Composition composition)
{
composition.Register(factory => new PublishedSnapshotServiceOptions { IgnoreLocalDb = true });
}
}

I have not experienced the issue since added the composer.

Cheers
Dale

[Shazwazza]

yes the IgnoreLocalDb = true will always work, the caveat is that you will incur a DB performance penalty on startup since the in-memory caches will be rebuilt from the DB instead of the persisted nucache file. This will be fine in most cases but if you have an Umbraco site with tons of content it might not be ideal.

for 8.6 please try https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-599546750

[OwainJ]

@Shazwazza

Apologies for bumping this thread but we're still experiencing this issue on Umbraco 8.8.0,
Umbraco.Core.Exceptions.BootFailedException: Boot failed.\n\n-> System.IO.IOException: The process cannot access the file 'C:\\local\\Temp\\UmbracoData\\a30e1f44822dd8ffa5319bc32ef15cb6\\NuCache\\NuCache.Content.db' because it is being used by another process

We have the recommended Azure App Settings in our web.config:
<add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" />
<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />

Our app is load balanced with the Azure Application Gateway, we have 2 front-end web apps and 1 back-office/master web app with explicit master scheduling set on boot as described here: Explicit Master Scheduling Server Docs

The IgnoreLocalDb = true method works, but as our site has over 250k Media nodes, we can't use that method because it causes our boot time to be around 30 mins. We only removed this today and then got the NuCache lock issue.

Also, is the WEBSITE_DISABLE_OVERLAPPED_RECYCLING setting still required? As this documentation doesn't mention if it is still needed in Umbraco v8.6.4+. We don't currently have it set, but I'll add it and report back if it changes anything.

[JoseMarcenaro]

@OwainJ - just in case, if you do set the WEBSITE_DISABLE_OVERLAPPED_RECYCLING flag, remember to do it on Azure configuartion and not in the web.config - it is ignored there.

[deMD]

Also set the WEBSITE_DISABLE_OVERLAPPED_RECYCLING flag on the Master server if load balancing, even though not mentioned on the documentation we found errors are still caused if not setting that flag on Master environments.

[OwainJ]

@JoseMarcenaro @deMD Thank you both 🙂 I'll keep that in mind.