Aspnetcore: IIS -> Lock .NET Core application DLL file

[radenkozec]

When I try to overwrite .NET Core application DLL file using FTP on production IIS server file is locked and it cannot be overwritten.

To deploy new version I need to stop application in IIS to release lock, then overwrite.
It is not possible to deploy without stopping application.

[Tratcher]

See app_offline.htm: https://docs.asp.net/en/latest/hosting/aspnet-core-module.html#asp-net-core-module-app-offline-htm

[guardrex]

If you end up desiring a PowerShell way of doing it, you can use the IIS Admin cmdlets: https://technet.microsoft.com/en-us/library/ee790599.aspx

Stop-WebAppPool -Name $appPoolName

... deploy ...

Start-WebAppPool -Name $appPoolName

[radenkozec]

Thanks for explanation @Tratcher .
I am not sure if this is on your plans but this worked for previous ASP.NET MVC I think?
Do you plan to implement this feature or not?

@GuardRex I cannot do it because it is shared hosting enviroment and don't have permission to stop AppPool

[chuchuva]

Can you make it to 'just work' with msdeploy.exe and Azure? If I understand correctly the web site needs to be restarted to prevent file locking. -enableRule:AppOffline works but the whole website is offline for few minutes which is not a great user experience especially given that we deploy few times per day.

See also http://stackoverflow.com/q/40276582/14131

[Tratcher]

@chuchuva maybe, but all magic comes with a cost. Previous versions of ASP.NET did some complicated shadow copying to work around the file locking problems.

[HarelM]

Magic or not, It worked well. I kinda miss it now after the migration to ASP.NET core...

[JaCraig]

Agreed with @HarelM. We've run into issues with this from our automated deployments to just end user experience. We've gone from deploying about 10 times a day with the old MVC to maybe a daily deployment at night and accepting that the people using the Core app are going to be annoyed when we take it offline. While not a show stopper, it's added friction towards Core adoption.

[cyriltovena]

👍

Not supporting overlapped recycling is a regression.

[deadlydog]

Are there plans to revisit this feature and put it on the road map? It's very inconvenient/unfriendly to require everybody deploying a .Net Core website to manually implement a staging-slot strategy if they want to support zero downtime deployments.

Having this feature would make the transition to .Net Core websites much smoother for many people, and would allow for faster adoption of .Net Core websites.

[ajeckmans]

We would like to know as well. Putting a app_offline.htm inside the applications directory does not work.

[DaleMckeown]

I've only just realised this 'feature' after moving a number of sites over to AspNetCore. I can't believe it's deemed to be acceptable to take your sites offline for a few minutes every time you want to publish!

This is bad enough for me as a leader of a small team - are those practicing continuous integration on a large scale shunning AspNetCore? There is no way they would be able to have their site going offline for minutes every hour to republish!

[davidfowl]

Are you deploying use FTP or xcopying? Or are you using webdeploy?

[DaleMckeown]

I'm publishing via webdeploy to IIS.

[ajeckmans]

We are currently working around this issue by just deleting the web.config first (effectively killing the application), but this is not really an acceptable solution in the long run.

@DaleMckeown ideally with a continuous integration workflow you'd have multiple servers behind a load balancer. You'd then pull one server, update it, put it back en move on to the next. Ofcourse this is not always possible (like in our case), so you'll have to live with a few minutes downtime. In our case the app is back up within 30s, so this is not really an issue.

[davidfowl]

@DaleMckeown

I'm publishing via webdeploy to IIS.

There are a few options that can be used to make deployment using webdeploy work well (it supports renaming locked files and dropping app offline automatically). Is that the case by default @shirhatti ?

@ajeckmans

We are currently working around this issue by just deleting the web.config first (effectively killing the application), but this is not really an acceptable solution in the long run.

Does that mean you're doing xcopy deployment?

[DaleMckeown]

There are a few options that can be used to make deployment using webdeploy work well (it supports renaming locked files and dropping app offline automatically). Is that the case by default @shirhatti ?

Thanks David - sounds better than what I am currently doing (manually stopping the site & app pool in IIS). Can you point to some resources so I can investigate the implications of these approaches?

[davidfowl]

@DaleMckeown some info untilI find the source of truth https://github.com/Microsoft/vsts-tasks/issues/5259#issuecomment-346202503

[ajeckmans]

@davidfowl We're using webdeploy to deploy to our testserver (which btw never gives us problems), from there we copy the files to our live servers using robocopy

[guardrex]

@ajeckmans

We are currently working around this issue by just deleting the web.config first

When the web.config is removed from the deployment, IIS will serve sensitive files from the deployment. An attacker could simply request files continuously day and night until your 30s window opens up.

@DaleMckeown

Following @ajeckmans advice if you ever want to get into the weeds a bit with a PS approach to do that, you can script the process to drop AppPools one at a time for deployment across a web farm. For an example of how that can be done, see https://github.com/guardrex/aspnetcore-iis-ps-publish ... but just look at that as an experimental example and not a production-quality script. I haven't played with that for a while, but it should (in theory) still work.

[ajeckmans]

@guardrex you're right. We copy an app_offline.htm to the dir first, then we remive the web.config, copy over the application, put back the web.config and remove the app_offline (all with a script ofc). Unfortunately just placing the app_offline file in the websites directory does not break the lock on the dlls. We need to remove the web.config, an action we don't need to do for the older asp.net full apps (like old webforms etc)

[guardrex]

@ajeckmans That shouldn't work. When the web.config file is removed, IIS picks that change up immediately, so it should default requests over to the Static File Module and start serving files. Try it and see if that's what happens ...

1. Add app_offline.htm.
2. Confirm that the site serves app_offline.htm.
3. Pull the web.config out.
4. Request a sensitive file (e.g., http://localhost:<PORT>/<ASSEMBLY_NAME>.deps.json ... substituting the PORT and the ASSEMBLY_NAME).
5. You should get the deps.json file served if your Static File Module is functioning correctly.

I wouldn't trust merely removing module with ...

<configuration> 
 <system.webServer> 
   <modules> 
     <remove name="StaticFileModule" /> 
   </modules> 
 </system.webServer> 
</configuration>

... since other modules would remain and possibly present other attack vectors. Maybe one could remove all of the non-essential modules, but we're getting into uncharted waters with a move like that just to cover a deployment via removal of web.config. That's never been an option under the official guidance, and so it's completely unsupported. I recommend PS scripting, for example, to take the AppPool down, or some other strategy.

[guardrex]

I guess I should add a note of sympathy to that ... it did raise a few eyebrows from a security perspective. When that deployment layout change was made, it was discussed, including putting the file back into wwwroot, see ...

Discussion for: Publish for IIS changes to web.config location (IISIntegration 158)
Head-check on moving web.config back to wwwroot (IISIntegration 164)

[EDIT] Perhaps that's your (unsupported) workaround: Move the web.config back to wwwroot, then go ahead with what ur doing. Still tho ... kind'a scary breaking the app to get it offline like that.

[DaleMckeown]

This leaves me even more confused about what we should be doing in this situation.

What's the current recommendation for publishing AspNetCore sites to IIS via web deploy in a way that can avoid file locks?

[JasonRodman]

I would like to know as well. This is becoming a major roadblock to .net core adoption at my organization.

[shahjay748]

When my .net core web application is running and I am trying to publish a new build, it gives me error that DLLs are in use. So I have to Stop the app pool, then update the DLLs and start the App Pool.

Is there any workaround in .Net Core by which I can publish the new Build/DLLs without stopping the App Pool ?

This is by default supported in .Net framework using shadow copy mechanism. Unfortunately, It's been 2 years since the launch of .Net core and seems like there's still no support for such very basic and required functionality. Is there even any plans to fix this in future ?

Thanks in advance for any help.

[ajeckmans]

@guardrex it is indeed scary to break the app to get IIS (or whatever is holding the lock) to release the dll's, however it is even scarier to not be able to push hotfixes to a product. For the time being breaking the app is the only thing we can do, but we're ofc looking into other more modern approaches to handling this :)

@shahjay748 don't hold your breath. If I were te redo the process here, I'd put the app inside a container and just put up a new container and switch traffic and pull the old version down (or something like that). It seems like the more sane thing to do and seeing as .net core is all about new modern ways, just pushing a new version of the app by simply copying over the new files is considered (and I agree) a bit of an arcane way to do it.
Which ofc doesn't help us who are, for the time being, stuck with old processes :)

[bpetersen1]

What i do (Not sure if it's a proper way of doing it), is upload the site to another directory and switch paths in iis

[jkotalik]

We did some internal investigation on this issue. From what I can tell, ANCM does not hold any files/handles in the deployed application. It seems to be an issue with webdeploy itself, and it flakily fails deployment. I never had it consistently fail to deploy the application after retrying a few times once app_offline was dropped.

[guardrex]

Adding a quick PowerShell side note to that: The lock on the app's assembly is always released ... I've never had a problem (yet! lol).

@bpetersen1 ... Your "staging" approach has the sweet side effect of offering a quick rollback option if the new deployment flakes out. That should be easy to script if needed.

[HarelM]

This is still externally annoying... we have decided to look into a docker solution in order to leave IIS behind.

[davidfowl]

I'm going to experiment with a solution. Stay tuned.

[niico100]

I'm also having this issue using FTP.

What is Microsoft's advice on this?!

[davidfowl]

I'm also having this issue using FTP.

What is Microsoft's advice on this?!

Drop an appoffline.htm file, then ftp then remove it.

[benzhi2011]

@davidfowl Yes,that is what i am doing right now with a batch file. It takes few seconds for IIS to release the lock and then copy the files over.

[niico100]

Thanks - out of interest how are you running the batch file?

Also a link to MIcrosoft's documentation on this appreciated; I can't find it on google. Thanks.

[benzhi2011]

@niico100 Basically, the batch file just copy the files to the project directory, and I make a backup folder first before doing that. Before copy the files, put the appoffline.htm to the project folder.
Cause the file locking problem, the copying will make a retry. I am using robcopy
https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy

[HarelM]

If anyone is interested, the following script downloads deployment file from appveyor, stops the site, copies the required files and brings the site back up:
https://github.com/IsraelHikingMap/Site/blob/master/Scripts/Deploy.ps1
It requires to be ran as administrator though. hopefully docker will solve all this non-sense once we migrate...
it uses Stop-WebAppPool and Start-WebAppPool powershell commands.

[mscrivo]

For local development, WebDeploy didn't work for me. I tried using it against my local IIS and it would still complain of locked files. I then tried using the above powershell commandlets from VS Pre/Post build events, but that didn't work for me either, probably because 32-bit powershell can't modify 64-bit IIS settings? Anyhow, what does seem to work nicely are the following pre/post build events in my ASP.Net Core project:

Pre-Build Event:
echo "App Offline" /a > $(ProjectDir)app_offline.htm

Post-Build Event:
del $(ProjectDir)app_offline.htm

[ronnyek]

I briefly perused complains here, and it seems like this may be related. I moved CI deployments for us from octopus deploy which generally "just worked", to usuing azure devops releases with the iis management tasks, and still fairly often have problems deploying new files because even after stopping webapp, stopping app domain, files are still in use.

This is probably at least 1 out of every 3 deploys and haven't found anything that seems to make the situation better, or deploys more reliable.

[kanadaj]

@Tratcher @ronnyek This seems to be related to Application Insights; after adding the App Insights extension, my web app DLL is suddenly very hard to update, no problems before then.

This would make sense given that App Insights has to connect the profiler to the app process and DLLs.

[dhtek]

I'm still facing to this problem. I use the flag -enableRule:AppOffline, sometimes it's working well but most of the times deployment is failing saing "ERROR_FILE_IN_USE".

After starting a release, I monitor the web application folder and see the App_Offline.html dropped by msdeploy. Next the release failed with the above message and note the App_Offline file still in the folder, I start the release again, this times it's working well as the module see the file in the folder.

last asp.net core runtime installed -> aspnetcore.dll 12.1.18263.2

Any ideas ? Still a bug ?

[poke]

@dhtek If you try to replace the files immediately after placing the app_offline.htm, then the server simply didn't have enough time yet to shut down the application. You should wait a bit before attempting to replace the files.

[dhtek]

Ok I understand but I'm using the -enableRule:AppOffline option from msdeploy and the page is automatically dropped it to the folder and the sync is done straight away. I do nothing myself.

[DM2489]

I'm still seeing this issue too. My project files are referencing Microsoft.NET.Sdk.Web, but it is not dropping an app_offline.htm file automatically, as stated it should in the docs.

[markuswolters]

Hi guys. This annoying issue is still present with Core 2.1 with the lastest VS 2017. I use appoffline, still the main project dll is always is use and requires a complete stop of website during publish. This renders web deploy almost useless.

[CJHarmath]

If you can't take your host offline behind the loadbalancer and would want to do it quicker, here is one way with symlinks and some powershell.
I was thinking of doing something like this.

It's quicker as you don't have to stop the app pool during the copy + recycle is better than a hard stop as it lets the current requests complete, so less downtime.

# Setup
Import-Module WebAdministration
# create 2 site root directories
$a = 'C:\inetpub\AspNetCoreSampleA'
$b = 'C:\inetpub\AspNetCoreSampleB'
$siteRoot = 'C:\inetpub\aspnetcoresample'
$siteName = 'AspNetCoreSample'
$poolName = "aspnetcore"
New-Item -Type Directory $a
New-Item -Type Directory $b
# create a symlink to targeting one side
New-Item -Type SymbolicLink -Path $siteRoot -Target $a
# point the site root to the symlink
Set-ItemProperty "IIS:\Sites\$siteName" -name physicalPath -value $siteRoot
# make sure it get's picked up
Restart-WebAppPool -Name $poolName

# this tells you the active side
Get-Item -Path $siteRoot | Select-Object -ExpandProperty target

# Flip the symlink
$current = (Get-Item -Path $siteRoot).Target
$newTarget = if ($current -eq $a) {$b} else {$a}
New-Item -Type SymbolicLink -Path $siteRoot -Target $newTarget -Force
# at this point w3wp.exe still locks the current target folder until it's getting recycled
# Deploy new version to the symlink which is now pointing to the other side which should have no locks
robocopy \\myshare\myapp $siteRoot /mir
# recycle app pool, so it picks up the new files
Restart-WebAppPool -Name $poolName

# bonus point: rollback is easy
$current = (Get-Item -Path $siteRoot).Target
$newTarget = if ($current -eq $a) {$b} else {$a}
New-Item -Type SymbolicLink -Path $siteRoot -Target $newTarget -Force
Restart-WebAppPool -Name $poolName

Here is the gist
https://gist.github.com/csharmath/b2af0f50700ce9fbdd8c5c3e582fd41b

[CJHarmath]

Restart-WebAppPool is basically recycle, which is useful if you have the overlapped recycle enabled ( the default) as it will spawn up a new w3wp.exe and all new requests will be served by that new process while the currently executing ones will be completed by the old w3wp.exe.
This way they are overlapping until the old is done with the requests, the you end up with a single w3wp.exe pointing to the new version and there is no locking issue.

This is kind of similar to shadow copying, but not 100% as that provides a way much nicer and seamless xcopy scenario.

So far it seems to be the best approach I can think of if you would need to stay online during a release.

[kanadaj]

@csharmath or there is also docker I guess. No file locks that way, and it can do rolling updates via swarm/k8s/okd

[CJHarmath]

Sure, that's even better of course. sigh, not every place has that yet.
For the rest of the oldschool setups you need to yakshave it until it works for you with no or minimal downtime.

[kanadaj]

@csharmath Agreed, but at this point, given that this has been an unfixed issue for _years_, might as well adapt and go after solutions that work instead of waiting for a miracle...

[dev8546]

I have used below power shell script and it is crashed the IIS and even affected the other application pool and I have to restart the machine to get everything back... this process I have got it from the official document and it was disaster , so my advice is DO NOT USE THE app_offline.htm better to use the script which stops the app pool and start it again, one I found in this thread https://github.com/IsraelHikingMap/Site/blob/master/Scripts/Deploy.ps1

$pathToApp = 'G:\prod_web_core'
$pathToAppOfflineHtml = 'G:\prod_web_core\app_offline.htm'
# Stop the AppPool 
New-Item -Path $pathToApp -Name app_offline.htm
# Provide script commands here to deploy the app
Copy-Item "G:\prod_web_core_temp\*" -Destination $pathToApp -Recurse -Force
# Restart the AppPool 
Remove-Item -Path $pathToAppOfflineHtml
Get-ChildItem -Path "G:\prod_web_core_temp" -Recurse | Remove-Item -Force

[xsoheilalizadeh]

I'm force to stop AppPool for success deploy, any progress for fixing this issue?!

[macfarmw]

We just encountered this file locking issue for the first time today since dotnet core is new for us. I was really caught off guard by this. It's funny that the number 1 reason I switched from Classic ASP to .Net Framework while it was still in beta (~2001) is because I could do hot deployments with no files locking. It was awesome! Almost 18 years later and now I'm right back where I started. Oh well, you win some you lose some.

[chrdlx]

I've just referenced my issue... in my case deploying Blazor Server Side solutions....
A couple of days ago it was a nightmare... I applied the app_offline.htm, shut down app pools, and even restarted whole IIS instances to unlock files. The next step would've been to restart the entire server, luckily after 5 minutes the .dlls unlocked themselves.

[Portekoi]

2019, same problem for me... and this issue was created in 2016... still no solution?

[bladefist]

2020, same problem for me. Biggest regression ever. .Net 4.6 you replace the dll files and the app reloads on the new version. .Net core locks everything.

[kanadaj]

@bladefist Your best bet as of today is to use Docker containers :/

[bladefist]

@kanadaj thanks but Dockers is another tier of headaches to deal with. I eventually figured out a work around which was to use App_offline.htm and then wait for a minute and hope the files are free. 99% of the time they are, you just have to wait, i guess my app takes a while to power down.

Gone are the days of .net 4.6 where you just copied your dlls over and your app auto recycled. I guess that was too simple. What doubly frustrating is this is not an issue on linux. You can replace files in use in linux.

[kanadaj]

Well yes I guess, but that method isn't very straightforward in a production environment. Without HA your app will end up not responding for a minute, and with HA you get some random bad requests until the load balancer figures out the app is down... Unless you manually cycle the hosts in your load balancer. But that sounds like even more trouble to set up.

[bladefist]

@kanadaj yep, we use a load balancer and it linearly updates the servers. No downtime.

[davidglassborow]

We have had success with

• Use of a symbolic link for the app directory
• Deploy a new version of the site to a different directory
• Update the symbolic link
• Restart the app pool using the appcmd utility that comes with IIS.

[bladefist]

@davidglassborow that's a creative solution. Thanks. Creative solutions shouldn't have to be needed though.

[davidglassborow]

In my experience, the old mechanism wasn't reliable when the site was under load anyway, we ended up putting reverse proxies anyway for high traffic sites.

[davidfowl]

Right, in-place deployments were never reliable or atomic, if you got lucky or didn't have much traffic it would be fine though. I'm assuming the majority of complains are around those scenarios. We will look into improving things in the .NET 5.0 time frame. Some caveats:

• We will look into making things better for framework dependent applications. If you deploy self contained then you're out of luck. It'll lock both native and managed files in the output folder. The locking native dlls problem has always existed and won't be fixed by whatever mitigation we put in place.
• We plan on looking into loading applications assemblies into memory as bytes instead of loading them from disk and locking the file. This should be fine for most cases but may result in more memory usage some cases.
• In order to recycle the application, you still need to add and remove an appoffline file. Adding the appoffline file is a way to signal that you want the application to restart as a result of the deployment.

[bladefist]

@davidfowl The plan you outlined for 5.0 sounds great. We use framework dependent as I'm sure most do.

Respectfully disagree about the reliability of in-place deployments. I've been doing it for years on sites with extremely high load, never had an issue, not once. The first detected change causes the app to reset, the load balancer sees the instance is not responding, takes it offline, once all the DLLs are replaced, load balancer completes a health check and it comes back online. Now if there was no load balancer health check, then ya, it's a gamble, but I mean who is replacing DLLs in production under high load w/o a proxy of some kind? Lets focus on the rule and not the exception. Without a proxy I cannot think of any way to update the app without causing an outage. Even if you supported hot reloading, some requests would probably hang or fail.

[davidfowl]

Respectfully disagree about the reliability of in-place deployments. I've been doing it for years on sites with extremely high load, never had an issue, not once. The first detected change causes the app to reset, the load balancer sees the instance is not responding, takes it offline, once all the DLLs are replaced, load balancer completes a health check and it comes back online. Now if there was no load balancer health check, then ya, it's a gamble, but I mean who is replacing DLLs in production under high load w/o a proxy of some kind? Lets focus on the rule and not the exception. Without a proxy I cannot think of any way to update the app without causing an outage. Even if you supported hot reloading, some requests would probably hang or fail.

You have a load balancer. Lots of people don't have a load balancer and are just replacing files on disk in place and expecting magic 😄

[bladefist]

@davidfowl Right but high traffic + no lb = reckless

Also, the issue with app_offline.htm is that from a ci/cd process you have no idea when the process stops. We had to add a huge sleep command between making that file and pushing the DLLs because our shut down process can take anywhere from 1s to 1 minute. When you're load balancing tons of instances those sleeps add up to very long deploy cycle.

Loading the files into memory fixes all that though.

[davidfowl]

@davidfowl Right but high traffic + no lb = reckless

👍

Also, the issue with app_offline.htm is that from a ci/cd process you have no idea when the process stops. We had to add a huge sleep command between making that file and pushing the DLLs because our shut down process can take anywhere from 1s to 1 minute. When you're load balancing tons of instances those sleeps add up to very long deploy cycle.

Right, so the problem is that dropping that file doesn't tell you when its ok to start the deployment as files are still locked until the notification is received by ANCM.

[Tratcher]

Also, the issue with app_offline.htm is that from a ci/cd process you have no idea when the process stops.

Right, the VS publisher uses short sleeps and several retries.

[Socolin]

• In order to recycle the application, you still need to add and remove an appoffline file. Adding the appoffline file is a way to signal that you want the application to restart as a result of the deployment.

Would it be possible to have a "apprestart" file to mark the app to be recycled without having to delete the appoffline file afterward ?

[davidfowl]

@Socolin why is that?

[Socolin]

@Socolin why is that?

If I understand well (tell me if I'm wrong), what you planning is to remove the lock, but we'll still have to create appoffline to signal IIS to recycle the app after the build is complete?

To do so we'll have to create the file appoffline, then delete it. Can this file be deleted immediately after the file was created ? or should we wait for IIS to detect it? If it's required to wait for a little I would prefer to have a file like "apprestart" that IIS will listen, then delete once it detects it and start recycling.

---

Also, I have another question in mind. It's about IDE optimization during the build.

Currently, I work on a project that uses dependencies, we have a project for the Web layer, and other project for Logic and Data. When I work in the Web layer (ASP.NET Core) during the build, I create an appoffline file before the build, then delete it after the build and it works.

But when I work in one of the dependencies, when I build them from Rider (I think it's doing the same on VS since it's using msbuild) when I look at what msbuild is doing, once the build of the project is is complete, the dll is copied directly in the .Web/bin directory without rebuilding the .Web project. And since the dll is locked, it silently fail, and I have to manually restart the server or manually trigger the build of the .Web project.

Have you any solution for this?
So far the workaround I have found is to write a small program running in background that look for the /bin of my .Web project. When a file change, it create appoffline file into the directory where IIS listen, then copy the modified files, then delete the appoffline. But it feels a bit hacky but so far it's the only way I've found to be able to build, then press F5 and test my changes.

[davidfowl]

To do so we'll have to create the file appoffline, then delete it. Can this file be deleted immediately after the file was created ? or should we wait for IIS to detect it? If it's required to wait for a little I would prefer to have a file like "apprestart" that IIS will listen, then delete once it detects it and start recycling.

How would IIS know when you’re done copying files? How do you avoid the torn state? You basically need to signalr the starting and completion of a transaction. The creation of the file signals the start and the deletion signals the end.

[davidfowl]

Have you any solution for this?
So far the workaround I have found is to write a small program running in background that look for the /bin of my .Web project. When a file change, it create appoffline file into the directory where IIS listen, then copy the modified files, then delete the appoffline. But it feels a bit hacky but so far it's the only way I've found to be able to build, then press F5 and test my changes.

VS handles this by killing the process before starting a new file for anything that might impact the building of that project.

[Socolin]

To do so we'll have to create the file appoffline, then delete it. Can this file be deleted immediately after the file was created ? or should we wait for IIS to detect it? If it's required to wait for a little I would prefer to have a file like "apprestart" that IIS will listen, then delete once it detects it and start recycling.

How would IIS know when you’re done copying files? How do you avoid the torn state? You basically need to signalr the starting and completion of a transaction. The creation of the file signals the start and the deletion signals the end.

If there is no lock, why notify IIS when the build start ? Can't we just signal when all the files are updated and the application need to be recycled ? I think I misunderstood something.

[davidfowl]

If there is no lock, why notify IIS when the build start ? Can't we just signal when all the files are updated and the application need to be recycled ? I think I misunderstood something.

Yes that would work. I thought you were describing the current state of the art. It might even be possible to do this today with web.config. If you touch the file it'll reboot the application.

[FlukeFan]

I've had success with in-place deployment (https://flukefan.github.io/ZipDeploy/), by:

1. Rename assembly files (apparently you can rename even though you can't delete/overwrite);
2. Copy in the new assemblies;
3. Touch the web-config to restart the app;
4. Delete the old renamed assemblies.

No load-balancing; single IIS instance; no IIS changes. YMMV.

[davidfowl]

@FlukeFan How do you avoid torn deployments or is that not a concern for you? By torn, I mean that there's a window where the old app loads a new assembly before restarting the application.

[FlukeFan]

I don't touch the web.config until all assemblies are renamed/updated. I guess there is a small chance that something else could cause an app-pool to recycle during that time, but it's not happened yet.

I usually turn off all the other auto-recycle options, and also disable overlapped recycle in the IIS app-pool settings.

[davidfowl]

@FlukeFan not a recycle but if you haven't loaded the assembly that got replaced as yet. For example, say you had an assembly A.dll and B.dll. Lets say B is used when you go to the settings page so is loaded then. If you went to the settings page when B.dll was replaced with the new one, you could end up with the old app running with a new B.dll.

[davidfowl]

I do wonder if we should make a dotnet global tool to assist in this type of deployment. The tool could basically do all the hard things people have mentioned on this thread.

Thoughts?

[FlukeFan]

@FlukeFan not a recycle but if you haven't loaded the assembly that got replaced as yet. For example, say you had an assembly A.dll and B.dll. Lets say B is used when you go to the settings page so is loaded then. If you went to the settings page when B.dll was replaced with the new one, you could end up with the old app running with a new B.dll.

So I don't handle that case just now. https://github.com/FlukeFan/ZipDeploy/blob/master/ZipDeploy/ZipDeploy.cs#L55

I guess I could do things in a different order, touch the web.config first, await all 'other' requests finishing, then finally do the replace just before the last request finished (while IIS queues up new requests for the about-to-recycle app-pool), but what I have works for me just now.

It's probably not currently robust enough for the scenario you're describing (but I haven't been doing anything fancy with dynamic assembly loading other than at startup time).

[atobrandon]

Are there plans to revisit this feature and put it on the road map? It's very inconvenient/unfriendly to require everybody deploying a .Net Core website to manually implement a staging-slot strategy if they want to support zero downtime deployments.

Having this feature would make the transition to .Net Core websites much smoother for many people, and would allow for faster adoption of .Net Core websites.

I never thought of that. But you may very well have hit the nail on the head. This could be a nefarious business decision to force people into using azure slots. If is was possible in the past why has it taken so long to "fix" now? It's not broken from a business perspective what that in mind. Maybe I'm a little cynical. But I've slowly watched azure features that used to be free whittled away and moved into more expensive tiers.

[FlukeFan]

I do wonder if we should make a dotnet global tool to assist in this type of deployment. The tool could basically do all the hard things people have mentioned on this thread.

Thoughts?

This is to run on the CI server, say? How would it push the binaries/package to the destination server?

[davidfowl]

I never thought of that. But you may very well have hit the nail on the head. This could be a nefarious business decision to force people into using azure slots. If is was possible in the past why has it taken so long to "fix" now? It's not broken from a business perspective what that in mind. Maybe I'm a little cynical. But I've slowly watched azure features that used to be free whittled away and moved into more expensive tiers.

In place deployments without coordination are fundamentally unreliable for reasons mentioned above. We’re never going to implement shadow copying again because that feature has historically unreliable and has caused lots of issues in ASP.NET on .NET framework. It also only works for a specific type of deployment, framework dependent which is all .NET Framework supported. .NET core supports self contained and single file which make it much harder to support. Even the suggestion above doesn’t cover those cases.

We still believe doing in place deployments should be as atomic as possible and lots of the problems on this issue come from bugs and unreliability of app_offline detection and recycle so we spent lots of time fixing that (and pdb locking as well).

Web deploy is the only tool today that embodies the flow with which we think it makes sense to deploy but it’s trapped in that tech so it doesn’t help if you’re ftp deploying to your servers or copying to a file share (though it could).

A dotnet global tool might be an answer here.

This is to run on the CI server, say? How would it push the binaries/package to the destination server?

I don’t think so. It would probably support FTP and a straight copy. Anything other than that is likely a proprietary protocol.

[bladefist]

Hopefully when you say FTP you also mean ftp over ssh. Such a tool would be very valuable.

[davidfowl]

Hopefully when you say FTP you also mean ftp over ssh. Such a tool would be very valuable.

No I meant ftp. But we would do whatever made sense. Maybe it’s best to be a copy tool that you need to run on the target machine. That gets us out of the business of having to send bits anywhere.

Also are you SSHing into your windows machines to deploy to IIS?

[BrightSoul]

@davidfowl Please add (opt-in) support for deployment slots.
https://github.com/dotnet/aspnetcore/issues/3719#issuecomment-473183712
https://github.com/dotnet/aspnetcore/issues/3793#issuecomment-335666414

We plan on looking into loading applications assemblies into memory as bytes instead of loading them from disk and locking the file. This should be fine for most cases but may result in more memory usage some cases.

Moreover, that won't let us achieve zero downtime.

[davidfowl]

There’s no way we’re going to invest in a feature like that with the module.

[bladefist]

Hopefully when you say FTP you also mean ftp over ssh. Such a tool would be very valuable.

No I meant ftp. But we would do whatever made sense. Maybe it’s best to be a copy tool that you need to run on the target machine. That gets us out of the business of having to send bits anywhere.

Also are you SSHing into your windows machines to deploy to IIS?

Yes. We use CI/CD and deploy to both linux and windows systems, so it was just a lot easier to install openssh on the windows boxes.

[zhongqingshen]

$appPool = 'default'
Stop-WebAppPool -Name $appPool -Passthru
... deploy...

Start-WebAppPool -Name $appPool -Passthru
...deploy.....

or

$appPool = 'default'
Stop-WebAppPool -Name $appPool
while((Get-WebAppPoolState -Name $appPool ).Value -ne 'Stop'){
sleep 1 #second
}
... deploy...

Start-WebAppPool -Name $appPool
while((Get-WebAppPoolState -Name $appPool ).Value -ne 'Start'){
sleep 1 #second
}

...deploy.....

[CJHarmath]

Poor mans deployment slots
About a year ago I’ve posted about creating 2 site root folders and have a symlink point to one of them configured for the IIS site.
The benefit is that your downtime is less comparing to the other simpler solutions.
If overlapping recycle is enabled then no downtime, only warmup penalty for the first request which is the case anyway.

If the deploy/copy operation takes N seconds to complete, you don’t need to stop the app pool for N seconds.
No messing with the questionable app offline.
Just copy to the inactive folder, swap the symlink when the files are there then recycle the app pool.
Atomic deploy, no downtime just slower first request as usual.

https://github.com/dotnet/aspnetcore/issues/3793#issuecomment-459870870

Did anyone try that?
If something like this works then could be turned into a service with a client side deploy tool.

• client -> server whats the inactive folder/share?
• S: here
• C: ok deployed / copied
• S: symlink swapped and recycled
• C: thx, http get /healthcheck (optional)
• C: ugh I messed up, rollback pls
• S: symlink swap, recycle (back in business)
• C: rinse repeat until works

[foxjazz]

So I am also having issue with deployment locked files. in app hosting. I have tried stopping the app pool, but the files remain locked. I have tried pushing the app_offline.htm but no luck there either.
Also if you want to manage deleting the app_offline.htm file you can easily do that on starting the app in program.cs. This way we know when it starts it becomes online.

The current problem I am facing is that the dll's are hung and even if you rename then copy over the files, the old process still running against the renamed files. So when we attempt to deploy again, we would have to rename the files again. I guess we could use a date of some sort as a file name replacement.
We are using dotnet core 3.1.4 for this on IIS server windows 2012 R2

[davidfowl]

@foxjazz how are the files still locked if the app pool/process is stopped? Can you elaborate?

[foxjazz]

@foxjazz how are the files still locked if the app pool/process is stopped? Can you elaborate?

When trying to delete the used dll's the picture says used by another process.
The app pool has been offline for a while.

[bladefist]

Check task manager to see if the process is still running. Hitting the stop button doesn't mean it immediately ends.

[foxjazz]

Check task manager to see if the process is still running. Hitting the stop button doesn't mean it immediately ends.

the above picture is from task manager. Not sure which one it is, but if I delete these, it will release the file. I guess I can use handle to figure out the pid to terminate. I didn't think I would have to go to these lengths for deployment. Handle says w3wp.exe 2 processes have one of the files locked. even though the app pool for it is shut down.

What would be really nice is to have a route to stop the process.
api\KillDeathKill
{
Program.KillProcess()
}

[davidfowl]

That’s unlikely. The file is locked by a process. If w3wp is that process then it was either never killed to begin with or a new one spun up with a different pid locking the same file. That’s why you need to make sure app offline is down first to make sure there’s no new process to lock the file, kill the existing process, deploy new bits, remove app offline

[foxjazz]

That’s unlikely. The file is locked by a process. If w3wp is that process then it was either never killed to begin with or a new one spun up with a different pid locking the same file. That’s why you need to make sure app offline is down first to make sure there’s no new process to lock the file, kill the existing process, deploy new bits, remove app offline

when you say app_offline.htm is down first. Yes this was the case. I have a status path I can check via a get request, and it is definitely offline. Also stopped the app pool manually. But the process w3wp was still hanging the file.

[davidfowl]

Then the process wasn’t stopped

[foxjazz]

Then the process wasn’t stopped

So why is it that stopping the app pool, not stop the process?

[davidfowl]

The app pool does represent the process and if you stop the app pool it will shit the process down. The file can’t be locked if there’s no process to lock it so one of those other theories seems more likely.

[foxjazz]

The app pool does represent the process and if you stop the app pool it will shit the process down. The file can’t be locked if there’s no process to lock it so one of those other theories seems more likely.

hah you said shit. I don't mean to seem like I am punking you. I am saying that stopping the app pool is NOT working as advertized in all the documentation. The process is not shit down, even after 90 seconds, it is still running. The file is locked even when you can see clearly that app pool has been stopped. The process contains a singleton to get some data. And other controllers. It would be nice to have a script that would shut down the process based on the locked file. I could then deploy without fear. Renaming may work also. Files can be renamed. (dotnet core 3.1.4) flavor

[drewmperkins]

About a year ago I’ve posted about creating 2 site root folders and have a symlink point to one of them configured for the IIS site.

Did anyone try that?
If something like this works then could be turned into a service with a client side deploy tool.

@CJHarmath - I did give this a try, and have the modified code I'm using below. Works very well, Thanks!

If I published updates more I might consider a client deploy tool, but really I can just remote shell into the IIS server and execute the powershell script to flip it.
psexec.exe \\IIS_SERVER cmd /c "powershell -noninteractive -file C:\FlipSymLink.ps1"

# FlipSymLink.ps1

Import-Module WebAdministration # run as admin
# create 2 site root directories
$a = 'C:\Websites\MySiteA'
$b = 'C:\Websites\MySiteB'
$appRoot  = 'C:\Websites\MySite'
$appName  = 'MyAppName'
$siteName = 'MySiteName'
$poolName = "MyPoolName"
New-Item -Type Directory $a -ErrorAction SilentlyContinue
New-Item -Type Directory $b -ErrorAction SilentlyContinue

# create a symlink to targeting one side
New-Item -Type SymbolicLink -Path $appRoot -Target $a -Name A -ErrorAction SilentlyContinue
New-Item -Type SymbolicLink -Path $appRoot -Target $b -Name B -ErrorAction SilentlyContinue

# point the site root to the symlink
$currentPath = (Get-ItemProperty "IIS:\Sites\$siteName\$appName" -name physicalPath)
if ($currentPath -eq "$appRoot\A") {
    Write-Host "Switched to B"
    New-Item $b\active.txt
    Remove-Item $a\active.txt
    Set-ItemProperty "IIS:\Sites\$siteName\$appName" -name physicalPath -value $appRoot\B
} else {
    Write-Host "Switched to A"
    New-Item $a\active.txt
    Remove-Item $b\active.txt
    Set-ItemProperty "IIS:\Sites\$siteName\$appName" -name physicalPath -value $appRoot\A
}
Restart-WebAppPool -Name $poolName

[CJHarmath]

psexec.exe \\IIS_SERVER cmd /c "powershell -noninteractive -file C:\FlipSymLink.ps1"

This could also be turned into a JEA endpoint, so you can run the flip as a non-elevated user (eg. from a CI/CD server deploy step)
Invoke-Command -ComputerName IIS_SERVER -ConfigurationName IIS-Flip -ScriptBlock { Switch-SymlinkTarget -SiteName MySite}
Used Switch instead of Flip as it's an approved verb.

[peter-dolkens]

Used Switch instead of Flip as it's an approved verb.

Swap is the verb used when dealing with slots - would make sense to re-use here it seems?

[arilishu]

having the same problem as described 4 years ago.
Any update on this?
I don't want to open Remote desktop to just upload a new dll file.

When I try to overwrite .NET Core application DLL file using FTP on production IIS server file is locked and it cannot be overwritten.

To deploy new version I need to stop application in IIS (opening RDP) to release lock, then overwrite.
It is not possible to deploy without stopping application.

Any solution for this? Thanks

[LucidObscurity]

If you can't take your host offline behind the loadbalancer and would want to do it quicker, here is one way with symlinks and some powershell.
I was thinking of doing something like this.

It's quicker as you don't have to stop the app pool during the copy + recycle is better than a hard stop as it lets the current requests complete, so less downtime.

# Setup
Import-Module WebAdministration
# create 2 site root directories
$a = 'C:\inetpub\AspNetCoreSampleA'
$b = 'C:\inetpub\AspNetCoreSampleB'
$siteRoot = 'C:\inetpub\aspnetcoresample'
$siteName = 'AspNetCoreSample'
$poolName = "aspnetcore"
New-Item -Type Directory $a
New-Item -Type Directory $b
# create a symlink to targeting one side
New-Item -Type SymbolicLink -Path $siteRoot -Target $a
# point the site root to the symlink
Set-ItemProperty "IIS:\Sites\$siteName" -name physicalPath -value $siteRoot
# make sure it get's picked up
Restart-WebAppPool -Name $poolName

# this tells you the active side
Get-Item -Path $siteRoot | Select-Object -ExpandProperty target

# Flip the symlink
$current = (Get-Item -Path $siteRoot).Target
$newTarget = if ($current -eq $a) {$b} else {$a}
New-Item -Type SymbolicLink -Path $siteRoot -Target $newTarget -Force
# at this point w3wp.exe still locks the current target folder until it's getting recycled
# Deploy new version to the symlink which is now pointing to the other side which should have no locks
robocopy \\myshare\myapp $siteRoot /mir
# recycle app pool, so it picks up the new files
Restart-WebAppPool -Name $poolName

# bonus point: rollback is easy
$current = (Get-Item -Path $siteRoot).Target
$newTarget = if ($current -eq $a) {$b} else {$a}
New-Item -Type SymbolicLink -Path $siteRoot -Target $newTarget -Force
Restart-WebAppPool -Name $poolName

Here is the gist
https://gist.github.com/csharmath/b2af0f50700ce9fbdd8c5c3e582fd41b

Here's something which I think I've got the kinks worked out. I've only tried with ASP.NET, but I imagine would work the same with the hosting module for net core. The first run does stop the site for a bit, so may want to tweak that a bit.

param(
    [Parameter(Mandatory = $true)]
    [string] $iisRootPath,
    [Parameter(Mandatory = $true)]
    [string] $siteName,
    [Parameter(Mandatory = $true)]
    [string] $artifactPath,
    [Parameter(Mandatory = $false)]
    [string] $siteFolderName,
    [Parameter(Mandatory = $false)]
    [string] $appPoolName
)
Import-Module WebAdministration

#populate optional parameters
if (!($PSBoundParameters.ContainsKey('siteFolderName'))) { $siteFolderName = $siteName }
if (!($PSBoundParameters.ContainsKey('appPoolName'))) { $appPoolName = $siteName }

#set A, B and C paths
$a = "$iisRootPath\$siteFolderName" + 'A'
$b = "$iisRootPath\$siteFolderName" + 'B'
$c = "$iisRootPath\$siteFolderName"

#existence test
$cName = Get-Item -Path $c -ErrorAction SilentlyContinue | Select-Object -ExpandProperty name -ErrorAction SilentlyContinue
$bName = Get-Item -Path $b -ErrorAction SilentlyContinue | Select-Object -ExpandProperty name -ErrorAction SilentlyContinue

#get the shudown timeout for the app pool
$shutdownTimeout = Get-WebConfigurationProperty -Filter 'system.web/httpRuntime' -PSPath "IIS:\Sites\$siteName" -Name shutdownTimeout | Select-Object -ExpandProperty Value

#create symlink, if symlink source is an existing directory, rename existing 
if($cName -eq $null) 
{
    Write-Output "Creating SymbolicLink $c -> $a" 
    New-Item -Type SymbolicLink -Path $c -Target $a
} 
else
{
    #directories don't have a target property
    $cTarget = Get-Item -Path $c | Select-Object -ExpandProperty target -ErrorAction SilentlyContinue

    #this is a directory, rename and create symlink
    if($cTarget -eq $null)
    {
        #restart AppPool first so no locked files
        #Write-Output "Restarting AppPool $appPoolName" 
        #Restart-WebAppPool -Name $appPoolName

        #stop AppPool first so no locked files
        Write-Output "Stopping AppPool $appPoolName" 
        Stop-WebAppPool -Name $appPoolName

        #sleep for shutdownTimeout
        Write-Output "Sleeping for $shutdownTimeout" 
        Start-Sleep -Seconds (([System.TimeSpan]::Parse("$shutdownTimeout").TotalSeconds) * 1.1)

        try {
             #if the rename fails, there are files in use. stop the script
            $newName = $siteFolderName + 'A'
            Write-Output "Renaming $c to $newName" 
            Rename-Item -Path $c -NewName $newName -ErrorAction Stop
        }
        catch {
            Write-Error -Message "Failed to rename $c to $newName due to files in use."

            #start AppPool 
            Write-Output "Starting AppPool $appPoolName due to failed folder rename. Consider trying again durring a time with reduced traffic." 
            Start-WebAppPool -Name $appPoolName

        }

        #create symlink
        Write-Output "Creating SymbolicLink $c -> $a" 
        New-Item -Type SymbolicLink -Path $c -Target $a

        #start AppPool 
        Write-Output "Starting AppPool $appPoolName" 
        Start-WebAppPool -Name $appPoolName

        #copy a to b to pick up directory permissions
        if($bName -eq $null) 
        {
            Write-Output "Copying Directory $a to $b"  
            robocopy $a $b /e /sec /sl
        }
    }
}

#existence test
$aName = Get-Item -Path $a -ErrorAction SilentlyContinue | Select-Object -ExpandProperty name -ErrorAction SilentlyContinue
$bName = Get-Item -Path $b -ErrorAction SilentlyContinue | Select-Object -ExpandProperty name -ErrorAction SilentlyContinue

#create if needed
if($aName -eq $null) 
{
    Write-Output "Creating Directory $a"  
    New-Item -Type Directory $a 
}
if($bName -eq $null) 
{
    Write-Output "Creating Directory $b"  
    New-Item -Type Directory $b 
}

#make sure site pointing to symlink
Write-Output "Pointing Website $siteName to Path $c" 
Set-ItemProperty "IIS:\Sites\$siteName" -name physicalPath -value $c

#restart so we know it's loading from symlink
Write-Output "Restarting AppPool $appPoolName" 
Restart-WebAppPool -Name $appPoolName

#sleep for shutdownTimeout 
Write-Output "Sleeping for $shutdownTimeout" 
Start-Sleep -Seconds ([System.TimeSpan]::Parse("$shutdownTimeout").TotalSeconds)

#get current symlink target and set new target
Get-Item -Path $c | Select-Object -ExpandProperty target
$current = (Get-Item -Path $c).Target
$newTarget = if ($current -eq $a) {$b} else {$a}

$fileOrDirectoryName = Get-Item $artifactPath | Select-Object -ExpandProperty name

#test for .zip
if(($fileOrDirectoryName).ToLower().EndsWith(".zip") -eq $true)
{
    #copy to temp location
    $guid = [System.Guid]::NewGuid()
    $tempPath = "$iisRootPath\temp-$guid"
    $tempDest = "$iisRootPath\temp-dest-$guid"

    Write-Output "Creating temp directory $tempPath" 
    New-Item -Type Directory $tempPath

    $artifactDirectory = [System.IO.Path]::GetDirectoryName($artifactPath)
    $artifactFile = [System.IO.Path]::GetFileName($artifactPath);

    #TODO: first robocopy arg needs to be a directory, not a file
    Write-Output "Copying archive from $artifactPath to $tempPath" 
    robocopy $artifactDirectory $tempPath $artifactFile

    #extract to temp destination (seems a failed extract can leave empty folders)
    Write-Output "Extracting archive to $tempDest" 
    Expand-Archive -Path "$tempPath\$fileOrDirectoryName" -DestinationPath $tempDest -Force

    #copy from temp destination to destination
    Write-Output "Copying files from $tempDest to $newTarget" 
    robocopy $tempDest $newTarget /e

    #delete temp directory
    Write-Output "Removing temp directory $tempPath" 
    Remove-Item -Path $tempPath -Recurse -Force

    #delete temp directory
    Write-Output "Removing temp directory $tempDest" 
    Remove-Item -Path $tempDest -Recurse -Force
}
else 
{
    #copy new files
    Write-Output "Copying files from $artifactPath to $newTarget" 
    robocopy $artifactPath $newTarget /e
}

#swap symlink
Write-Output "Pointing SymbolicLink $c -> $newTarget" 
New-Item -Type SymbolicLink -Path $c -Target $newTarget -Force

#restart so it loads from newly swapped symlink
Write-Output "Restarting AppPool $appPoolName" 
Restart-WebAppPool -Name $appPoolName

[arilishu]

@LucidObscurity what is that? a cmd code? where do i have to add that code?
Thanks

[joseotavioq]

What about the idea of increase the number of retries (parameter: -retryAttempts:X) or the retry time interval (parameter: -retryInterval:XXXX) for the msdeploy command, until the process release the file lock?

[minhhut]

I think @davidfowl is right, the process still there somehow and locks the files. Even Reset IIS does not help, it still wait for the process until timeout. If you drop app_offline.htm and wait for 90 seconds (default timeout setting to shutdown AppPool), and try again, you will see it works.
For me on local, I don't want to wait that long so I updated this Shutdown Time Limit (seconds) to 5 and update project file to generate app_offline.htm before build, remove it after built. It works well.
For server, I use msdeploy to drop app_offline.htm, sync new version, and then remove app_offline.htm. With msdeploy, it does retry if the files are locked but sometime gave up before 90 seconds, so I have to trigger deploy again, then it works at second try

[arilishu]

Any update about this? When this new release is gonna be ready?

[msftbot[bot]]

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[joelmandell]

I found about this problem this week, after having converting a .NET 4.5.2 application to .NET 5.0. I have another approach to overcome this. I created a Windows Service that is running on the IIS server. On that Service I have a simple HTTP Server running on custom port. The webapplication can do post-request to that server to initiate a upgrade. It stops the app-pool makes sure to kill all processes for that application, create backup of current release, removes current release, download upgradepackage and extract it. Then starts app-pool again.

I created a simple Windows Forms application, to create upgrade packages that is pushed to a repo that the above mentioned Windows Service fetches from.