Azure-pipelines-tasks: Azure Web App Deployment error: Web Deploy cannot modify the file - ERROR_FILE_IN_USE

[AndreaPic]

Hi,
I'm using Azure Web App Deployment Task in a step of continuous integration on Visual Studio Team Services in hosted environment.
Nightly are scheduled many rebuild of different app in a different build definition.
Every night some of this build definition goes in error in Web Deploy step whit this message:

Web Deploy cannot modify the file 'X' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

The file in use isn't ever the same but can change.
Can you help me ?
What can I do ?
There is a way to force stopping and restarting the app ?
Thank you
Andrea

[yaananth]

could you try setting donotdelete flag?

[AndreaPic]

I've set the "do not delete" flag and tomorrow I'll to see the output.
But I've a question about this flag.
What are "additional files" ?
Thanks
Andrea

[AndreaPic]

Hi @yaananth ,
In the last nightly build with "do not delete" flag active the problem did not occur.
I'll see the next nightly build and if it will still ok the problem could be solved.
Thanks
Andrea

[AndreaPic]

Hi @yaananth ,
Past two nightly build and complete continuous integration has the same problem.
It seems that donotdelete flag didn't solve the problem.
The last problem is :

Web Deploy cannot modify the file 'Microsoft.CodeAnalysis.CSharp.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE

Have you another idea ?
Thanks

[codito]

@AndreaPic this issue is similar to #1233, can you please try the workaround mentioned there?

Publish-AzureWebsite doesn't support AppOffline option, the newer msdeploy based azure website deployment task will fix this issue.

[AndreaPic]

Hi @codito @yaananth I've tried the solution mentioned but I've the "login" problem too.
Is the support for AppOffline option planned for "Publish-AzureWebsite" Task ?

[codito]

@AndreaPic sorry, there isn't a way to fix AppOffline issue with this task. Can you please try the msdeploy based website deployment task? It is in preview at the moment, however you can use tfx-cli to upload the task to your account and use it. We will be glad to hear your feedback.

[codito]

Just realized your account may not be enrolled into preview. Kindly drop an email to rm_customer_queries on the domain microsoft dot com if you don't see the task. /cc:@roopeshnair

[tebeco]

@codito is the issue fixed ? It's marked as closed but still occurs from vsts release

I'd like to keep release managment instead of powershell publishing if possible

I guess this issue should NOT be closed right ?

[codito]

@tebeco this was closed because there is an msdeploy based alternative for this task available which supports taking app offline. Can you please try AzureRMWebAppDeployment?

There is no plan to support app offline in this task.

[tebeco]

A ok. No pb ;)

Thx for the precision

[sebvst]

Even if I enable "take app offline" the error occurs. In my case it is always msvcr100.dll that causes the problem.

Does anyone else have the same problem?

[knoxitus]

@sebvst I have exactly the same problem that msvcr100.dll is locked by using the AzureRMWebAppDeployment build step, . Enbling or disabling "take app offline" has no effect.

[asranja]

can you mail the release logs to [email protected]. Make sure to keep the system.debug variable to true and enable the app offline feature.

[KrishnaAdityaB]

Please set below setting on the app.
MSDEPLOY_RENAME_LOCKED_FILES = 1
Automating this is in backlog. Marking this thread as enhancement

[dpodder]

@KrishnaAdityaB @RoopeshNair I've started hitting ERROR_FILE_IN_USE recently when deploying to staging slot using AzureRmWebAppDeployment (take app offline is set). Oddly does not happen when deploying directly to production; only happens when deploying to a slot.

Could you clarify what MSDEPLOY_RENAME_LOCKED_FILES = 1 is expected to do? Do the old files get deleted, after a rename? Alternatively, would you recommend adding a manual step to take the app offline prior to executing the deployment task?

[nickspiers]

I have the take app offline option selected, am not deploying to a slot but directly to production and am getting the ERROR_FILE_IN_USE.

[YanivDa]

@asranja @KrishnaAdityaB MSDEPLOY_RENAME_LOCKED_FILES = 1 solved the issue.
Is this workaround a permanent solution? what is the effect of this parameter?

[erinha]

The VSTS Azure App Service Deploy task fails occasionally with ERROR_FILE_IN_USE when deploying to a deployment slot and unfortunately using MSDEPLOY_RENAME_LOCKED_FILES=1 did not solve the issue for me. Checking "Take App Offline" also doesn't prevent the failure. The only solution I've found so far that works is to have an Azure PowerShell task run before the deployment to stop the service slot and then restart it again after the deployment. This doesn't seem expected though, based on the previous comments.

Can someone please provide more clarity about what the MSDEPLOY_RENAME_LOCKED_FILES setting does, and how it should be used?

[rajatagrawal-dev]

Hi @erinha

Can you provide us with the debug/verbose logs? We are in the process of forwarding this issue to the Azure team and your logs will help them in debugging it.
By setting MSDEPLOY_RENAME_LOCKED_FILES=1 in the Azure App Settings, WebDeploy will attempt to rename the DLL if it can’t be overwritten.
Azure team is better equipped to answer these questions in detail. So please send us the debug logs at RM_Customer_Queries at Microsoft dot com.

To get the debug logs please add a variable "_system.debug_" in your release definition and set its value to "_true_". Triggering a release then will give you the release logs.

[erinha]

Sure, I can send logs.

There are a couple of problems though:

1. The issue is intermittent (if I trigger the release deployment manually, it does not occur). I have to wait for a triggered deployment to occur (based on when someone on our team pushes to master).
2. I've currently worked around the issue by adding an Azure PowerShell task to stop the service before deploying to the slot and another after the deployment to start the service. So with this in place I don't see the error.

I'm assuming that the debug logs are only useful when the error occurs, so I'll enable debug logs for the definition and then have to disable my workarounds and wait for a failure.

[erinha]

FYI @rajatagrawal-dev, I was able to capture logs and sent them to the RM_Customer_Queries at Microsoft dot com email address, as you instructed. Hopefully they'll help diagnose the issue!

[jdshkolnik]

This is also happening with TFS 2017.

[onionhammer]

Yeah I'm using the new MS deploy task in release management VSTS, and I'm seeing this issue. I'm using the 'take app offline' flag and I'm using the MSDEPLOY_RENAME_LOCKED_FILES=1 setting.

This is the error I'm seeing. Could it be a byproduct of using the 'always online' setting for the web app? Also this is a aspnet core (on full .net framework) app.

2016-12-22T15:33:04.8639698Z Info: Using ID '755cb0bc-77e3-49eb-a055-b7342067560f' for connections to the remote server.

2016-12-22T15:33:06.7768978Z Info: Using ID 'a21a4759-da36-4412-ac68-fda97671b780' for connections to the remote server.

2016-12-22T15:33:11.2419112Z Info: Updating file (prod-c2c-borrowers\CRF.Connect2Capital.BorrowerWeb.exe).

2016-12-22T15:33:13.0488909Z ##[debug]rc:4294967295

2016-12-22T15:33:13.0488909Z ##[debug]rc:4294967295

2016-12-22T15:33:13.0488909Z ##[debug]success:false

2016-12-22T15:33:13.0488909Z ##[debug]success:false

2016-12-22T15:33:13.0608910Z ##[error]Failed to deploy website.

2016-12-22T15:33:13.0608910Z ##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy website.

2016-12-22T15:33:13.0618962Z ##[debug]System.DefaultWorkingDirectory=C:\a\r1\a

2016-12-22T15:33:13.0618962Z ##[error]Error Code: ERROR_FILE_IN_USE

2016-12-22T15:33:13.0618962Z ##[debug]Processed: ##vso[task.issue type=error;]Error Code: ERROR_FILE_IN_USE

2016-12-22T15:33:13.0618962Z More Information: Web Deploy cannot modify the file 'CRF.Connect2Capital.BorrowerWeb.exe' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

2016-12-22T15:33:13.0618962Z Error count: 1.

2016-12-22T15:33:13.0618962Z 

[mxa0079]

My team is also running into this issue (even after setting the flag mentioned in this thread... when we first added the flag the deployment started to work but this morning is failing again).

This is preventing us from using our Release Pipeline to publish to production, which has a signifcant impact in our productivity.

Could Microsoft please confirm that there is active work on this issue? Has a root cause been identified? Do we have an expected ETA? Any logs we can provide to help solve the issue?

Thanks!

[geeeyetee]

And with TFS 2012 deploying to Windows Server 2012. Locked file is msvcr100.dll. Fix is to restart app pool. Restarting appdomain does not resolve. EnableMSDeployAppOffline=true does not resolve, but it does cause the application to go offline until someone manually deletes app_offline.htm, which is, ahem, quite problematic.

[moozzyk]

Linking https://github.com/aspnet/Home/issues/694 which seems to be related.

[pgmolloy]

This problem just started for me on my normal deployment process. Could we back up to the previous version of whatever was modified in the last few days. Having to stop and restart the webapp every time even in development debug mode is getting very inefficient.

[flagbug]

Same happening here since just now

[Eneuman]

I'm having this issue as well now

[evermeire]

Same here.

[minidfx]

I use teamcity for deploying my app on Azure and I've the same error even with the MSDEPLOY_RENAME_LOCKED_FILES set to 1.

[moozzyk]

https://github.com/aspnet/Home/issues/694#issuecomment-269997757

[alex-oswald]

Same problem. MSDEPLOY_RENAME_LOCKED_FILES set to 1 worked for me. In my case it was the application dll that was causing the issue.

[bhenn]

Same here.

[NBelham]

Also seeing the same issue in the past 20 days approx. Using VS2015 Web Project Publish which includes this WebDeploy parameter -enablerule:AppOffline but I am finding that the web application isn't going offline and hence the web publish fails with the error message:

Web Deploy cannot modify the file 'someName.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

NOTE: Setting MSDEPLOY_RENAME_LOCKED_FILES set to 1 will cause the Web App to restart and so the first publish afterwards should work, but there's no guarantee your publish will beat another user visiting your site and then causing the Web Deploy to fail because of a lock assembly!

[zhangpengchen]

Same here

[nezoic]

Same problem here. The deployment task is set to take offline but it never works unless I manually stop the application in the portal, run the build/deployment and then turn the site back on.

[ArashMotamedi]

Started experiencing this on my Asp.net core web app just now as well. The .exe file is locked and can't be overwritten.

For reference, this is the deployment command:
Executing command ["C:\...\msdeploy.exe" -source:manifest='C:\...\SourceManifest.xml' -dest:manifest='C:\...\DestinationManifest.xml',ComputerName='https://....scm.azurewebsites.net/msdeploy.axd?site=...',UserName='...',Password='...',IncludeAcls='False',AuthType='Basic' -verb:sync -enablerule:AppOffline -retryAttempts:20]

[vincent1173]

@bhenn , @NBelham , @zhangpeng-kooboo , @nezoic , @ArashMotamedi ,
FILE_IN_USE issue occurs when DLL’s are locked. App_offline is believed to bring the app service down and unlock the resource. But the latest ASP.NET Core module expects app_offline.htm in root folder, but MSDeploy provides App_Offline.htm( case-sensitive). This issue is already in an active github issue with ASP.net repo and they informed that it will be fixed soon.

You can stop the App Service, Deploy the app and then start the app service to avoid FILE_IN_USE issue
using Azure App Service Manage Task.

[kirkone]

@vincentdass maybe I did something wrong, but the Azure App Service Manage Task only provides the possibility to switch deployment slots for me.

Thanks, KirK

[ArashMotamedi]

@vincentdass Thanks. Could you please elaborate if there are any differences in the deployment behavior of Asp.net Core apps vs. Asp.net apps? My understanding is that when a classic (non-core) Asp.net app was deployed, IIS would simply react to the presence of new files and restart the app pool, and therefore it resulted in continuous, uninterrupted service. Conversely, it sounds like the behavior being designed for Asp.net Core is to stop, deploy, and start IIS. My question is, is there a difference in how requests will be served during the deployment period (short as it may be)? Is it possible that some Asp.net Core requests that coincide with deployment will see a "App is offline" page, whereas this wasn't the case before? (My deployment target is an Azure Web App)

[vincent1173]

@kirkone , Latest version of Azure App Service Manage Task Deployment is yet to be completed. Can you please try managing the app using Azure PowerShell Task.

You can start-stop the app service using the Azure PowerShell Task.
Please use the following commands to manage app service.

Stop-AzureRmWebApp -ResourceGroupName "RG_Name" -Name "App_Service_Name"
Start-AzureRmWebApp -ResourceGroupName "RG_Name" -Name "App_Service_Name"

For start-stop Azure App Service Slot , we need to use different command.

Stop-AzureRmWebAppSlot -ResourceGroupName "RG_Name" -Name "App_Name" -Slot "Slot_Name"
Start-AzureRmWebAppSlot -ResourceGroupName "RG_Name" -Name "App_Name" -Slot "Slot_Name"

[vincent1173]

@ArashMotamedi ,The behavior of ASPCore and ASP.NET apps are different. There is an active thread with this issue. Will provide you the complete details, once the fix is planned.

[kirkone]

@vincentdass
I will give this a try.
so far I used the the Azure CLI task to do so but sometimes this failed with ResourceGroup not found error in my own build controllers. I think using the PS is the better option here.
I did something similar here PublishWebPackage.ps1 but I tried to use only build in tasks.
Thank you

[mvvsubbu]

There is another tracking issue, so I am closing this will update on that issue is resolved. You can find status here https://github.com/Microsoft/vsts-tasks/issues/3312

[thuru]

In the new build definition there's an option to Rename the file, which sets the MSDEPLOY_RENAME_LOCKED_FILES = 1 in the app settings of the web app.

Also, the new build definition has the Take App Offiline checkbox.

Also, Azure app service has introduced a CI option (still in preview) this could solve the problems.

[hkusulja]

I have still the same problem, on Task - Deploy AzureRM App Service - v3, checked options: Take App Offline, Publish using Web Deploy, Rename locked files.

2017-08-01T14:11:53.9735840Z Info: Updating file (myProject\ServerBackEnd.exe).
2017-08-01T14:11:57.3279522Z ##[error]Failed to deploy web package to App Service.
2017-08-01T14:11:57.3279522Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-08-01T14:11:57.3279522Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'ServerBackEnd.exe' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-08-01T14:11:57.3279522Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3msdeploy.exe failed with return code: 4294967295

[vincent1173]

@hkusulja , Did you try stopping the App Service, Deploy and start the same?
You can use Azure App Service Manage Task to start/stop App Service.

Zero downtime deployment can also be achieved by deploying to a staging slot and swapping with production slot.

• Stop Staging slot
• Deploy to staging slot
• Start staging slot
• Swap with production

[hkusulja]

I am aware of that, I want to simplify the steps for smaller projects. I believe that option "Take App Offline" should be enough, to have a downtime and upload new app. So when will be fix in that direction?

[vincent1173]

@hkusulja , you are right. But Webjobs and internal applications (ServerBackEnd.exe) may lock few resources even in App_offline state. In order to stop the above applications, deployment to staging slot is preferred.

[hkusulja]

I am talking about simple Azure web site page with dotnet core, no other stuff like WebJobs... , also if App Service is offline, my serverbackend.exe should not be running

[vincent1173]

@hkusulja , currently in process of taking this thread to Azure App Service team. Will update the thread once proper data points are received.

[davidebbo]

@hkusulja Can you share your web app name, either directly or indirectly? This will help us investigate. Thanks!

[hkusulja]

@davidebbo It is in Azure subscription in West Europe, dummy2461.azurewebsites.net is example and real web app starts with p on same App Service , it is also happening on other my Azure web sites.

[davidebbo]

@hkusulja there are a lot of different sites is the same subscription/region. Can you provide a differentiating detail (as well as the slot name if applicable)?

[hkusulja]

@davidebbo I am unsure what information do you need? I have created dummy2461 web app so you can check. I think it is general problem. dotnet core build and publish and then like in thuru picture above. It seems that our aspnet core web backend is running after app offline setting, and can not be updated. It always passes second time :/

[davidebbo]

@hkusulja did you see https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly? Your dummy site helps me identify your subscription, but I need further info to identify the specific web app where you hit this error. i.e. the one that has ServerBackEnd.exe running and locked during deployment.

[hkusulja]

@davidebbo few of my azure web apps have different ServerBackEnd.exe but all have the problem, you can pick any, or focus the one that contains "kusulja" in a name and have more letters.

[davidebbo]

@hkusulja thanks, that is what I needed. Is this a production app or a test app? i.e. is it ok to try things that may briefly take it offline?

[hkusulja]

yes and yes ;)

[davidebbo]

@hkusulja Initial findings:

• On my test ASP.NET app using .NET Core, creating app_offline in site/wwwroot instantly causes the process to shut down
• But in your app, creating it has no effect.

One difference is that you're using ASP.NET Core on full framework and not on .NET Core, though I don't know if that's the reason it doesn't work. I will discuss with Core team.

[davidebbo]

@hkusulja other question: I see that hitting the root of you app returns a 404. Is that expected? Could the app be in a weird error state?

[hkusulja]

this is by app design since it is only webapi/rest. app is working fine

[dmmusil]

One difference is that you're using ASP.NET Core on full framework and not on .NET Core

I am seeing this same issue when trying to deploy. My app matches the quoted format. Take app offline and rename locked files has no effect for me.

[davidebbo]

@dmmusil so you mean you are also using the full framework and not .NET Core?

@pan-wang, we will need to test this scenario. I've only ever used .NET Core.

[dmmusil]

@davidebbo Yes I am using an api built with ASP.NET Core (.NET Framework version) and the libraries it references are written in .NET Framework so I can use EF6.

[hkusulja]

@davidebbo I think problem is that it creates App_offline.htm and it should create app_offline.htm , somehow it is case sensitive and this VS Task has this bug using camlcase.

[pan-wang]

@hkusulja @dmmusil could you share the windows log of your application collected from Azure portal during the deployment? ASP.NET Core module will try to shutdown the backend process after detecting the existence of app_offline.htm (case insensitive). If failed, it will log an error to the log.

[hkusulja]

@pan-wang I have shared my log from my Windows Server 2016 which is VS TS build server and has latest agent in few posts above on 2017-08-01T14:11

[pan-wang]

@hkusulja That log is from WebDeploy tool which only responses for uploading files. I need a log from your Azure application to see whether the file change notification was created and the error code for terminating the backend process.
Per David's test
•But in your app, creating it has no effect.

[hkusulja]

Please provide me the concrete way how to get those logs that you need.

[pan-wang]

https://help.sumologic.com/Send-Data/Data-Types/Azure_Web_Apps/01Collect_Logs_for_Azure_Web_Apps
this blog shows how to enable logging from Azure portal. You can just download the log after test and no need to install Sumo.

[dmmusil]

45763e-5184.applicationLog.txt

[pan-wang]

@dmmusil that is Webdploy log from the deployment client. I need the windows event log from targeted server.

[dmmusil]

@pan-wang here's the other log that was stored during the failed deploy.

45763e (1).txt

[OnTheMike]

Experiencing the same issue with the app_offline.htm, worked fine via msdeploy (-enableRule:AppOffline) for a while, but since last week (an update on Azure?) it stopped working. On our end nothing has changed.

[jim-strang]

@OnTheMike same thing has been happening to me since sometime last week, with no change in the VSTS build definition, so it must be something Azure-related that broke this. Is the solution to do a web deploy instead of msdeploy?

[OnTheMike]

@jim-strang haven’t found a decent solution yet, for now we just publish the app_offline.htm manually, than the site and finally remove the app_offline. We had to disable auto swap for this to work, the second (full) sync triggert auto swapping before we could remove the app_offline resulting in a site being completely down.

[pan-wang]

@OnTheMike After manually deploying the app_offline.htm file, did you test whether your application were took offline?

[OnTheMike]

@pan-wang yes, that worked. Afterwards I was able to update the web app,
since there were no more files locked. Also had to make sure to include
app_offline.htm otherwise it would instantly removed by the sync again.
On Mon, 7 Aug 2017 at 17:26, pan-wang [e-mail] wrote:

@OnTheMike https://github.com/onthemike After manually deploying the
app_offline.htm file, did you test whether your application were took
offline?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-tasks/issues/1607#issuecomment-320695476,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABsR-TR1QsBuNToCfHK__4Z7HApP2LgTks5sVyy-gaJpZM4ISlBp
.

[jim-strang]

@OnTheMike @pan-wang good to know, thanks. Did you try the azure CLI/powershell solution to stop/start the web app before going to publishing app_offline.htm manually?

[pan-wang]

@OnTheMike @jim-strang could you please try increase webdeploy retryAttempts counter (e.g., 5) and retryInterval (e.g., 5000 , i.e. 5 seconds) to see whether it helps? Here is the link about webdeploy settings https://technet.microsoft.com/en-us/library/dd569089(v=ws.10).aspx. The new ANCM adds some graceful shutdown logic which increases the process shutdown time.

[davidebbo]

Here is a simple workaround that you should all try. In your .csproj, add this line in the <PropertyGroup>:

<RetryAttemptsForDeployment>20</RetryAttemptsForDeployment>

See here for a full csproj sample with this change.

[TimPosey2]

@davidebbo: Is that specifically for .NET Core 2.0? I tried it on 1.1 and it did not work consistently. The first build it worked (previously it would fail every time). Tried it again to make sure it wasn't a fluke and it failed. Same ERROR_FILE_IN_USE.

[davidebbo]

@TPoise it should be agnostic of the version. What if you try a higher retry count, like 30? It all comes down to how long it takes for the core process to shut down after app_offline.htm gets created.

[TimPosey2]

@davidebbo: Using @pan-wang's suggestions, I set -retryInterval:6000 and -retryAttempts:10 and I've gotten it to work successfully 4 attempts in a row.

[davidebbo]

@TPoise then you probably just needed a bit more than 10 seconds. I updated my workaround above to say 20. In the end, it's the same flag, though it's unclear whether there is a way to change the retry interval via the csproj.

[vijayrkn]

Currently the retryinterval is not configurable from an msbuild property. I have created a bug to track this work - https://github.com/aspnet/websdk/issues/226

The default retryinterval for msdeploy is 1sec.

[hkusulja]

I believe that it worked normally, than something about 2017-07-15 it stopped working, every even build passes normally..., multiple projects, all are dotnet core 1.1 . Something happened to Azure or VS TS Task...

[OnTheMike]

We are deploying via Jenkins, calling msdeploy directly. So it seems to be Azure-related. I've just tried the option with the extended interval and retry, that seems to work. Haven't tried it with autoswap yet.

[OnTheMike]

Seems to work fine with autoswap as well.

[jim-strang]

The msdeploy retry attempts and retry interval flags seemed to have solved the problem for me as well. I also went a step further and used some powershell scripts to deploy and remove a custom app_offline.htm to the wwwroot folder before and after my VSTS deployment task using the Kudu API. This is a good resource if you want to go down that route too.

[davidebbo]

FYI, I documented the issue and workaround on https://github.com/Azure/app-service-announcements/issues/24.

Note that I suggest only tweaking retry attempts and leave the interval at the default 1 second. Increasing the interval can slow down your deployment more than necessary, and there is nothing wrong with trying every second.

Let me know if you have any feedback/clarifications on the announcement.

[AaronRM]

@davidebbo Even when setting -retryAttempts:80, we're still consistently hitting ERROR_FILE_IN_USE. We also have 'Take App Offline' and 'Rename locked files' checked. Should we just continue trying to increase the retryAttempts value until we hit a threshold where it consistently succeeds?

[davidebbo]

@AArnott you may have an unrelated issue. What specific file is in use? Try manually creating app_offline.html in Kudu console, and check whether the dotnet process eventually shuts down (you can check via Kudu Process Explorer).

[AaronRM]

@davidebbo I presume you meant me. :) Indeed, there was some other issue going on with the particular app in question (was in a strange state where the app exe wasn't running). The file in use error we were seeing was the app executable itself. Stopping the app and re-deploying appears to have fixed it. Thanks!

[davidebbo]

@AaronRM yep, sorry, GitHub name completion fail on my part :)

Glad you figured it out.

[AaronRM]

@davidebbo I retract my previous comment. We are still hitting this regularly across several different apps/environments. FWIW, this is an ASP .NET Core app running on Full .NET Framework. When it does fail, it's the app executable that's the file in use:

2017-08-09T19:26:14.3813460Z` ##[error]Failed to deploy web package to App Service.
2017-08-09T19:26:14.3813460Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-08-09T19:26:14.3813460Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'App.exe' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
Error count: 1.

2017-08-09T19:26:14.3823659Z ##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295

[mehalick]

Same issue for me very consistently, also using ASP.NET Core running on full .NET framework.

2017-08-09T21:41:30.0475840Z Info: Updating file (bilbayt-api\Bilbayt.Api.exe).
2017-08-09T21:41:33.7547205Z ##[error]Failed to deploy web package to App Service.
2017-08-09T21:41:33.7557130Z ##[warning]Try to deploy app service again with Rename locked files option selected.
2017-08-09T21:41:33.7557130Z ##[error]Error Code: ERROR_FILE_IN_USE
More Information: Web Deploy cannot modify the file 'Bilbayt.Api.exe' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

I've tried all workarounds above with no success. I can confirm that the VSTS Azure Publish task writes App_Offline.html and not app_offline.html if that makes a difference.

[davidebbo]

@mehalick did you try manually creating app_offline.htm from Kudu console to see if that triggers the eventual process shutdown? That is a good way to help isolate.

[pan-wang]

@AaronRM @mehalick Once ANCM module detects the creation of app_offline.htm (case insensitive), it will try to gracefully shutdown the backend process (default timeout 10s). If failed, ANCM will kill the backend and log a warning to Windows event log. Could you please enable collecting windows event log on Portal during deployment and share the log?

[AaronRM]

@pan-wang Where is the setting in the Azure Portal to enable collecting the Windows Event Log during deployments?

[thuru]

@AaronRM : yes I experience the same, and the builds one after the other gets succeeded, this pattern is a very consistent from last week.

2/2 occurring from Aug 3rd. When queued again it goes success. Every build one after the other getting success. @VSTS pic.twitter.com/QVIunduzxw

— Thuru (@ThuruTweets) August 4, 2017

[mehalick]

I can consistently reproduce the issue in Kudu with an ASP.NET Core 1.1 site running on the full framework (Net47). When creating App_Offline.htm manually in Kudu I log the shutdown process starting but failing with a 404:

Sent shutdown HTTP message to process '8468' and received http status '404'.

Within a few seconds I log a second message:

Failed to gracefully shutdown process '8468'.

Here's the full eventlog.xml from Kudu:

<Events>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module"/>
            <EventID>1001</EventID>
            <Level>0</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2017-08-11T12:49:45Z"/>
            <EventRecordID>563262140</EventRecordID>
            <Channel>Application</Channel>
            <Computer>RD0003FF70DE37</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Application 'MACHINE/WEBROOT/APPHOST/BILBAYT-API' started process '8468' successfully and is listening on port '33198'.</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module"/>
            <EventID>1006</EventID>
            <Level>0</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2017-08-11T12:50:35Z"/>
            <EventRecordID>563312156</EventRecordID>
            <Channel>Application</Channel>
            <Computer>RD0003FF70DE37</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Sent shutdown HTTP message to process '8468' and received http status '404'.</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module"/>
            <EventID>1005</EventID>
            <Level>0</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2017-08-11T12:50:45Z"/>
            <EventRecordID>563321984</EventRecordID>
            <Channel>Application</Channel>
            <Computer>RD0003FF70DE37</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Failed to gracefully shutdown process '8468'.</Data>
        </EventData>
    </Event>
</Events>

Here's a 35 second screen capture of the process that consistently generates the error for me:

https://andy.azureedge.net/blog/2017-08-11_8-50-58.mp4

[pan-wang]

@mehalick This is a regression in ANCM (build 1983) which added graceful shutdown support allowing the application to finish long running request in given time. The change somehow ignored some failure path. As result, ANCM will wait for timeout (default 10 seconds) in some code path instead of aggressively terminating the backend process. This regression has been fixed in new ANCM (build 1984) which is till under deployment.

After the graceful shutdown failure event, the backend process should have been killed and you should be able to deploy the new assemblies. If this not the case, please let me know.

Please do add retry mechanism in your deployment as posted by David even after the regression has been fixed. This is because due to graceful shutdown, the application is given more time (configurable) to finish requests and exit. A long running request may hold the backend process till timeout.

[hkusulja]

@pan-wang any expected date when this is expected to be fixed on Azure production? Thank you

[mehalick]

@pan-wang I can confirm that adding a retry to my deployment is a successful workaround, thank you.

For the record I'm deploying via VSTS, while there's not exactly a retry option this can be achieved by duplicating the deployment task but set to run only on failure of a previous task. This isn't a great long-term solution but it's acceptable for now.

[davidebbo]

@hkusulja the fix should be deployed by the end of the month.

[matthieugd]

We are using a Powershell script to publish our app. We tried to set the retryAttemps but it doesn't fix the problem. It seems that the retryAttemps value is not correctly handled?

Log:

[Step 1/4] PowerShell Executable: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe [10:26:21][Step 1/4] Working directory: C:\Data\teamcity\server\buildAgent\work\392f4ac975965f5\scripts_backend [10:26:21][Step 1/4] Command: C:\Windows\sysnative\WindowsPowerShell\v1.0\powershell.exe [10:26:21][Step 1/4] PowerShell arguments: -Version, 5.0, -NonInteractive, -ExecutionPolicy, ByPass, -File, C:\Data\teamcity\server\buildAgent\work\392f4ac975965f5\scripts_backend\publish-backend.ps1, -slot, dev [10:26:22][Step 1/4] Path: C:\Data\teamcity\server\buildAgent\work\392f4ac975965f5\scripts_backend\publish.ps1 [10:26:23][Step 1/4] VERBOSE: Loading module from path [10:26:23][Step 1/4] 'C:\Data\teamcity\server\buildAgent\work\392f4ac975965f5\scripts_backend\publis [10:26:23][Step 1/4] h-module.psm1'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Get-AspnetPublishHandler'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Get-MSDeploy'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Get-PropertiesFromPublishProfile'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Publish-AspNet'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Publish-AspNetFileSystem'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Publish-AspNetMSDeploy'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Publish-AspNetMSDeployPackage'. [10:26:23][Step 1/4] VERBOSE: Importing function 'Register-AspnetPublishHandler'. [10:26:23][Step 1/4] Publishing with publish method [MSDeploy] [10:26:23][Step 1/4] Executing command ["C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:manifest='C:\Data\teamcity\server\buildAgent\temp\buildTmp\PublishTemp\obj\publish_backend\SourceManifest.xml' -dest:manifest='C:\Data\teamcity\server\buildAgent\temp\buildTmp\PublishTemp\obj\publish_backend\DestinationManifest.xml',ComputerName='https://XXXX.scm.azurewebsites.net/msdeploy.axd?site=XXXX-backend__dev',UserName='$XXXX-backend__dev', Password='{PASSWORD-REMOVED-FROM-LOG}',IncludeAcls='False',AuthType='Basic' -verb:sync -enablerule:AppOffline -enableRule:DoNotDeleteRule -retryAttempts:20 -disablerule:BackupRule] [10:26:52][Step 1/4] Info: Using ID '9baa4f23-3dbe-49e4-93c2-286f08be116c' for connections to the remote server. [10:26:52][Step 1/4] Info: Using ID '0b6871d3-5e6b-48ee-9e79-380490592301' for connections to the remote server. [10:26:52][Step 1/4] Info: Updating file (XXXX-backend__dev\AngleSharp.dll). [10:26:52][Step 1/4] Info: Updating file (XXXX-backend__dev\Apcurium.Boost.dll). [10:26:52][Step 1/4] [10:26:52][Step 1/4] Error Code: ERROR_FILE_IN_USEMore Information: Web Deploy cannot modify the file 'XXXX.Boost.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.Error count: 1.

[IvanAlekseev]

@davidebbo what is a current state of this issue? It is really annoying

[davidebbo]

@IvanAlekseev The fixed ANCM @pan-wang's refers to is fully deployed. So if you are still running into issues, it is likely something different, and it may be best to open a new issue.

[CanisLupus]

Is this really fixed? I have VS updated as per the Visual Studio Installer, I have MSDEPLOY_RENAME_LOCKED_FILES set to 1 on the server, I have added <EnableMSDeployAppOffline>True</EnableMSDeployAppOffline> to the .pubxml file and most of the time I get a "Publishing failed" message. After it appears, restarting the server doesn't help. The only way I can publish at that time is delete every file in the server and then reupload everything. Am I missing something? I'm trying to upload an ASP.NET Core application to Azure (with WebDeploy).

[rmarinho]

Hey @vincentdass discuss moving from #4365

Nop i didn't put the app offline, I will try. I m not using webjobs but i m using ApplicationInsights.

[henkmollema]

How can this be closed? We still hit this issue daily with our ASP.NET Core apps and it's really killing a productive CI workflow.

[vincent1173]

@henkmollema , apologies for the inconvenienvce.
We have an active thread here to follow on this issue.

[henkmollema]

@vincentdass thanks, I'll keep an eye on it.

[danieldiazastudillo]

My first Core app, candidate to go on production and this error comes up, from nothing, out of nowhere :(. No update, no changes, nothing. I had to stop the app on Azure, publish via VS and it worked. I'll configure CI/CD with GitHub and see what happens, i've never had this problem before (.NET Classic and so on)

[diegohb]

In teamcity, only way I could get it to work was to stop and start the app/slot.. i use an auto-swap slot named "production-alternate" and only deploy to it.. which I think means either that slot or production instance will always be up.. i also have build config using retry trigger.. in addition to sleeping after stopping site cause I was getting errors sporadically... i used azure-cli.

here's a teamcity metarunner i created: https://gist.github.com/diegohb/1e32fa0728110024c904de6c7ac5130b

[Ro3A]

I've landed here from https://github.com/aspnet/Home/issues/694 and https://github.com/Azure/app-service-announcements/issues/24.

I'm still seeing an issue with locked files during deployment from Visual Studio 2017. I've been using MSDEPLOY_RENAME_LOCKED_FILES for two years now with a staging slot with no issues. All of a sudden, I'm guessing after a VS 2017 update, I'm unable to deploy to my staging slot.

Adding 20 into the csproj fixed the issue but the comments on the azure announcement indicate the issue has been resolved and thus the RetryAttempts workaround should not be necessary.

That is not the case, I've just tried it and it still fails without it.

[buzallen]

This is still an issue for me, but only on certain projects and I can't find a common thread as of this point. This is a .net core 2.0 MVC application being published to a azure web app. Only way around it is to stop the service and then restart it. Note this is not a slot in this case, it's the regular app.

Over the summer this was an issue on some other projects but that went away and I assumed the problem was fixed. In two new projects this issue is back. I have the retry attempts set at 20 but per another issue thread that is not needed anymore.

[yaananth]

Please move discussions to https://github.com/Microsoft/vsts-tasks/issues/5259 to make sure it gets traction.