Kudu: Web App service not restarting after zip deploy, previous site package serving requests/Loaded into memory

[jstallm]

I noticed today that upon successfully deploying my application from an Azure Devops release pipeline that after deployment, the previous artifact was serving responses to requests vs the latest deployed artifact UNTIL I restarted the application. This was very strange since I was under the impression that previous assemblies were unloaded after a deployment to an app service.

Repro steps.

Here were steps for me to produce;
Deploy app service using Azure Devops App Service Deploy task. Create a rest api endpoint on the application which simply returns the assembly version using a setter equal to

typeof(Program).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>().InformationalVersion
as a string.
Even if the two artifacts have completely different versions, the assembly that is loaded into memory appears to still be version of application that was previously overridden during the deployment. Run a restart on the application using azure portal. Re-run a request to the endpoint that returns the assembly version and this time it will have the latest deployed version.

How is it that after a deployment the previous assembly is still loaded into memory such that a request to the app service returns a response from the application artifact that was JUST overriden with a new version? Why does restarting load the latest deployed artifact but not a deployment task in azure devops?

@suwatch could this be a bug from commit 8c8eba7?

This odd issue happened on both an api app and web app. Also happened on free and paid plan.

[jstallm]

Still seeing this happen today. I even tried the following to ensure I wasnt crazy.
I deployed an application, with dll versioned to 1.0.0-beta.1+0
Then, I deploying a different release, versioned as 1.0.0-beta.1+1.
Using App Service > Advanced Tools (Kudo) I downloaded the dll and using ildasm.exe checked the version and it was 1.0.0-beta.1+0.
I restarted the application and downloaded the dll again, opened in the dll using ildasm.exe checked the version and it was 1.0.0-beta.1+1.

Why does restart change the file? Why isnt deploy restarting the application itself after the deploy, and still having this old dll still exist on app service app wwwroot folder?

This is really very unexpected and makes deployments fail essentially, since Id imagine many engineers are necessarily creating restart tasks after every deployment explicitly.

[suwatch]

The Invalid access to memory location could be because you have WEBSITE_RUN_FROM_ZIP or WEBSITE_RUN_FROM_PACKAGE set to 1 or url. This is benign error since, in this mode, one cannot modify the file (nor touch its datetime).

As for change did not take effect (has nothing to do. We will need a site name and UTC time of the deployment and we can help investigate why.

[jstallm]

It would be strange to me if nobody else is having this issue since it has appeared to be systemic as it is happening to multiple web apps I have. I dont think this is idiosyncratic to me.

[jstallm]

Is there any explanation as to why the previous dll was loaded into memory and only until I restarted the app did the new dll replace the old? This is super strange behavior, and very confusing.

[suwatch]

If you provide a sitename and UTC time of deployment having issue, we like to look at the log to see what was going on and fix it.

[jstallm]

sitename: bin...jws

logs of azure devops release deploying:
2019-03-22T01:02:52.5758815Z ##[section]Starting: Deploy Azure App Service 2019-03-22T01:02:52.5874844Z ============================================================================== 2019-03-22T01:02:52.5875190Z Task : Azure App Service Deploy 2019-03-22T01:02:52.5875367Z Description : Update Azure App Services on Windows, Web App on Linux with built-in images or Docker containers, ASP.NET, .NET Core, PHP, Python or Node.js based Web applications, Function Apps on Windows or Linux with Docker Containers, Mobile Apps, API applications, Web Jobs using Web Deploy / Kudu REST APIs 2019-03-22T01:02:52.5875533Z Version : 4.3.21 2019-03-22T01:02:52.5875625Z Author : Microsoft Corporation 2019-03-22T01:02:52.5875781Z Help : [More information](https://aka.ms/azurermwebdeployreadme) 2019-03-22T01:02:52.5875932Z ============================================================================== 2019-03-22T01:02:53.6488025Z Got service connection details for Azure App Service:'Bin...jws' 2019-03-22T01:02:54.5347695Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"} 2019-03-22T01:02:55.2718123Z Updated App Service Application settings and Kudu Application settings. 2019-03-22T01:02:55.5846751Z Package deployment using ZIP Deploy initiated. 2019-03-22T01:02:57.1736856Z Successfully deployed web package to App Service. 2019-03-22T01:02:58.2294350Z App Service Application URL: -- 2019-03-22T01:02:58.2533008Z ##[section]Finishing: Deploy Azure App Service

The logs above deployed version '1.4.0-alpha.14', but despite this, the previously deployed version '1.4.0-alpha.11' is still running/loaded into memory.

[jstallm]

At what point does the azure app service unload the previously deployed site package and load the newly pushed Site Package? I assumed it was upon deployment but my issue above invalidates that assumption.

Do I literally need to create dedicated steps in the release pipeline to restart the app service in order for force unloading the previous binary? Isnt the site restarted as a result of the release task 'Azure App Service Deploy'?

[suwatch]

The whole idea was after deployment, site will restart to pick up change automatically (within 5-10s). User need not do any extra work. We will look into the log and update. I notice you are using FREE plan - I wonder if that might be some limitation of such. Do you have a paid plan that you can try to repro this?

[DavidZidar]

I'm not on a free plan and I'm experiencing the same or a similar issue. The zip deployment seemingly succeeds and the packagename.txt file is updated correctly but the site never restarts so the old deployment is still active many minutes later, until the site is manually restarted it seems.

My deployment logs say nothing of any invalid accesses to memory though.

[jstallm]

@suwatch , thank you for looking into the logs. I am looking forward to hearing your response. I initially noticed the issue on both the free plan and paid plan. After, I created the website above using free plan in order to reproduce. I hope the importance of understanding the restart behavior is equal regardless of plan type as I have viewed the issue above in both free and paid, and we have 1 more report of this from @DavidZidar . What other places might this issue be reported on to see if this is in fact a systemic concern of the community or idiosyncratic to me and @DavidZidar ?

[jstallm]

@suwatch , if restart is guaranteed as you mention above, you should the be able to see a restart perhaps within some internal logs you have available shortly after 2019-03-22T01:02:57.1736856Z. But you wont see that the app restarted, since if it did I wouldnt have seen the issue. Can you please confirm that the logs do not have restarts? What code is responsible for the restart?

[jstallm]

I also see that restart is not guaranteed necessarily and that its based on
https://github.com/Microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureRmWebAppDeploymentV4/deploymentProvider/WindowsWebAppZipDeployProvider.ts

I see
await this.kuduServiceUtility.deployUsingZipDeploy(webPackage, this.taskParams.TakeAppOfflineFlag, { slotName: this.appService.getSlot() });

whichs indicates to be that the only value that determines whether or not to restart the app is this argument.
Did the default value for this change recently to false?

fyi, within my azure devops jobs I have Select Deployment Method unchecked. This has remain unchecked and I have never changed this setting. Yet the app has always restarted automatically, except starting around the 20th where the app did not appear to restart.

[jstallm]

Interesting, I see its set the false by default. I now wonder how in the world the restart was initiated in the past?
TakeAppOfflineFlag: tl.getBoolInput('TakeAppOfflineFlag', false),
https://github.com/Microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureRmWebAppDeploymentV4/operations/TaskParameters.ts#L123

[suwatch]

@thaishankar please assist.

[jstallm]

K, I have resolved this by being explicit about the 'deployment method' within the stages task.

Its not necessarily evident to someone who is creating/configuring the 'Azure App Service Deploy' task that the implied/auto-detect will not necessarily restart the application.

Configuration shown below for my resolution.

It may be that in the past the reason why the application was restarted was because the site had a change to the application settings and then restarted as a result of detecting a change in the configuration. Just my best guess.

[DavidZidar]

Previously sites were always automatically restarted when ZIP deploy was performed even if no settings had been changed. My deployment issues started yesterday.

For now I have worked around the issue by sending DELETE /api/processes/-1 to terminate the main process if the site does not start serving the expected version within a specified amount of time.

[suwatch]

We are still looking .. thanks for reporting

[suwatch]

We figured out the issue. Our latest upgrade caused this regression. If your Azure App Service version is 81 and higher, you are affected. We are working on the hot fix which will roll out next week. Apology for the inconvenience. The work around is to restart the site. We understand it wasn't the best workaround but, at minimum, should get you unblocked.

[jstallm]

@suwatch , where is the best place to track the progress of this ? This issue? Another? thank you.

[suwatch]

I will personally update this once the fix goes out. If you have sites that would want to get the fix early (please share you site name here or indirectly). The earliest time for the fix is next Mon (3/25) and may take a few days after.

[pharring]

@suwatch thanks for the update. I bumped into this myself yesterday. I thought the error was on my end (did the deployment silently fail?) and I spent some time going over our Azure DevOps deployment tasks looking for the error.

[danzel]

This sounds like what we are running in to.

Watching the logs while the deploy happens, we see this:

2019-03-26T01:55:14  Fetching changes.
2019-03-26T01:55:14  Updating submodules.
2019-03-26T01:55:14  Preparing deployment for commit id '5bbf85541b'.
2019-03-26T01:55:14  Skipping build. Project type: Run-From-Zip
2019-03-26T01:55:14  Skipping post build. Project type: Run-From-Zip
2019-03-26T01:55:15            Error occurred, type: error, text: Invalid access to memory location., stackTrace:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)at System.IO.File.SetLastWriteTimeUtc(String path, DateTime lastWriteTimeUtc)at Kudu.Core.Deployment.DeploymentManager.TryTouchWatchedFile(DeploymentContext context, DeploymentInfoBase deploymentInfo) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\DeploymentManager.cs:line 858
2019-03-26T01:55:15  Updating D:\home\data\SitePackages\packagename.txt with deployment 20190326015514.zip
2019-03-26T01:55:15  Deployment successful.

After that, the app never gets reloaded for the new version and we continue running the old version.

[jstallm]

@danzel , yeah you are having exact same issue. @suwatch , any updates?

[jstallm]

@suwatch , I accidentally closed this issue while trying to post. please advise.

[suwatch]

The fix has been rolled out to 90% of the scale units .. please verify and let us know if still not fix and provide us the sitename.

[suwatch]

The issue should be mitigated now. Thanks for reporting, apology for the inconvenience.

[blueelvis]

@suwatch - I am also facing this issue across our various environments for multiple web apps which are using the Zip Deploy method. The only way is to stop the web app, wait for it to give 403 and then start it again. We do have a support ticket (120011524002498) going on but it has been quite some time.

Web App name -- aze...-d-web-02-webapp

[suwatch]

Umm .. that was over a year ago. Please provide UTC time of your deployment and we can look into it.

[blueelvis]

@suwatch - I know :( . If you want, I can create a new Github issue. Here is the UTC timestamp -- 11:21 AM on Wed Mar 11 2020.

[suwatch]

Yes please create a new one .. put as many such as time of inicident, partial site and subscription name.

[robert-skarzycki]

@suwatch , what exact Kudu version is a fixed one? In my case 85.11226.4297 seems to be buggy (I was acing similar problem in April), but on freshly created app plan & app service (with 86.20224.4450) I can't reproduce that. If the latter version is fine, how to update that? Is it possible without recreation of app plan / app service?

[suwatch]

restart the webapp should pick up the latest Kudu automatically. If not, check d:\home\siteextensions\kudu and see if exists (it will override the Kudu available on VM). If still not, share your site name.

[Stephen-McDowell]

We're being bit by the 'Invalid access to memory location.' issue. We are deploying to a slot. Taking some tips from the comments from last year I did the following to get the deployments working again: 1) enabled 'Select deployment method' 2) selected 'Zip Deploy' as the Deployment method and 3) enabled 'Take App Offline'. Interested in getting this sorted out.
Subscription ID: 89b....1f85
Site: ...account-scus/staging

[robert-skarzycki]

@suwatch:

Subscription: 2e17...66fb
Site: sq-sto...od

[suwatch]

@robert-skarzycki did you restart (stop and start) the site? this is to force the pickup of new Kudu.

[robert-skarzycki]

@suwatch, in fact start & stop helped with getting latest Kudu. During upcoming deployment I'll check if this will help, but I guess it should. Thanks! :)