Kudu: Receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app

[mcdurdin]

This issue has been moved over from https://developercommunity.visualstudio.com/content/problem/753520/receiving-error-error-failed-to-deploy-web-package.html

We are sporadically receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app, with no other useful details that I can find. Following this, I can usually log in to dev.azure.com and start a new deployment which runs successfully. I just can't find any more detail about what the root issue is or how we can address this problem.

For deployment issues, please provide us with the following information:

Repro steps.

your project built successfully on your dev machine but failed on Azure?
please write down your build tools and their versions (ie Msbuild 15.1.0.0)

The project builds successfully both locally and on Azure (Node.js backend / Angular+Typescript front end). It fails only in the Release pipeline.

Project structures.

in order to reproduce your issue at our end we need a simple
github repository
that highlights structure of the project

See below; the site is straightforward in structure and documented, hopefully fairly minimal as it stands. The site won't run without specific keys for build.palaso.org (and perhaps github.com, although any user's key should work here); I can supply test keys if required.

The log/error given by the failure.

Normally this include a stack trace, error code and some more information.

2019-09-29T20:01:05.5384672Z ##[section]Starting: Deploy Azure App Service
2019-09-29T20:01:05.5523287Z ==============================================================================
2019-09-29T20:01:05.5523400Z Task         : Azure App Service deploy
2019-09-29T20:01:05.5523499Z Description  : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2019-09-29T20:01:05.5523568Z Version      : 3.4.31
2019-09-29T20:01:05.5523636Z Author       : Microsoft Corporation
2019-09-29T20:01:05.5523703Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-rm-web-app-deployment
2019-09-29T20:01:05.5523797Z ==============================================================================
2019-09-29T20:01:06.3287589Z Got connection details for Azure App Service:'status-keyman-com'
2019-09-29T20:01:07.4387353Z Package deployment using ZIP Deploy initiated.
2019-09-29T20:03:25.3932504Z ##[error]Failed to deploy web package to App Service.
2019-09-29T20:03:25.3976382Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)
2019-09-29T20:03:29.6500803Z ##[warning]Error: Failed to update deployment history. Error: Conflict (CODE: 409)
2019-09-29T20:03:29.6646084Z ##[section]Finishing: Deploy Azure App Service

Debug your Azure website remotely.

it is recommanded that you share your Web App name, directly or indirectly
we can take a look at what's going on.

The GH repo is https://github.com/keymanapp/status.keyman.com; live site is https://status.keyman.com

Mention any other details that might be useful.

---

Thanks! We'll be in touch soon.

[suwatch]

@sanchitmehta please investigate this linux deployment issue.

@mcdurdin the 409 conflict indicated ongoing deployment. Please check if you have other deployments active during such time.

[mcdurdin]

Please check if you have other deployments active during such time.

We didn't have any other ongoing deployments as far as I can tell from the logs.

[asmendez]

Hi, I have the same problem, since I have 2 applications in the same app service plan and I have both integrated with Azure devops through the Development Center. Even I have only one pipeline that deploys first to staging and after to production. I don't know what's wrong because a few days ago this was working well.

[sanchitmehta]

@asmendez Did the first deployment fail for some reason ? We have a cool off period in Deployment Locks (when an unexpected error occurs leading to a Kudu Crash) to avoid conflicts. Could you share your app name ?

@mcdurdin are you still facing this problem ?

[mcdurdin]

@mcdurdin are you still facing this problem ?

We have not seen any errors in the last few deployments so I think it's good now.

[asmendez]

@asmendez Did the first deployment fail for some reason ? We have a cool off period in Deployment Locks (when an unexpected error occurs leading to a Kudu Crash) to avoid conflicts. Could you share your app name ?

@mcdurdin are you still facing this problem ?

Hi, at the end I changed to a Windows App Service because in Linux this problem came up even with a fresh and new instance with the CD/CI. Even when I deleted all my instances from the same App Services Plan. I don't know why this was happening but I decided to change to a Windows PHP Web App and I don't have the problem now.

[evelo2]

Sorry, this issue is very much still alive.
Existing builds that were deploying successfully are now failing consistently.

I have posted the YAML to StackOverflow

Any help would be appreciated.

[nmaarse]

I have the same problem :
Package deployment using ZIP Deploy initiated.

[error]Failed to deploy web package to App Service.

[error]To debug further please check Kudu stack trace URL : https://$dqipowebportal0001:*@portal0001.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace

[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

[guiihlopes]

I had the same problemas some days ago, and the issue is caused sometimes when the azure plan CPU reaches 100%.

It gives conflict error message or this one:

An unknown error has occurred. Check the diagnostic log for details.
##[error]Failed to deploy web package to App Service.

[sanchitmehta]

@nmaarse @evelo2 This error implies that Kudu was already processing a deployment request and to ensure that deployment is an atomic operation Kudu throws 409. We would have to see if the pipelines was sending multiple requests. Could you open a support ticket (thereby you could share the app name) and we can investigate it further.

[nmaarse]

@nmaarse @evelo2 This error implies that Kudu was already processing a deployment request and to ensure that deployment is an atomic operation Kudu throws 409. We would have to see if the pipelines was sending multiple requests. Could you open a support ticket (thereby you could share the app name) and we can investigate it further.

Hi sanchimehta. Happy to hear from you. Could you explain to me how and where to open a support ticket.

[wadeem]

same problem here

[akisvolanis]

If you are on a free subscription plan, make sure you haven't reached your storage limit which is 1gb. If so, you have to upgrade to a paid one..

[i3anaan]

Had the same issue today. The service plan had plenty of room left (CPU/Memory/Disk), no other deployments were being processed at the time and a run that previously succeeded failed when retried today (same commit). Additionally it seemed that kudu crashed as well or something, the suggested link for more debug information gave a 500 and other attempts to access kudu, logs or files of the function all failed (the ones I tried at least).

In the end I managed to resolve the issue by deleting the function app, and re-creating it (same settings). Thankfully this was a (new) function in a development environment, this would be an unacceptable workaround on a live production environment.

[jstolwijk]

Since today all our applications cannot be deployed, some applications receive a 500 while others receive a 409 response. Like @i3anaan pointed out deleting the function app and re-creating it resolves the issue. We cannot do this in prod since it will result in tens of minutes of down time

[jjavierdguezas]

same issue since today here (same scenario than @i3anaan ), any solution other than recreating everything?

[jjavierdguezas]

I have solved the problem by scaling up the service plan where my azure functions were. I have switched from B2 to B3 tier, which only increase RAM and ACUs so I'm guessing that we had consumed all ACUs ?

[IFYates]

This also started happening for us today on S1 tier in multiple App Service Plans.
Scaling up to S2 tier solved it. After scaling back down to S1, I was still able to release.

Out typical ASP usage on S1 is 6% CPU and 60% memory.

Interesting note, which may not be relevant: in one ASP, I changed the tier up and then down again without doing a release in between. This did not solve the issue.
The workaround has only fixed the issue if I do at least one release to the ASP at the new tier, before reverting it.

Edit: Should say thanks to @jjavierdguezas for the idea of changing tier

[Kenneth-Abrams]

Experienced the same 409 error issue during deployment starting on 7/27 . Function apps on linux ASP.

[inatlewski]

I'm experiencing the same problem.
2020-07-29T15:06:45.5681580Z ##[error]Failed to deploy web package to App Service. 2020-07-29T15:06:45.5701015Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

[benoutram]

Experiencing the same problem here in a DevOps Release pipeline of App Service and Windows Function App projects. We first saw this error occur on 21/07/20 but seeing several occurrences today. The region for both the App Service and Functions is West Europe.

2020-07-29T09:50:50.1288995Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2020-07-29T09:50:50.4010044Z App Service Application settings are already present.
2020-07-29T09:51:46.6646202Z Package deployment using ZIP Deploy initiated.
2020-07-29T09:52:53.1693953Z ##[error]Failed to deploy web package to App Service.
2020-07-29T09:52:53.1707005Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

A possible workaround seems to be to change the deployment method to Zip deploy, but for one of our Application Services this has increased the deployment time from 1 minute to 21 minutes which is not going to be acceptable.

[jjavierdguezas]

Lets see if they can do something... https://twitter.com/jotajota930621/status/1288503705994764290

[pazcaldj]

Had this issue also several times today... Retried it a couple of minutes ago and succeeded... Seems like its fixed?

[mahmoudhammouda]

Hello , we faced this issue several time today and we used the webdeploy
as workaround.

Le mer. 29 juil. 2020 à 19:19, Pascal de Jonge [e-mail] a
écrit :

Had this issue also several times today... Retried it a couple of minutes
ago and succeeded... Seems like its fixed?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/projectkudu/kudu/issues/3042#issuecomment-665793402,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL67I6UGR4WEPSOCLLIKBATR6BK2JANCNFSM4I3ZILCQ
.

[zorge]

Having the same issue on S1, after upgrading the apps plan to a higher tier the issue is gone and the build succeeds. This is of course is not acceptable, Microsoft!

[suwatch]

I would be great if you all can share the webapp name - this would help us pinpoint.

[grantwar]

I'm having the same issue with multiple function apps, and they are hosted in a P2V2 and in another case in an S1.

[sanchitmehta]

@grantwar Are you still experiencing the same problem ? Could you provide us your app name indirectly ?

[grantwar]

@grantwar Are you still experiencing the same problem ? Could you provide us your app name indirectly ?

@sanchitmehta Yes I'm still experiencing the issue, the app is failing when I'm trying to deploy to both my P2V2 and S1 service plans.

I have another app name if you need it

[jjavierdguezas]

@suwatch I sent you an email with the information to the address in your profile, but since I scaled up the app service we have not seen the error anymore

[sanchitmehta]

@jjavierdguezas Was your app deployed in West Europe ?

[jjavierdguezas]

@jjavierdguezas Was your app deployed in West Europe ?

yes they are

[zorge]

Ive also had my service deployed in West Europe and after scale-up the error was gone. But I'd like to scale down and know what the error was caused by. Unfortunately I cannot give the app name here.

[suwatch]

Please follow this to report site name https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly

@jjavierdguezas I did receive your message - but can't make anything out of the image. Please share the site name (here or indirectly) as well as the UTC time of the incident. We will investigate

[jjavierdguezas]

Please follow this to report site name https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly

@jjavierdguezas I did receive your message - but can't make anything out of the image. Please share the site name (here or indirectly) as well as the UTC time of the incident. We will investigate

In my email I listed the six apps affected,

The subscription id pattern: 58747***-****-****-****-***f432a26a1

We noticed the incident on July 29 around 8:45 am UTC
We did the "fix" around 1:00 pm UTC

Let me know if you need more info

[suwatch]

From our telemetry, we observed below exception happening to multiple sites and vms started a few days ago. We are unable to repro. If anyone is still stuck in this state, please try to go to portal, click restart the site (assuming you can afford site to be down for such short period). Try zipdeploy again - and see if that mitigate the issue.

Error occurred type="error" text="The type initializer for '<Module>' threw an exception." stackTrace="   at Newtonsoft.Json.Converters.BinaryConverter.CanConvert(Type objectType)
   at Newtonsoft.Json.JsonSerializer.GetMatchingConverter(IList`1 converters, Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.InitializeContract(JsonContract contract)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateObjectContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.ResolveContract(Type type)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(String operationName, Stream lockStream) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 213
   at Kudu.Core.Infrastructure.LockFile.Lock(String operationName) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 153" innerText="Attempted to read or write protected memory. This is often an indication that other memory is corrupt." innerStackTrace="   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   at .cctor()

[suwatch]

Update

We are currently investigating. It is likely due to recent update on the VMs leading to this. It affects "West Europe", "West US 2", "North Central US" and "West Central US" regions.

You can mitigate yourself by performing either of the below.

1) Restarting Kudu process. Browse to SCM site -> Process Explorer and Kill w3wp (SCM). The new w3wp will start up automatically. If you have multiple instances, you may need to use instance dropdown to iterate and do the same on all instances.

2) Goto portal. Navigate to site blade, click Restart. This unfortunately will cause short down time for the site.

3) Scale up and scale down. This indirectly restarts SCM sites.

We are also gradually mitigate and will provide ETA once we know it. Apology for the inconvenience.

[i3anaan]

@suwatch Good to know there is being worked on an a fix!
Would it still help you if we shared more information about the apps being affected?

I think this information should give you the ability to pinpoint one of our function apps affected by this issue:

• 2020-07-30T08:02:12Z [Information] Id=eec0c97e-8138-4bd9-81fe-cd233ec4dedf)

(The guide/steps in the docs seems lightly outdated)

[jjavierdguezas]

Today we faced this issue in another subscription, this time I restarted the app (as @suwatch said) and then the release went just fine. So, no scaling is really needed... good news to save money... 😅

[julen-folky]

I had the issue too.
I could not kill the w3wp process because Kudu did not show anything in the process' list.
Restarting the site did not work.
I stop the site, deploy again (using Azure DevOps pipelines), start again. Worked!
Then I tried a new deploy without stopping the site. Also works.

The problem and steps above executed on staging slot. Production slot was not affected by the issue.

[lewis-harris]

I had the issue too, in west europe..
I could not kill the w3wp process because Kudu did not show anything in the process' list.
Restarted the site and deployed again succesfully

[julen-folky]

FWIW, @lewis-harris I have the site in west europe too.

[remcorakers]

We received the same issue since Friday last week for one of our App Services in West Europe, using Zip-Deploy. We tried to do the following:

1. Restart the App Service and re-deploy our code via our Azure build pipeline => still received the 409 error
2. Stop the App Service re-deploy our code via our Azure build pipeline => still received the 409 error
3. Kill all w3wp processes in the SCM console and re-deploy our code via our Azure build pipeline => still received the 409 error
4. Delete the App Service, re-deploy the App Service via our ARM templates, stop the App Service, and re-deploy our code via our Azure build pipeline => success!

So only deleting the entire App Service and re-deploying fixed the issue.

[kayhantolga]

Had the same issue
** Scaled up app service, stop and wait a bit, run again, run the deployment => worked fine

I tried scaling up and down then deployment, didn't work for me.

[yoelOhayon]

Same issue ,
could not kill the w3wp process because Kudu did not show anything in the process list.
restart the app changed nothing

I added a new slot and configure my Azure Devops Release to the new slot and it worked !

[sympthom]

This is all of the sudden a problem for us (this has not been a problem since 1.5 years). Why do we get "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying to WE using zip deploy? How can we do to fix it? This is not a descibable error message.

[suwatch]

@sympthom did you try the mitigation mentioned above?

[atmike]

I have the same problem.
but i could solve it by stopping the app-service, starting it again and then restarting it 3 times in quick succession. This seems to stop the w3wp and the release is working again.

[Gismork]

We started experiencing this issue on one App Service on July, 29th hosted in West Europe.
We are now seeing this issue on other App Service.

We were able to workaround this issue by killing the Kudu process and redeploying our application again on the App Service. We use Azure DevOps to deploy our applications (using the Run from package deployment mode).

[abercea]

I have the same problem.
but i could solve it by stopping the app-service, starting it again and then restarting it 3 times in quick succession. This seems to stop the w3wp and the release is working again.

This also worked for me (azure func on win, run from package deploy)

[paulhickman-a365]

I have a pipeline that sequentially deploys:

• 4 .net core azure function apps to app service plan 1 on elastic premium
• 4 .net core web applications to app service plan 2 on standard 1
• 4 static html sites to app service plan 3 also on standard 1

All are using zip deploy, and we intermittently get this error on any of the 12 apps. We can usually work around it by stopping the failing app and then re-running the deployment.

[Tasboo]

I ran into the same error when deploying a function app. "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)"

After several failed attempts, we restarted the function app through the Azure portal. After it the function app finished restarting the deploy finally worked.

[timg83]

I also ran into the Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" today. Kudu did not respond when clicking Debug Console > CMD. Got the "Azure Functions Runtime is unreachable" notification on the Azure Function overview.

What did work for us was to create a new slot, deploy to that slot, swap it to production. After this I removed the new slot and redeployed without using slots and this was successful.

[pietersap]

Confirmed that scaling the app service plan up and down works. Could not kill the w3wp process since the process explorer in Kudu wouldn't load.

Scaled up my B1 app service plan to B2, did the deployment and succeeded, scaled down to B1 and afterwards the deployments continued succeeding.

[pietersap]

edit: killing w3wp processes also works

[arynaq]

Any updates here? Is less than ideal uncertainty for deployment of business critical software.

[suwatch]

Are you repeatedly running into issue? If so, do share the App Service name. The expected is you should run into it once. As for rollout, we are currently pushing out the fix. It should be done by end of the month.

[arynaq]

Are you repeatedly running into issue? If so, do share the App Service name. The expected is you should run into it once. As for rollout, we are currently pushing out the fix. It should be done by end of the month.

We managed to circumvent this by doing the scale up/down approach. It is not a persistent issue but we have many different apps where this has happened and we don't always have direct access to them (outsourced ops) to do this.

It is just an inconvenience/annoyance that costs time (therefore money) though the workarounds work.

[suwatch]

Totally understand the pain and appreciate the patience. We will continue to roll out fix to address this - ETA end of Aug 2020.

[J0rgeSerran0]

Same issue here

Linux machine running an ASP.NET Core 2.2 web app at West Europe

The Release Azure DevOps process to deploy the App on Azure was running successfully, but suddenly we have received an error during the automatic deploy process that is being executed on Azure DevOps:

_##[error]Failed to deploy web package to App Service.

[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

[warning]Error: Failed to update deployment history. Error: Bad Request (CODE: 400)_

My steps to fix the problem as workaround:

• Scale up the App Service Plan on Azure
• Execute from Azure DevOps the Release process manually
• Scale out the App Service Plan again.

This thread and comments are correct and saved my time and problems in a critical issue.

Thanks all.

[NickTolhurst26]

We're also experiencing this issue and are currently manually invoking the scale-up/scale-down workaround for a couple of our client's production apps.

@suwatch - Was the fix for this rolled out on time?

[suwatch]

First off, this should not impact Linux App Service - if you encounter this on Linux, please open a service ticket or http://aka.ms/CRI-WebApps. As for Windows, you should only experience this during deployment and, once manual mitigation by restart/scale, you should not encounter it again. As for the rollout, we have fix about 50% on the envs - it may take another week or so to rollout to the rest. Apology for the delay.

[tomkuijsten]

2 weeks further, we got plenty of 409's today, is the fix on all envs @suwatch ?

[suwatch]

Umm .. you might be running into different issue. could you share a few incidents, site names and UTC time of operations?

[panuoksala]

I'm having the same problem and scaling from S1 to S2 solved the issue... This is one easy way to make more money.
I'm using the Linux App Service and deploying Wordpress site with Azure App Service Deploy task.

[suwatch]

It would be best to create a new gissue with details for us to help investigate. As far as the main regression goes, it only impacted Windows and fix were rolled out completely weeks ago. Yours must be something new or transient issue.

[delroyh]

I'm having this issue all of a sudden. Deploys were fine for forever (at least the last year), then all of a sudden starting to see this. A quick google search brought me to this thread.
Repeated deployment attempts _eventually_ succeed. I haven't tried the scale up and down solution as a work around yet.
If I find some motivation in the next few days I'll see if I can gather some more details to throw in the pile here.

[suwatch]

I would suggest to open a new gissue with all the necessary info.

[ikuosaito1989]

This problem occurred on East Japan and B1 instance of Azure App Service on Linux. I would like to know how to solve this problem other than scaling up.

More information can be found in the following issue.

https://github.com/Azure/webapps-deploy/issues/98

[JV-conseil]

Error: Failed to update deployment history. Conflict (CODE: 409)

• Location: West Europe
• App Service Plan: P2v2: 1
• Operating System: Linux
• Build: GitHub Action

[group]Run azure/webapps-deploy@v2
with:
  app-name: some-app-name
  slot-name: dev
  publish-profile: ***
  package: .
env:
  pythonLocation: /opt/hostedtoolcache/Python/3.8.7/x64
[endgroup]
Package deployment using ZIP Deploy initiated.
[error]Failed to deploy web package to App Service.
[error]Deployment Failed with Error: Error: Failed to deploy web package to App Service.
Internal Server Error (CODE: 500)
[warning]Error: Failed to update deployment history.
Conflict (CODE: 409)

[nertim]

@JV-conseil,

Are you still running in to this issue? If so, can you please open a new GitHub issue with following information:

1. Site/slot names
2. latest Location, OS
3. latest GitHub Action log

A new GitHub issue will help us manage investigation of the specific issues a little better.

[delroyh]

I'm having this issue all of a sudden. Deploys were fine for forever (at least the last year), then all of a sudden starting to see this. A quick google search brought me to this thread.
Repeated deployment attempts _eventually_ succeed. I haven't tried the scale up and down solution as a work around yet.
If I find some motivation in the next few days I'll see if I can gather some more details to throw in the pile here.

I wanted to circle back on this (after getting some notifications about these recent comments on the thread)
My issues were resolved by fixing a completely unrelated issue that _happened_ to manifest this way. It's been...a while...since I got it fixed and can't for the life of me remember the specifics, other than it had absolutely nothing to with Kudu despite the issue manifesting as a Kudu deploy failure.
So, not the most useful update, but I don't have problems anymore 😂

[vip32]

ran into the issue today, gosh (linux webap, B2)!? scaling up to B3 really did the trick

[martinlarosa]

I believe that's pretty random @vip32
I sometimes fixed it like that but then also made it work with lower tiers. Also had it fail with higher tiers.

I "fixed"it completely by going to the pipeline and forcing it to use web deployment:

[emmanuelvr]

I had this issue and all I did was go into the Deployment Center, click on "Disconnect" and then re-deploy and it worked

[nikidandwani]

Ran into the same issue today in west europe!

[adriannasui]

Still happenning to us on linux app service

[adriannasui]

ok. Found the issue - it is related to our app service plan storage limit - we were right at the limit of 10GB due to some logs.
This would explain why changing the app service plan seems to be a very good workaround to this issue.
Would be great if KUDU shows a proper error for this, seems that there are many people spending many hours investigating these issues.

Still happenning to us on linux app service

[scottrudy]

We just ran into this issue on a production service and managed to resolve it in the short term. There are few things going on. First, there is a storage limit across all app services in a region and resource group. In our case we hit 500 GiB across all the app service plans. Our plans were all P3v2, so we had no ability to scale up/down when this happened, as we were at the highest tier.

In addition, we were trying to run multiple DevOps pipelines to multiple applications at the same time. When a deployment happens some storage is allocated to allow for the deployment to complete. In our case this allocation was enough to put us over the limit. We found if we continually monitored the storage quota across the region and resource group it eventually freed up enough storage where we were able to deploy.

Our long term fix is to identify the applications that are using a large amount of storage and determine how we can either limit the amount of storage each one of them is using, or provision some of those applications into new resource groups so the limit isn't a concern.