Azure-pipelines-tasks: Azure App Service Deploy 4.3.0 zip deploy broken

[lachiemurray]

Environment

• Server - VSTS

• Agent - Private (ubuntu-16.04)

Issue Description

Using Azure App Service Deploy 4.3.0 task to deploy a package does not work. This appears to be a regression in 4.3.0 since we have successful deployments using version 4.2.9.

Error logs

2018-08-22T12:15:39.4085490Z ##[section]Starting: Deploy Azure App Service
2018-08-22T12:15:39.4219696Z ==============================================================================
2018-08-22T12:15:39.4234791Z Task         : Azure App Service Deploy
2018-08-22T12:15:39.4251895Z Description  : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2018-08-22T12:15:39.4269792Z Version      : 4.3.0
2018-08-22T12:15:39.4284717Z Author       : Microsoft Corporation
2018-08-22T12:15:39.4301165Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2018-08-22T12:15:39.4316089Z ==============================================================================
2018-08-22T12:15:40.3371032Z Got service connection details for Azure App Service:'<redacted>'
2018-08-22T12:15:42.8677452Z App Service Application URL: <redacted>
2018-08-22T12:15:42.8831994Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_ZIP":"0"}
2018-08-22T12:15:44.3087425Z Updated App Service Application settings and Kudu Application settings.
2018-08-22T12:15:44.3115631Z Package deployment using ZIP Deploy initiated.
2018-08-22T12:15:44.3265211Z ##[error]Unhandled: EISDIR: illegal operation on a directory, read
2018-08-22T12:15:48.7934048Z Fetching changes.
2018-08-22T12:15:49.0716808Z Cleaning up temp folders from previous zip deployments and extracting pushed zip file D:\local\Temp\zipdeploy\bohkjgxu.zip (0.00 MB) to D:\local\Temp\zipdeploy\extracted
2018-08-22T12:15:49.0732278Z Central Directory corrupt.
2018-08-22T12:15:49.0768705Z ##[error]Failed to deploy web package to App Service.
2018-08-22T12:15:49.0824994Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2018-08-22T12:15:51.1259489Z ##[warning]Can\'t find loc string for key: FailedToUpdateApplicationInsightsResource
2018-08-22T12:15:51.1298708Z Failed to add release annotation. Error: FailedToUpdateApplicationInsightsResource <redacted> undefined - undefined (CODE: 201)
2018-08-22T12:15:51.5233410Z Successfully updated deployment History at <redacted>
2018-08-22T12:15:51.6098126Z ##[section]Finishing: Deploy Azure App Service

[vincent1173]

@lachiemurray , sorry for the issue you are facing.
We found the cause of the issue and working on the fix.

[kwilson803]

A similar error has now cropped up in Azure App Service Deploy 4.3.1.

Server - VSTS
Agent - Private (Win2016)

Error logs

2018-08-23T21:47:00.8517174Z ##[section]Starting: Deploy Azure Application
2018-08-23T21:47:00.8528803Z ==================================================================
2018-08-23T21:47:00.8529324Z Task : Azure App Service Deploy
2018-08-23T21:47:00.8532083Z Description : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2018-08-23T21:47:00.8532702Z Version : 4.3.1
2018-08-23T21:47:00.8532995Z Author : Microsoft Corporation
2018-08-23T21:47:00.8533362Z Help : More Information
2018-08-23T21:47:00.8533801Z ==================================================================
2018-08-23T21:47:02.1495068Z Got service connection details for Azure App Service:'---'
2018-08-23T21:47:03.6960798Z App Service Application URL: http://---.azurewebsites.net
2018-08-23T21:47:03.7098734Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_ZIP":"1"}
2018-08-23T21:47:04.1528015Z Updated App Service Application settings and Kudu Application settings.
2018-08-23T21:47:04.1536863Z Package deployment using ZIP Deploy initiated.
2018-08-23T21:47:04.1822942Z ##[error]Unhandled: EISDIR: illegal operation on a directory, read
2018-08-23T21:47:12.2002685Z Fetching changes.
2018-08-23T21:47:12.4501125Z Central Directory corrupt.
2018-08-23T21:47:12.4510483Z ##[error]Failed to deploy web package to App Service.
2018-08-23T21:47:12.4515932Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2018-08-23T21:47:12.4656982Z ##[section]Finishing: Deploy Azure Application

[kwilson803]

So, A work around for anyone one else who may have this weird issue, here's what I did:

• On the Azure App Service Deploy task in your VSTS definition, Select Version 3.*, dropping down from 4. *(preview)

• In Azure, drop the App Service resource group

• Recreate the App Service resource group in Azure, either manually or through ARM templates

• In VSTS, Create a new release and deploy

• The release should now succeed

It seems to have something to do with the compatibility of the Azure App Service Deploy 4.*(preview) VSTS task and the Azure App Service. Common sense assumption is the App Service Deploy preview version has gotten further ahead of the now much older version of the App Service that was deployed to Azure some time ago. Selecting a stable version of App Service Deploy VSTS task and deploying a more current App Service seems to resolve the issue.

[vincent1173]

@kwilson803 , the bug is a regression in our recent code change.
There is no need to re-create the App Service or release. We are in process of rolling out the fix and will update the thread once the deployment is completed.

[mlinnell]

Could this be related to an issue in Kudu right now where ZipDeploy fails on Fetching Changes? (it gets to Fetching changes and just stops. Not sure how to see a more detailed log view from Kudu).

[vincent1173]

@mlinnell , @kwilson803 , its a regression from VSTS side.
The issue is resolved and the fix is deployed in all the rings. Please do requeue a release for the updated task.

[CopperStarSystems]

I'm still experiencing this issue with V4.3.3. I just ran a full build/release cycle which failed on the Azure App Service Deploy step:

2018-08-27T15:37:50.7738221Z ##[section]Starting: Deploy UI
2018-08-27T15:37:50.7747457Z ==============================================================================
2018-08-27T15:37:50.7747777Z Task : Azure App Service Deploy
2018-08-27T15:37:50.7748253Z Description : Update Azure App Services on Windows, Web App on Linux with built-in images or Docker containers, ASP.NET, .NET Core, PHP, Python or Node.js based Web applications, Function Apps, Mobile Apps, API applications, Web Jobs using Web Deploy / Kudu REST APIs
2018-08-27T15:37:50.7748705Z Version : 4.3.3
2018-08-27T15:37:50.7748938Z Author : Microsoft Corporation
2018-08-27T15:37:50.7749232Z Help : More information
2018-08-27T15:37:50.7749558Z ==============================================================================
2018-08-27T15:37:53.8488403Z Got service connection details for Azure App Service:'[Redacted]'
2018-08-27T15:37:58.6379922Z App Service Application URL: http://[Redacted].azurewebsites.net
2018-08-27T15:37:59.1675263Z Package deployment using ZIP Deploy initiated.
2018-08-27T15:37:59.1778625Z ##[error]Unhandled: EISDIR: illegal operation on a directory, read
2018-08-27T15:38:12.4693030Z Fetching changes.
2018-08-27T15:38:13.0299899Z Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/c8bgktfl.zip (0.00 MB) to /tmp/zipdeploy/extracted
2018-08-27T15:38:13.0305077Z Central Directory corrupt.
2018-08-27T15:38:13.0315668Z ##[error]Failed to deploy web package to App Service.
2018-08-27T15:38:13.0326315Z ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
2018-08-27T15:38:15.6971440Z Successfully updated deployment History at https://[Redacted].scm.azurewebsites.net/api/deployments/2571535384294227
2018-08-27T15:38:15.7067229Z ##[section]Finishing: Deploy UI

[kwilson803]

@vincentdass I've already downgraded to App Service Deploy 3.* and that appears to work without issue.

[vincent1173]

@CopperStarSystems , Can you please share the debug logs (set variable system.debug to true and re-run the release) to [email protected]?

[CopperStarSystems]

@vincentdass Debug logs have been submitted, thanks for your help!

[CopperStarSystems]

@vincentdass Debug logs have been submitted, thanks for your help!

[CopperStarSystems]

@vincentdass Any word on ETA for a resolution? We have been blocked and unable to deploy our application for over a week now. Any information on progress will be greatly appreciated.

[vincent1173]

@CopperStarSystems , the issue is resolved and the fix is rolled out in all the accounts.
Please requeue the release for updated task

[vincent1173]

Closing the issue as the fix is deployed in all the rings.
Summary
A regression was found in App Service Deploy task, where when the package type is folder, Zip Deploy (Windows & Linux) and RunFromZip fails. It was because the folder was not archived before invoking the Zip Deploy API. The fix was to restore the existing flow of archiving the folder and then deploy the same.

[ferantivero]

@vincentdass just wanted to mention that today I faced this issue working with v4 when switching from DeployToSlotorASE from true to false.

In my case, downgrading to v3 did the trick when working without slots.

[vincent1173]

@ferantivero , Can you please share the debug logs to investigate?

[ferantivero]

@vincentdass upgraded back to v4 and turned on debug logs but couldn't reprod.

[ryanspletzer]

I'm seeing 400's now suddenly appear for zip deploy method with 4.3.7. I didn't really change anything.

[ryanspletzer]

<step title="Incoming Request" date="2018-10-19T19:58:39.075" instance="fddb40" url="/api/zipdeploy?isAsync=true&amp;deployer=VSTS&amp;message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%2208ee53dabc6b7baf6d46319b94333704c46eae5d%22%2C%22buildId%22%3A%22220%22%2C%22releaseId%22%3A%2247%22%2C%22buildNumber%22%3A%22220%22%2C%22releaseName%22%3A%22Release-47%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22DirectoryServices-ObjectManagement-FunctionApp%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fadsk-dpe-des-paas.visualstudio.com%2F%22%2C%22teamProject%22%3A%225b535658-a89f-4b99-82c0-297bd50aa2dd%22%2C%22slotName%22%3A%22production%22%7D" method="POST" type="request" pid="7692,2,25" Connection="Keep-Alive" Transfer-Encoding="chunked" Content-Type="application/json; charset=utf-8" Authorization="Bas..." Host="objectmanagement-usw2-d-func-6g22t4x3ajl7c.scm.azurewebsites.net" User-Agent="VSTS_cbcb75ec-f836-4041-bb32-ebd50bb954ec_release_1_47_95_1" X-WAWS-Unencoded-URL="/api/zipdeploy?isAsync=true&amp;deployer=VSTS&amp;message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%2208ee53dabc6b7baf6d46319b94333704c46eae5d%22%2C%22buildId%22%3A%22220%22%2C%22releaseId%22%3A%2247%22%2C%22buildNumber%22%3A%22220%22%2C%22releaseName%22%3A%22Release-47%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22DirectoryServices-ObjectManagement-FunctionApp%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fadsk-dpe-des-paas.visualstudio.com%2F%22%2C%22teamProject%22%3A%225b535658-a89f-4b99-82c0-297bd50aa2dd%22%2C%22slotName%22%3A%22production%22%7D" CLIENT-IP="52.34.1.16:52839" IS-SERVICE-TUNNELED="0" X-ARR-LOG-ID="8ec13938-17ec-4533-972f-221698b3cce1" X-SITE-DEPLOYMENT-ID="objectmanagement-usw2-d-func-6g22t4x3ajl7c" WAS-DEFAULT-HOSTNAME="objectmanagement-usw2-d-func-6g22t4x3ajl7c.scm.azurewebsites.net" X-Forwarded-Proto="https" >
  <step title="Cleanup Xml Logs" date="2018-10-19T19:58:39.229" /><!-- duration: 128ms -->
  <step title="ZipPushDeploy" date="2018-10-19T19:58:39.571" >
    <step title="Reading the zip URL from the request JSON" date="2018-10-19T19:58:39.591" /><!-- duration: 48ms -->
  </step><!-- duration: 126ms -->
  <step title="Outgoing response" date="2018-10-19T19:58:39.727" type="response" statusCode="400" statusText="Bad Request" Server="Microsoft-IIS/10.0" x-ms-request-id="8ec13938-17ec-4533-972f-221698b3cce1" Cache-Control="private" X-AspNet-Version="4.0.30319" Content-Type="application/json; charset=utf-8" /><!-- duration: 31ms -->
</step><!-- duration: 749ms -->

[hackked]

I too am now seeing this same issue when using zip deploy and version 4.3.7. I know it was working as recent as the 16th without issue.

[jmsalvo]

I am also seeing issues with zip deploy and version 4.3.7. The deployment process was working last Friday, but now I get a vague error "Failed to deploy web package to App Service. Bad Request (CODE: 400)". I updated the VSTS Azure App Service Deploy task to use version 3 but then see a different error: "An error was encountered when processing operation 'Create Directory' on D:\home\site\wwwroot\bin. The error code was 0x800703E6. Invalid access to memory location'.

[eflorespalma]

Same problem here, I had my pipeline configured one month ago, but since 3 or 4 hours ago started to fail in web app deploys(version 4.3.7).. it throw me:

2018-10-22T18:58:23.3366444Z ##[error]Failed to deploy web package to App Service. 2018-10-22T18:58:23.3377987Z ##[error]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)

I create a new web app in azure and i have the result. Please need help with this... whats is happening?

[eflorespalma]

I Solve the problem adding this to the application settings.

KUDU_EXTENSION_VERSION 78.11002.3584

[A100K]

I Solve the problem adding this to the application settings.

KUDU_EXTENSION_VERSION 78.11002.3584

Confirmed - fixed deployment.
Thank you Edgar

[g2udevelopment]

I can confirm the bug in version 4.3.7 of the App Service deployement task.

there is a null reference exception in the zip deployement step.

2018-10-23T06:57:16.2461922Z ##[debug]ZIP Deploy response: {"statusCode":400,"statusMessage":"Bad Request","headers":{"cache-control":"no-cache","pragma":"no-cache","content-length":"836","content-type":"application/json; charset=utf-8","expires":"-1","server":"Microsoft-IIS/10.0","x-ms-request-id":"12eb856b-b9f2-4911-8985-596816d96f51","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET","set-cookie":["ARRAffinity=3a234e8e230552df7a9a30d300382f0ffb671a7856031027bd689b25d47a9ef8;Path=/;HttpOnly;Domain=stagingdocumenttoolboxv2.scm.azurewebsites.net"],"date":"Tue, 23 Oct 2018 06:57:15 GMT","connection":"close"},"body":{"Message":"An error has occurred.","ExceptionMessage":"Object reference not set to an instance of an object.","ExceptionType":"System.NullReferenceException","StackTrace":" at Kudu.Services.Deployment.PushDeploymentController.<GetZipURLFromJSON>d__10.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 150\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Kudu.Services.Deployment.PushDeploymentController.<PushDeployAsync>d__11.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 164"}} 2018-10-23T06:57:16.2687328Z ##[error]Failed to deploy web package to App Service.

The work around of @eflorespalma works!

[darewreck54]

@eflorespalma where do you specify KUDU_EXTENSION_VERSION. I added as part of the variables pipeline

But i'm still getting the same exception mentioend by @g2udevelopment

Thanks
Derek

[g2udevelopment]

@darewreck54

Add the setting to the application settings of your App Service.

[vincent1173]

@g2udevelopment , @darewreck54 , @A100K , @eflorespalma , @jmsalvo , @jwyoung9

The issue is resolved and the hotfix is completed in all the accounts.
The fix is available in 4.3.8 version of the task. Can you please remove the KUDU_EXTENSION_VERSION application setting and rerun the release?

[nhart12]

@vincentdass Can confirm 4.3.8 fixed the issue for me.

[g2udevelopment]

@vincentdass

When the agents run version 4.3.8 of the task then the issue is fixed (removed KUDU_EXTENSION_VERSION)

In one case a hosted agent still had the 4.3.7 task.

[vincent1173]

Summary
Azure App Service Deploy task - Zip Deploy failed with 400 error. It was because the content type on the request was set to application/json instead of application/octet-steam. Changing to octet-stream content type resolved the issue.

[metin]

Just FYI - I was having same problem when selecting tomcat as runtime stack. I changed it to Java SE and it worked fine.

[FuturistiCoder]

I'm deploying my nodejs app on Version : 4.3.24 and I can still get ##[error]Unhandled: EISDIR: illegal operation on a directory, read

[charlieswing]

My pipeline has been working until yesterday when I began getting this error as well:

Error: EISDIR: illegal operation on a directory, open
2019-10-30T15:25:00.6345048Z An error has occurred during web site deployment.
2019-10-30T15:25:00.6345135Z Kudu Sync failed

I am using Zip Deploy on Linux. Is there a current issue with Kudu Sync?

[madmatt112]

I have been getting the same error(s) shown above, in the last week as well. Seems something was changed recently that may have broken Kudu deployments again. Using Azure Devops AzureRmWebAppDeployment@4

[luiscbraga]

Same issue, version 4.157.4 on Windows agent

2019-12-04T17:29:09.4420751Z Number of entries expected in End Of Central Directory does not correspond to number of entries in Central Directory.
2019-12-04T17:29:09.4479732Z ##[error]Failed to deploy web package to App Service.

[mattwoberts]

Hi.

I'm getting the same issue here. My azure devops version is running version 4.* - here's my yaml:

steps:
- task: AzureRmWebAppDeployment@4
  displayName: 'Azure App Service Deploy'
  inputs:
    azureSubscription: 'xxx'
    WebAppName: xxx
    deployToSlotOrASE: true
    ResourceGroupName: xxx
    SlotName: staging

I also observe that if I try to manually do a zip deploy through kudu, that just hangs on "fetching changes". I observed this on another deployment slot and the only workaround there was to delete the entire deployment slot and recreate it - then it worked fine. But I'd rather not spend days removing and recreating deployment slots.

[Mehdi13k8]

I had same problem on a Kudu deployment, using a Github private repo,
the only way for me was to delete inside of the folder throug ftp...

It was just hanged on "fetching change" and other respons. (was using node 10.19.0)