Azure-pipelines-tasks: App Service Deploy 4.3.7 throws 400's for Zip Deploy of Function App

[ryanspletzer]

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Environment

• Server - Azure Pipelines or TFS on-premises?

  • If using TFS on-premises, provide the version:

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number: adsk-dpe-des-paas, DirectoryServices, Release-52

• Agent - Hosted or Private:

  • If using Hosted agent, provide agent queue name:

  • If using private agent, provide the OS of the machine running the agent and the agent version: Windows Server 2016

Issue Description

App Service Deploy 4.3.7 throws 400's for Zip Deploy of Function App (tested on multiple environments, logs attached from one of them). This worked successfully for a number of releases, all with the same artifact, until today (10/19), and at the time those successful releases were using version 4.3.7 as well. Has something changed on the Kudu side with Zip Deploy?

Is it possible to do a Zip Deploy with 3.* version of App Service Deploy? I'd like to continue to use the Zip Deploy method because of the Azure Functions startup speed benefits.

Task logs

logs attached here:
tasklog_9.log

Error logs

2018-10-19T20:55:44.3730939Z Package deployment using ZIP Deploy initiated.
2018-10-19T20:55:44.3734056Z ##[debug]build.sourceVersionAuthor=undefined
2018-10-19T20:55:44.3735166Z ##[debug]build.requestedfor=Ryan Spletzer
2018-10-19T20:55:44.3735924Z ##[debug]build.buildUri=vstfs:///Build/Build/220
2018-10-19T20:55:44.3736624Z ##[debug]release.releaseUri=vstfs:///ReleaseManagement/Release/52
2018-10-19T20:55:44.3737268Z ##[debug]build.buildId=220
2018-10-19T20:55:44.3765247Z ##[debug]release.releaseId=52
2018-10-19T20:55:44.3766181Z ##[debug]build.buildNumber=220
2018-10-19T20:55:44.3766865Z ##[debug]release.releaseName=Release-52
2018-10-19T20:55:44.3767549Z ##[debug]system.TeamFoundationCollectionUri=https://adsk-dpe-des-paas.visualstudio.com/
2018-10-19T20:55:44.3768238Z ##[debug]system.teamProjectId=5b535658-a89f-4b99-82c0-297bd50aa2dd
2018-10-19T20:55:44.3768897Z ##[debug]build.sourceVersion=08ee53dabc6b7baf6d46319b94333704c46eae5d
2018-10-19T20:55:44.3769552Z ##[debug]build.repository.name=DirectoryServices-ObjectManagement-FunctionApp
2018-10-19T20:55:44.3770191Z ##[debug]build.repository.provider=TfsGit
2018-10-19T20:55:44.3770826Z ##[debug]build.buildUri=vstfs:///Build/Build/220
2018-10-19T20:55:44.3792535Z ##[debug]release.releaseUri=vstfs:///ReleaseManagement/Release/52
2018-10-19T20:55:44.3793445Z ##[debug]build.buildId=220
2018-10-19T20:55:44.3794133Z ##[debug]release.releaseId=52
2018-10-19T20:55:44.3794801Z ##[debug]build.buildNumber=220
2018-10-19T20:55:44.3795419Z ##[debug]release.releaseName=Release-52
2018-10-19T20:55:44.3796120Z ##[debug]system.TeamFoundationCollectionUri=https://adsk-dpe-des-paas.visualstudio.com/
2018-10-19T20:55:44.3796878Z ##[debug]system.teamProjectId=5b535658-a89f-4b99-82c0-297bd50aa2dd
2018-10-19T20:55:44.3797553Z ##[debug]build.sourceVersion=08ee53dabc6b7baf6d46319b94333704c46eae5d
2018-10-19T20:55:44.3798258Z ##[debug]build.repository.name=DirectoryServices-ObjectManagement-FunctionApp
2018-10-19T20:55:44.3798905Z ##[debug]build.repository.provider=TfsGit
2018-10-19T20:55:44.3800388Z ##[debug][POST]https://$objectmanagement-usw2-t-func-mzlbxf6av3w7y:***@objectmanagement-usw2-t-func-mzlbxf6av3w7y.scm.azurewebsites.net/api/zipdeploy?isAsync=true&deployer=VSTS&message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%2208ee53dabc6b7baf6d46319b94333704c46eae5d%22%2C%22buildId%22%3A%22220%22%2C%22releaseId%22%3A%2252%22%2C%22buildNumber%22%3A%22220%22%2C%22releaseName%22%3A%22Release-52%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22DirectoryServices-ObjectManagement-FunctionApp%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fadsk-dpe-des-paas.visualstudio.com%2F%22%2C%22teamProject%22%3A%225b535658-a89f-4b99-82c0-297bd50aa2dd%22%2C%22slotName%22%3A%22production%22%7D
2018-10-19T20:55:45.3185820Z ##[debug]ZIP Deploy response: {"statusCode":400,"statusMessage":"Bad Request","headers":{"cache-control":"no-cache","pragma":"no-cache","content-length":"836","content-type":"application/json; charset=utf-8","expires":"-1","server":"Microsoft-IIS/10.0","x-ms-request-id":"2fc5b88f-f3dd-4d63-a7c2-735b4127f0f5","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET","set-cookie":["ARRAffinity=9714196f41068fce281a4026ac9dd72cedb21c3be646cc89032276108ed752b8;Path=/;HttpOnly;Domain=objectmanagement-usw2-t-func-mzlbxf6av3w7y.scm.azurewebsites.net"],"date":"Fri, 19 Oct 2018 20:55:44 GMT","connection":"close"},"body":{"Message":"An error has occurred.","ExceptionMessage":"Object reference not set to an instance of an object.","ExceptionType":"System.NullReferenceException","StackTrace":"   at Kudu.Services.Deployment.PushDeploymentController.<GetZipURLFromJSON>d__10.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 150\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Kudu.Services.Deployment.PushDeploymentController.<PushDeployAsync>d__11.MoveNext() in C:\\Kudu Files\\Private\\src\\master\\Kudu.Services\\Deployment\\PushDeploymentController.cs:line 164"}}
2018-10-19T20:55:45.3189883Z ##[error]Failed to deploy web package to App Service.
2018-10-19T20:55:45.3190576Z ##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy web package to App Service.
2018-10-19T20:55:45.3191608Z ##[debug]Deployment Failed with Error: Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)
2018-10-19T20:55:45.3192460Z ##[debug]task result: Failed
2018-10-19T20:55:45.3193508Z ##[error]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)
2018-10-19T20:55:45.3194016Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)
2018-10-19T20:55:45.3195503Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)
2018-10-19T20:55:45.3198743Z ##[debug]Deployment failed

From the Kudu side:

<step title="Incoming Request" date="2018-10-19T20:35:24.423" instance="c3c4ca" url="/api/zipdeploy?isAsync=true&amp;deployer=VSTS&amp;message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%2208ee53dabc6b7baf6d46319b94333704c46eae5d%22%2C%22buildId%22%3A%22220%22%2C%22releaseId%22%3A%2250%22%2C%22buildNumber%22%3A%22220%22%2C%22releaseName%22%3A%22Release-50%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22DirectoryServices-ObjectManagement-FunctionApp%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fadsk-dpe-des-paas.visualstudio.com%2F%22%2C%22teamProject%22%3A%225b535658-a89f-4b99-82c0-297bd50aa2dd%22%2C%22slotName%22%3A%22production%22%7D" method="POST" type="request" pid="19180,2,31" Connection="Keep-Alive" Transfer-Encoding="chunked" Content-Type="application/json; charset=utf-8" Authorization="Bas..." Host="objectmanagement-usw2-d-func-6g22t4x3ajl7c.scm.azurewebsites.net" User-Agent="VSTS_cbcb75ec-f836-4041-bb32-ebd50bb954ec_release_1_50_104_2" X-WAWS-Unencoded-URL="/api/zipdeploy?isAsync=true&amp;deployer=VSTS&amp;message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%2208ee53dabc6b7baf6d46319b94333704c46eae5d%22%2C%22buildId%22%3A%22220%22%2C%22releaseId%22%3A%2250%22%2C%22buildNumber%22%3A%22220%22%2C%22releaseName%22%3A%22Release-50%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22DirectoryServices-ObjectManagement-FunctionApp%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fadsk-dpe-des-paas.visualstudio.com%2F%22%2C%22teamProject%22%3A%225b535658-a89f-4b99-82c0-297bd50aa2dd%22%2C%22slotName%22%3A%22production%22%7D" CLIENT-IP="52.34.1.16:4772" IS-SERVICE-TUNNELED="0" X-ARR-LOG-ID="be764cee-7e74-45dd-8e42-35a10ce1f672" X-SITE-DEPLOYMENT-ID="objectmanagement-usw2-d-func-6g22t4x3ajl7c" WAS-DEFAULT-HOSTNAME="objectmanagement-usw2-d-func-6g22t4x3ajl7c.scm.azurewebsites.net" X-Forwarded-Proto="https" >
  <step title="Cleanup Xml Logs" date="2018-10-19T20:35:24.454" /><!-- duration: 141ms -->
  <step title="ZipPushDeploy" date="2018-10-19T20:35:24.642" >
    <step title="Reading the zip URL from the request JSON" date="2018-10-19T20:35:24.673" /><!-- duration: 16ms -->
  </step><!-- duration: 63ms -->
  <step title="Outgoing response" date="2018-10-19T20:35:24.736" type="response" statusCode="400" statusText="Bad Request" Server="Microsoft-IIS/10.0" x-ms-request-id="be764cee-7e74-45dd-8e42-35a10ce1f672" Cache-Control="private" X-AspNet-Version="4.0.30319" Content-Type="application/json; charset=utf-8" /><!-- duration: 31ms -->
</step><!-- duration: 375ms -->

[ryanspletzer]

Addendum: I tested the deploy with 4.3.7 to a freshly provisioned Azure Functions instance and used the OOTB settings (just set the bare minimum settings necessary to deploy the function app with the task), and turned off all the other tasks -- it similarly fails.

[ryanspletzer]

This task from the marketplace works: https://marketplace.visualstudio.com/items?itemName=arjanvanrijn.kudu-zipdeploy

[yggdrasil-tynor]

Same issue here, worked fine last week.

[jsheetzati]

Also seeing this error for a normal Azure App Service

[pjirsa]

Workaround documented here. Also a good description of the change to kudu that is causing this to fail.

• https://developercommunity.visualstudio.com/content/problem/363405/azure-app-service-deploy-task-version-437-fails-wh.html
• https://github.com/projectkudu/kudu/issues/2893

[pjirsa]

... and here's the line of code causing the failure when zipDeploy is called
https://github.com/Microsoft/azure-pipelines-tasks/blob/master/Tasks/Common/azure-arm-rest/azure-arm-app-service-kudu.ts#L23

[yggdrasil-tynor]

Setting _KUDU_EXTENSION_VERSION_ to _78.11002.3584_ in Function AppSettings works.

[vincent1173]

@ryanspletzer @ryanspletzer @TynThordin @jsheetzati

The issue is resolved and the hotfix is completed in all the rings.
please remove application setting KUDU_EXTENSION_VERSION (if set before)
Can you please rerun the deployment and share the release status?

[yggdrasil-tynor]

@vincentdass I can confirm it works with Kudo 4.3.8

2018-10-23T12:44:36.5501534Z ##[section]Starting: Deploy Azure Function
2018-10-23T12:44:36.5508016Z ==============================================================================
2018-10-23T12:44:36.5508150Z Task : Azure App Service Deploy
2018-10-23T12:44:36.5508283Z Description : Update Azure App Services on Windows, Web App on Linux with built-in images or Docker containers, ASP.NET, .NET Core, PHP, Python or Node.js based Web applications, Function Apps on Windows or Linux with Docker Containers, Mobile Apps, API applications, Web Jobs using Web Deploy / Kudu REST APIs
2018-10-23T12:44:36.5508415Z Version : 4.3.8
2018-10-23T12:44:36.5508481Z Author : Microsoft Corporation
2018-10-23T12:44:36.5508576Z Help : More information
2018-10-23T12:44:36.5508661Z ==============================================================================
2018-10-23T12:44:37.4321426Z Got service connection details for Azure App Service:'XXX'
2018-10-23T12:44:39.2233157Z App Service Application URL: http://XXX.azurewebsites.net
2018-10-23T12:44:39.2235835Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2018-10-23T12:44:40.2470560Z Updated App Service Application settings and Kudu Application settings.
2018-10-23T12:44:40.2475251Z Package deployment using ZIP Deploy initiated.
2018-10-23T12:45:26.4028833Z Deploy logs can be viewed at https://XXX.scm.azurewebsites.net/api/deployments/XXX
2018-10-23T12:45:26.4029177Z Successfully deployed web package to App Service.
2018-10-23T12:45:26.9372008Z ##[section]Finishing: Deploy Azure Function

[vincent1173]

@TynThordin , Can you please remove the application setting KUDU_EXTENSION_VERSION and rerun the deployment?

[yggdrasil-tynor]

@vincentdass That was with the _KUDU_EXTENSION_VERSION_ removed.

[vincent1173]

Summary
Azure App Service Deploy task - Zip Deploy failed with 400 error. It was because the content type on the request was set to application/json instead of application/octet-steam. Changing to octet-stream content type resolved the issue.