Azure-pipelines-tasks: Bug: Failed Successful Deployment - Error Handling AzureFunctionAppV1

[damienwebdev]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: AzureFunctionAppV1

Environment

Azure Pipelines

• Account: lostgolfballs
• Project Name: www.lostgolfballs.com
• BuildId: releaseId=1101
  

  Issue Description

Upon running a AzureFunction Task with a zip file, I anticipate that I update an existing function app with the new package files.

Instead, I get a blue screen that looks like:

If I re-run the release, I get my function loading successfully after the task completes.

Task logs

Got service connection details for Azure App Service:'MY_SERVICE'
2019-09-26T19:38:57.8311929Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2019-09-26T19:38:57.8312354Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2019-09-26T19:39:01.7703054Z Package deployment using ZIP Deploy initiated.
2019-09-26T19:39:22.8310452Z Successfully deployed web package to App Service.
2019-09-26T19:39:28.6247890Z Successfully updated App Service configuration details
2019-09-26T19:39:28.9075674Z App Service Application URL: MY_SERVICE_URL

Other Information:

When it successfully fails, the resulting app content looks like:

Contents of Web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name = "Site Unavailable" stopProcessing = "true">
                    <match url = ".*" />
                    <action type = "CustomResponse" statusCode = "503" subStatusCode = "0" statusReason = "Site Unavailable" statusDescription = "Could not download zip" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Contents of FAILED TO INITIALIZE RUN FROM PACKAGE.txt:

Run From Package Initialization failed.

[damienwebdev]

Turns out, the bug is due to the task running as Zip Deploy but still setting WEBSITE_RUN_FROM_PACKAGE=1.

If I manually remove the WEBSITE_RUN_FROM_PACKAGE this is fixed.

[SumiranAgg]

@damienwebdev post deploy action is not supported with RUN_FROM__PACKAGE deployment technique, since it makes code mounted in site/wwwroot read only. If you are trying to write to the app files, I suggest using zipdeploy. You can select the option from additional deployment options section in the task.

[damienwebdev]

@SumiranAgg I'm not sure I understand. If it's not supported, then why does running the release more than a single time cause it to intermittently work as anticipated?

Additionally, I tried setting a few releases to use Zip Deploy and it resulted in the same problem.

Edit: after re-reading what you wrote, I'm not using a post-deploy action.

I'm merely taking a folder or zipped archive (depending on the version of the release I'm using) and trying to deploy it to my Azure Function App. Neither "Deployment Configuration" setting appears to work 100% of the time and even more curiously, this process has been without issue for the past few months until a few weeks ago so I'm assuming something broke in DevOps or Kudu in that time.

[damienwebdev]

This is potentially related to #10961

[eaarora-ms]

@damienwebdev We have made some changes which might potentially fix the issue. We will let you know the updates.

[dlyson]

I have a similar problem while deployment of my function app. It noticed that at Friday 29 Nov. At that time it didn't upload the zip file to data\Site Packages and it mounted the site\wwwroot to an empty folder.
If any ZIP file existed in the data\Site Packages then it silently used the latest package despite the fact that new deployment has already overriden app settings. I used Deployment method = auto-detect. Here are logs from such a scenario: (false positive logs)

2019-11-29T16:12:02.5560151Z ##[section]Starting: Deploy Functions
2019-11-29T16:12:02.5685150Z ==============================================================================
2019-11-29T16:12:02.5685305Z Task         : Azure Functions
2019-11-29T16:12:02.5685409Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2019-11-29T16:12:02.5685525Z Version      : 1.158.0
2019-11-29T16:12:02.5685615Z Author       : Microsoft Corporation
2019-11-29T16:12:02.5685710Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
2019-11-29T16:12:02.5685833Z ==============================================================================
2019-11-29T16:12:03.5981278Z Got service connection details for Azure App Service:'myapp-copy-eus2-functions'
2019-11-29T16:12:04.2934343Z Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2019-11-29T16:12:04.2934540Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2019-11-29T16:12:07.1392639Z Package deployment using ZIP Deploy initiated.
2019-11-29T16:12:16.8948463Z Successfully deployed web package to App Service.
2019-11-29T16:12:19.0313435Z Successfully updated App Service configuration details
2019-11-29T16:12:23.3088598Z Successfully added release annotation to the Application Insight : myapp-copy-eus2-appInsights
2019-11-29T16:12:23.4540492Z App Service Application URL: http://myapp-copy-eus2-functions.azurewebsites.net
2019-11-29T16:12:23.4751098Z ##[section]Finishing: Deploy Functions

Today I restarted the deployment and it generated the FAILED TO INITIALIZE RUN FROM PACKAGE.txt file which looks quite similar to the current bug.
I enforced Deployment method = Zip Deploy but I encountered another error that I reported in kudu issues.

I deploy Function App v2 with .net core code. It's using a consumption plan in eastus2.
I'm using a Hosted VS 2017 agent (agent version: '2.160.1, image version: '20191028.1').

[damienwebdev]

@eaarora-ms What are the specific changes (can you point me to a PR), and do you know when they go out for release/preview?

[eaarora-ms]

I have a similar problem while deployment of my function app. It noticed that at Friday 29 Nov. At that time it didn't upload the zip file to data\Site Packages and it mounted the site\wwwroot to an empty folder.

This is an issue with one of the REST APIs (Update App Settings API to be specific) and we are working with the App service team to get this resolved. You can check the update here: https://github.com/microsoft/azure-pipelines-tasks/issues/11661

Sorry that this is taking longer than expected.

[eaarora-ms]

@eaarora-ms What are the specific changes (can you point me to a PR), and do you know when they go out for release/preview?

Hi, you mentioned in the issue that If you manually remove the WEBSITE_RUN_FROM_PACKAGE this is fixed. We fixed it in https://github.com/microsoft/azure-pipelines-tasks/pull/11821

Please let me know if you're still facing the issue as deployment must have been done by now.

[damienwebdev]

@eaarora-ms I'm still having the issue. It's still very intermittent. I do about ~20 deployments to function apps a day, and I'd say about ~1-2 have this issue.

[HansOlavS]

@damienwebdev I experienced the exact same issue today. I've been stuck for 4 hours trying to figure out why a newly created Azure Functions app wouldn't get deployed correctly. After setting the runtime to ~2 (the newly created app has ~3 as default) then zipdeploy started working for me!

[markgar]

I was having this issue but I found that I just needed to get the function app runtime version to be the same as what I set in the csproj. I think that the function app could probably detect this and give a better error message, maybe?

[mjisaak]

I face the same issue when I deploy an Azure Function with an ARM template that uses the listkey function and the setting WEBSITE_RUN_FROM_PACKAGE=1.

I posted a question on stackoverflow that contains the ARM template I am using to reproduce this issue.

[eaarora-ms]

@eaarora-ms I'm still having the issue. It's still very intermittent. I do about ~20 deployments to function apps a day, and I'd say about ~1-2 have this issue.

Hi,
Since 1-2/20 deployments are failing, this might be a network issue. We request you please open an issue with the kudu team. Link: https://github.com/projectkudu/kudu/issues

[eaarora-ms]

I have a similar problem while deployment of my function app. It noticed that at Friday 29 Nov. At that time it didn't upload the zip file to data\Site Packages and it mounted the site\wwwroot to an empty folder.

@dlyson The issue is happening since the first deployment to root (site/wwwroot) is done via Run From Package and deployment to virtual directory is done via Web Deploy. Please try deploying first time using Web Deploy as well and share results. You can go to "Additional Deployment Options" section and select Web Deploy and continue.

[eaarora-ms]

I was having this issue but I found that I just needed to get the function app runtime version to be the same as what I set in the csproj. I think that the function app could probably detect this and give a better error message, maybe?

@markgar We will add this to our enhancement lists. Thanks for reporting.

[eaarora-ms]

@mjisaak I see you have opened an issue with azure-functions-host. Please create a new issue in this repo if you don't get a solution.

[qJake]

I am experiencing the same issue when deploying to Azure App Service from DevOps. I am using all default settings, and providing the .zip file as the artifact from the dotnet publish command.

Sometimes, my websites will not load. It is intermittent. Refreshing/recycling the website does not work. I have found that the Azure App Service Deploy task is in fact setting WEBSITE_RUN_FROM_PACKAGE to 1 for all new deployments. Previous deployments that we have from TFS 2017 were not setting this flag.

I can also confirm that, on a broken site, removing the WEBSITE_RUN_FROM_PACKAGE app setting causes the website to load again.

This is what the error looks like in Azure:

Is this a problem with Azure App Service or the App Service Deploy task? How can I ensure my deployments don't break like this?

[damienwebdev]

@eaarora-ms I can't confirm that this bug was fixed, so I don't really know why the issue was closed. I'm still having failing deployments on v1.

[nofield]

I am also intermittently seeing this issue. It appears most often on an initial deployment of the Function App, or seems to be, anyway.

[saleelpk95]

+1
Only after I manually delete the WEBSITE_RUN_FROM_PACKAGE setting, the function becomes visible in the portal. I believe that this setting is being added automatically. I had deployed using ARM templates and I didn't have this setting in it. If this setting is enabled by default, is there a way to override and disable it using ARM templates?

[dgard1981]

For us the issue is that we want to run from a package so have deliberatly set the App Setting to WEBSITE_RUN_FROM_PACKAGE = 1, but rather then deploying the zip file to /data/SitePackages, it's being extracted to /site/wwwroot. Obvioulsy removing the App Setting would cause the Function App to look in /site/wwwroot, but that's not what we want.

I'm not sure however that the fault lies with Azure DevOps - I think it's more likely a Kudu issue.

[saleelpk95]

Cleaning up the resource group and redeploying worked for me. It was weird but I was shocked when the same release from Azure DevOps deployed correctly to one resource group while it didn't to the other. The release itself was successful. Simply tried cleaning up the resource group and it worked.

[qJake]

@saleelpk95 Can you verify what you mean by "clean up the resource group"? What differences are there between one that is broken vs. one that is good?

We have a number of resource groups that are manually created/maintained, not deployed automatically as templates, so I would be interested to know what tasks we can complete to potentially fix this issue.

[saleelpk95]

@qJake
Clean up means deleting all the resources within the resource group. I thought I'd have to recreate the resource group as I have read in other threads. But fortunately removing the resources worked.

[Roshan-Kumar-Microsoft]

@saleelpk95 If you still face same issue please let us know . Closing this issue.

[synergiator]

Side note: I get same blue screen on testing standard Azure Function App Hello World tutorials for NodeJS and Python.

Paths tried:

• restart
• reuse existing resource group/storage
• recreate with new resource group/storage (relying on the hint above to cleanup)

[Ravi-Macha]

I am facing same issue where I am doing "deploymentMethod: runFromPackage" and setting WEBSITE_RUN_FROM_PACKAGE to 1.

Why its deploying as ZIP ? My Zip file in /sitepackages is valid one. Same code is deploying in another environment.

[WizX20]

Facing this issue today with deploy-as-package (zip) to Web App's. Multiple times, on different resource groups on different Web Apps. This is hindering us in deploying to production environments!

YAML:

- task: AzureWebApp@1
    displayName: 'Deploy Web App ''${{ parameters.artifactName }}'''
    inputs:
      azureSubscription: $(connectionName)
      appName: ${{ parameters.webAppName }}
      package: '$(Pipeline.Workspace)/${{ parameters.artifactName }}/**/*.zip'
      enableCustomDeployment: true
      deploymentMethod: runFromPackage