Kudu: Multiple deployment errors for Python and Dotnet Core Azure Functions

[devedse]

Introduction

For an internal company project we're currently deploying 2 azure functions:

1. The Blob function (Written in C#, being triggered when a write occurs to a blob)
2. The Python function (Written in Python, will be triggered by an Azure Search web request)

During the first deployment everything seems to work (the few times we've tested it). However when we do any subsequent deployments to the Azure Functions we run into multiple deployment errors. I've been trying to figure out what's going on but there seems to be no consistency. Sometimes there's 502 bad gateway errors, sometimes 400 bad request, and sometimes just generic zip deploy failed.

Next to that, deployment times vary wildly as well for deploying the Azure functions. Ranging from a few minutes to 15-20 minutes.

After the 2nd function (python function) is deployed we run a script that extracts the function keys. This works fine during the first deployment but somehow fails sometimes as well. I've build in some logic to retry to obtain the keys but this keeps on failing for about an hour and then times out.

Project structure

We deploy our infrastructure using Terraform. Here's the current relevant Terraform file:

resource "azurerm_storage_account" "azurefunction_storage" {
  name                     = "${lower(var.applicationShortName)}azfuncstor${var.environment}"
  resource_group_name      = azurerm_resource_group.resourcegroup.name
  location                 = azurerm_resource_group.resourcegroup.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_function_app" "azurefunction_python_skill" {
  name                      = "${var.applicationName}-azurefunction-python-skill-${var.environment}"
  location                  = azurerm_resource_group.resourcegroup.location
  resource_group_name       = azurerm_resource_group.resourcegroup.name
  app_service_plan_id       = azurerm_app_service_plan.plan.id
  storage_connection_string = azurerm_storage_account.azurefunction_storage.primary_connection_string

  os_type                   = "linux"
  version                   = "~3"

  site_config {
    always_on               = true

    ftps_state = "Disabled"
  }

  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME"              = "python"
    "WEBSITE_RUN_FROM_PACKAGE"              = ""
    "WEBSITE_NODE_DEFAULT_VERSION"          = "10.14.1"
    "APPINSIGHTS_INSTRUMENTATIONKEY"        = ""
  }

  https_only = true
  identity {
    type = "SystemAssigned"
  }
}


resource "azurerm_storage_account" "azureblobfunction_storage" {
  name                     = "${lower(var.applicationShortName)}azblbfuncstor${var.environment}"
  resource_group_name      = azurerm_resource_group.resourcegroup.name
  location                 = azurerm_resource_group.resourcegroup.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_function_app" "azureblobfunction" {
  name                      = "${var.applicationName}-azureblobfunction-${var.environment}"
  location                  = azurerm_resource_group.resourcegroup.location
  resource_group_name       = azurerm_resource_group.resourcegroup.name
  app_service_plan_id       = azurerm_app_service_plan.plan.id
  storage_connection_string = azurerm_storage_account.azureblobfunction_storage.primary_connection_string

  os_type                   = "linux"
  version                   = "~3"

  site_config {
    always_on               = true

    ftps_state = "Disabled"
  }

  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME"              = "dotnet"
    "WEBSITE_RUN_FROM_PACKAGE"              = ""
    "WEBSITE_NODE_DEFAULT_VERSION"          = "10.14.1"
    "APPINSIGHTS_INSTRUMENTATIONKEY"        = ""
  }

  https_only = true
  identity {
    type = "SystemAssigned"
  }
}

We expected the issue to be due to the different AppSettings (e.g. WEBSITE_RUN_FROM_PACKAGE = 1 or 0) but have not been able to figure out a combination that works.

During every deployment we first deploy Terraform, then we deploy the Azure Functions in Parallel (I also tried to make them deploy sequentially but this didn't solve the problem)

This is how a good deployment looks like:

The build/release is done using yaml files.

Build:

  - job: Build_AzureFunction_Blob
    pool:
          vmImage: 'ubuntu-latest'
    steps:
    - script: |
        dotnet restore
        dotnet build --configuration Release
      workingDirectory: 'src/DocumentMining_AzureFunction_Blob/'
    - task: DotNetCoreCLI@2
      inputs:
        command: publish
        arguments: '--configuration Release --output publish_output'
        projects: 'src/DocumentMining_AzureFunction_Blob/DocumentMining.AzureFunction_Blob.csproj'
        publishWebProjects: false
        modifyOutputPath: false
        zipAfterPublish: false
    - task: ArchiveFiles@2
      displayName: "Archive files"
      inputs:
        rootFolderOrFile: "$(System.DefaultWorkingDirectory)/publish_output"
        includeRootFolder: false
        archiveFile: "$(Build.ArtifactStagingDirectory)/blob$(Build.BuildId).zip"
    - task: PublishPipelineArtifact@1
      displayName: 'Publish Pipeline Artifact: dropAzureBlobFunction'
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)/blob$(Build.BuildId).zip'
        artifact: 'dropAzureBlobFunction'
        publishLocation: 'pipeline'  
  - job: Build_AzureFunction_Python
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - bash: |
        if [ -f extensions.csproj ]
        then
            dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
        fi
      workingDirectory: '/'
      displayName: 'Build extensions'
    - task: UsePythonVersion@0
      displayName: 'Use Python 3.6'
      inputs:
        versionSpec: 3.6 # Functions V2 supports Python 3.6 as of today
    - task: Bash@3
      displayName: 'Install application dependencies'
      inputs:
        targetType: 'inline'
        script: |
          python -m venv worker_venv
          source worker_venv/bin/activate
          pip install -r requirements.txt
        workingDirectory: 'src/DocumentMining_AzureFunction_Python_Skill'
    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: 'src/DocumentMining_AzureFunction_Python_Skill'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true
    - task: PublishPipelineArtifact@1
      displayName: 'Publish Pipeline Artifact: dropAzureFunction'
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
        artifact: 'dropAzureFunction'
        publishLocation: 'pipeline'

Release

- deployment: Deploy_Python_Function
  dependsOn:
    - Run_terraform
  environment: '${{ parameters.ENVIRONMENT }}'
  pool:
    vmImage: 'ubuntu-latest'
  strategy:
    runOnce:
      deploy:
        steps:
        - task: DownloadPipelineArtifact@2
          displayName: 'Download Pipeline Artifact: dropAzureFunction'
          inputs:
            buildType: 'current'
            artifactName: 'dropAzureFunction'
            targetPath: '$(System.ArtifactsDirectory)/dropAzureFunction'
        - task: AzureFunctionApp@1
          displayName: 'Azure functions app deploy'
          inputs:
            azureSubscription: 'Microsoft Azure Enterprise (DS) (06e4a***********8dbc0a40)'
            appType: functionAppLinux
            appName: 'DSDocumentMining-azurefunction-python-skill-${{ parameters.ENVIRONMENT }}'
            package: '$(System.ArtifactsDirectory)/dropAzureFunction/$(Build.BuildId).zip'
        - task: AzureCLI@2
          displayName: 'Export function key to KeyVault'
          inputs:
            azureSubscription: 'Microsoft Azure Enterprise (DS) (06e4a***********8dbc0a40)'
            scriptType: 'bash'
            scriptLocation: 'inlineScript'
            inlineScript: |
              resourceGroup="DS-DocumentMining-${{ parameters.ENVIRONMENT }}"
              echo Resource group:
              echo $resourceGroup

              webAppName="DSDocumentMining-azurefunction-python-skill-${{ parameters.ENVIRONMENT }}"
              echo webAppName:
              echo $webAppName

              resourceId="/subscriptions/$(azureSubscriptionId)/resourceGroups/$resourceGroup/providers/Microsoft.Web/sites/$webAppName"
              echo resourceId:
              echo $resourceId

              n=0
              until [ $n -ge 200 ]
              do
                echo Obtaining keys...
                keys=$(az rest --method post --uri "$resourceId/host/default/listKeys?api-version=2018-11-01") && break
                echo Obtaining keys failed, waiting 15 before retrying...
                n=$[$n+1]
                sleep 15
              done

              echo Keys:
              echo $keys

              key=$(echo $keys | jq .masterKey -r)
              echo Key:
              echo $key

              az keyvault secret set --vault-name "DSDocumentMining-kv-${{ parameters.ENVIRONMENT }}" --name "DSDocumentMining-azurefunction-python-skill-key" --value "$key"
- deployment: Deploy_Blob_Function
  dependsOn:
    - Run_terraform
  environment: '${{ parameters.ENVIRONMENT }}'
  pool:
    vmImage: 'ubuntu-latest'
  strategy:
    runOnce:
      deploy:
        steps:
        - task: DownloadPipelineArtifact@2
          displayName: 'Download Pipeline Artifact: dropAzureBlobFunction'
          inputs:
            buildType: 'current'
            artifactName: 'dropAzureBlobFunction'
            targetPath: '$(System.ArtifactsDirectory)/dropAzureBlobFunction'
        - task: AzureFunctionApp@1
          displayName: 'Azure functions app deploy'
          inputs:
            azureSubscription: 'Microsoft Azure Enterprise (DS) (06e4a***********8dbc0a40)'
            appType: functionApp
            appName: 'DSDocumentMining-azureblobfunction-${{ parameters.ENVIRONMENT }}'
            package: '$(System.ArtifactsDirectory)/dropAzureBlobFunction/blob$(Build.BuildId).zip'

Repro steps

1. Create a new Python or DotNet core Azure Function
2. Deploy the infrastructure using the terraform files mentioned above
3. Deploy the code using the YAML files mentioned above

The log/error given by the failure.

I've tried to add as much logging as possible.

First failed deployment

Terraform log

 [1m  # azurerm_function_app.azureblobfunction[0m will be updated in-place[0m[0m
 [0m  [33m~[0m[0m resource "azurerm_function_app" "azureblobfunction" {
         [1m[0mapp_service_plan_id[0m[0m            = "/subscriptions/06e4a***********8dbc0a40/resourceGroups/DS-DocumentMining-dev/providers/Microsoft.Web/serverfarms/DSDocumentMining-plan-dev"
       [33m~[0m [0m[1m[0mapp_settings[0m[0m                   = {
             "APPINSIGHTS_INSTRUMENTATIONKEY"  = ""
             "FUNCTIONS_WORKER_RUNTIME"        = "dotnet"
           [31m-[0m [0m"WEBSITE_ENABLE_SYNC_UPDATE_SITE" = "true" [90m->[0m [0m[90mnull[0m[0m
             "WEBSITE_NODE_DEFAULT_VERSION"    = "10.14.1"
           [33m~[0m [0m"WEBSITE_RUN_FROM_PACKAGE"        = "1" [33m->[0m [0m""
         }
         [1m[0mclient_affinity_enabled[0m[0m        = false
         [1m[0mdaily_memory_time_quota[0m[0m        = 0
         [1m[0mdefault_hostname[0m[0m               = "dsdocumentmining-azureblobfunction-dev.azurewebsites.net"
         [1m[0menable_builtin_logging[0m[0m         = true
         [1m[0menabled[0m[0m                        = true
         [1m[0mhttps_only[0m[0m                     = true
         [1m[0mid[0m[0m                             = "/subscriptions/06e4a***********8dbc0a40/resourceGroups/DS-DocumentMining-dev/providers/Microsoft.Web/sites/DSDocumentMining-azureblobfunction-dev"
         [1m[0mkind[0m[0m                           = "functionapp,linux"
         [1m[0mlocation[0m[0m                       = "westeurope"
         [1m[0mname[0m[0m                           = "DSDocumentMining-azureblobfunction-dev"
         [1m[0mos_type[0m[0m                        = "linux"
         [1m[0moutbound_ip_addresses[0m[0m          = "13.69.68.62,13.94.144.170,13.81.243.243,13.94.139.19,13.94.138.226"
         [1m[0mpossible_outbound_ip_addresses[0m[0m = "13.69.68.62,13.94.144.170,13.81.243.243,13.94.139.19,13.94.138.226,13.69.121.16,13.80.116.51,13.94.140.153,13.81.244.130,104.40.176.213"
         [1m[0mresource_group_name[0m[0m            = "DS-DocumentMining-dev"
         [1m[0msite_credential[0m[0m                = [
             {
                 password = "*******************"
                 username = "$DSDocumentMining-azureblobfunction-dev"
             },
         ]
         [1m[0mstorage_account_access_key[0m[0m     = (sensitive value)
         [1m[0mstorage_account_name[0m[0m           = "dsdocminazblbfuncstordev"
         [1m[0mstorage_connection_string[0m[0m      = (sensitive value)
         [1m[0mtags[0m[0m                           = {}
         [1m[0mversion[0m[0m                        = "~3"

         auth_settings {
             [1m[0madditional_login_params[0m[0m        = {}
             [1m[0mallowed_external_redirect_urls[0m[0m = []
             [1m[0menabled[0m[0m                        = false
             [1m[0mtoken_refresh_extension_hours[0m[0m  = 0
             [1m[0mtoken_store_enabled[0m[0m            = false
         }

         identity {
             [1m[0midentity_ids[0m[0m = []
             [1m[0mprincipal_id[0m[0m = "892e95e1-a910-4cd6-a20d-998336aa8e22"
             [1m[0mtenant_id[0m[0m    = "***"
             [1m[0mtype[0m[0m         = "SystemAssigned"
         }

         site_config {
             [1m[0malways_on[0m[0m                 = true
             [1m[0mftps_state[0m[0m                = "Disabled"
             [1m[0mhttp2_enabled[0m[0m             = false
             [1m[0mip_restriction[0m[0m            = []
             [1m[0mmin_tls_version[0m[0m           = "1.2"
             [1m[0mpre_warmed_instance_count[0m[0m = 0
             [1m[0muse_32_bit_worker_process[0m[0m = true
             [1m[0mwebsockets_enabled[0m[0m        = false

             cors {
                 [1m[0mallowed_origins[0m[0m     = []
                 [1m[0msupport_credentials[0m[0m = false
             }
         }
     }

 [1m  # azurerm_function_app.azurefunction_python_skill[0m will be updated in-place[0m[0m
 [0m  [33m~[0m[0m resource "azurerm_function_app" "azurefunction_python_skill" {
         [1m[0mapp_service_plan_id[0m[0m            = "/subscriptions/06e4a***********8dbc0a40/resourceGroups/DS-DocumentMining-dev/providers/Microsoft.Web/serverfarms/DSDocumentMining-plan-dev"
       [33m~[0m [0m[1m[0mapp_settings[0m[0m                   = {
             "APPINSIGHTS_INSTRUMENTATIONKEY"      = ""
             "FUNCTIONS_WORKER_RUNTIME"            = "python"
           [31m-[0m [0m"WEBSITES_ENABLE_APP_SERVICE_STORAGE" = "true" [90m->[0m [0m[90mnull[0m[0m
           [31m-[0m [0m"WEBSITE_ENABLE_SYNC_UPDATE_SITE"     = "true" [90m->[0m [0m[90mnull[0m[0m
             "WEBSITE_NODE_DEFAULT_VERSION"        = "10.14.1"
             "WEBSITE_RUN_FROM_PACKAGE"            = ""
         }
         [1m[0mclient_affinity_enabled[0m[0m        = false
         [1m[0mdaily_memory_time_quota[0m[0m        = 0
         [1m[0mdefault_hostname[0m[0m               = "dsdocumentmining-azurefunction-python-skill-dev.azurewebsites.net"
         [1m[0menable_builtin_logging[0m[0m         = true
         [1m[0menabled[0m[0m                        = true
         [1m[0mhttps_only[0m[0m                     = true
         [1m[0mid[0m[0m                             = "/subscriptions/06e4a***********8dbc0a40/resourceGroups/DS-DocumentMining-dev/providers/Microsoft.Web/sites/DSDocumentMining-azurefunction-python-skill-dev"
         [1m[0mkind[0m[0m                           = "functionapp,linux"
         [1m[0mlocation[0m[0m                       = "westeurope"
         [1m[0mname[0m[0m                           = "DSDocumentMining-azurefunction-python-skill-dev"
         [1m[0mos_type[0m[0m                        = "linux"
         [1m[0moutbound_ip_addresses[0m[0m          = "13.69.68.62,13.94.144.170,13.81.243.243,13.94.139.19,13.94.138.226"
         [1m[0mpossible_outbound_ip_addresses[0m[0m = "13.69.68.62,13.94.144.170,13.81.243.243,13.94.139.19,13.94.138.226,13.69.121.16,13.80.116.51,13.94.140.153,13.81.244.130,104.40.176.213"
         [1m[0mresource_group_name[0m[0m            = "DS-DocumentMining-dev"
         [1m[0msite_credential[0m[0m                = [
             {
                 password = "****************"
                 username = "$DSDocumentMining-azurefunction-python-skill-dev"
             },
         ]
         [1m[0mstorage_account_access_key[0m[0m     = (sensitive value)
         [1m[0mstorage_account_name[0m[0m           = "dsdocminazfuncstordev"
         [1m[0mstorage_connection_string[0m[0m      = (sensitive value)
         [1m[0mtags[0m[0m                           = {}
         [1m[0mversion[0m[0m                        = "~3"

         auth_settings {
             [1m[0madditional_login_params[0m[0m        = {}
             [1m[0mallowed_external_redirect_urls[0m[0m = []
             [1m[0menabled[0m[0m                        = false
             [1m[0mtoken_refresh_extension_hours[0m[0m  = 0
             [1m[0mtoken_store_enabled[0m[0m            = false
         }

         identity {
             [1m[0midentity_ids[0m[0m = []
             [1m[0mprincipal_id[0m[0m = "c315e343-2cf2-4584-bebb-2a11308e3596"
             [1m[0mtenant_id[0m[0m    = "***"
             [1m[0mtype[0m[0m         = "SystemAssigned"
         }

         site_config {
             [1m[0malways_on[0m[0m                 = true
             [1m[0mftps_state[0m[0m                = "Disabled"
             [1m[0mhttp2_enabled[0m[0m             = false
             [1m[0mip_restriction[0m[0m            = []
             [1m[0mmin_tls_version[0m[0m           = "1.2"
             [1m[0mpre_warmed_instance_count[0m[0m = 0
             [1m[0muse_32_bit_worker_process[0m[0m = true
             [1m[0mwebsockets_enabled[0m[0m        = false

             cors {
                 [1m[0mallowed_origins[0m[0m     = []
                 [1m[0msupport_credentials[0m[0m = false
             }
         }
     }

Python Function

Azure DevOps log:

Got service connection details for Azure App Service:'DSDocumentMining-azurefunction-python-skill-dev'
Trying to update App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true"}
Updated App Service Application settings and Kudu Application settings.
Package deployment using ZIP Deploy initiated.
Fetching changes.
Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/f1e18e78-6540-4399-9380-fab58cb0338c.zip (263.60 MB) to /tmp/zipdeploy/extracted
An unknown error has occurred. Check the diagnostic log for details.
##[error]Failed to deploy web package to App Service.
##[error]To debug further please check Kudu stack trace URL : https://$DSDocumentMining-azurefunction-python-skill-dev:***@dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
Successfully updated deployment History at https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19361592573312534
App Service Application URL: https://dsdocumentmining-azurefunction-python-skill-dev.azurewebsites.net
Finishing: Azure functions app deploy

The Kudu deployment url:

{
    "id": "19361592573312534",
    "status": 3,
    "status_text": "",
    "author_email": "",
    "author": "***************",
    "deployer": "VSTS",
    "message": "{\"type\":\"Deployment\",\"commitId\":\"033795ae8eae7158dcd7807a36c17eac90d287a3\",\"buildId\":\"1936\",\"buildNumber\":\"20200619.1\",\"repoProvider\":\"TfsGit\",\"repoName\":\"DS-DocumentMining\",\"collectionUrl\":\"https://av*********udionl.visualstudio.com/\",\"teamProject\":\"ce91777b-9d7b-4f5a-bf58-75deec665954\",\"buildProjectUrl\":\"https://av*********udionl.visualstudio.com/ce91777b-9d7b-4f5a-bf58-75deec665954\",\"repositoryUrl\":\"https://av*********udionl.visualstudio.com/DS-DocumentMining/_git/DS-DocumentMining\",\"branch\":\"master\",\"teamProjectName\":\"DS-DocumentMining\",\"slotName\":\"production\"}",
    "progress": "",
    "received_time": "2020-06-19T13:28:32.6339965Z",
    "start_time": "2020-06-19T13:28:32.6339965Z",
    "end_time": "2020-06-19T13:28:32.6339965Z",
    "last_success_end_time": "2020-06-19T13:28:32.6339965Z",
    "complete": true,
    "active": false,
    "is_temp": false,
    "is_readonly": true,
    "url": "https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19361592573312534",
    "log_url": "https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19361592573312534/log",
    "site_name": "dsdocumentmining-azurefunction-python-skill-dev"
}

The log URL (https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19361592573312534/log):

[{"log_time":"2020-06-19T13:28:33.3181054Z","id":"6cfe119a-cfc1-4fef-9a47-5b1e62a0438d","message":"Deployment failed.","type":2,"details_url":null}]

However if we look up that ID in Kudu itself we can find this info:

/home/site/deployments/19361592573312534>cat log.log
2020-06-19T13:28:33.3181054Z,Deployment failed.,6cfe119a-cfc1-4fef-9a47-5b1e62a0438d,2

and

/home/site/deployments/19361592573312534>cat status.xml
<?xml version="1.0" encoding="utf-8"?>
<deployment>
  <id>19361592573312534</id>
  <author>***************</author>
  <deployer>VSTS</deployer>
  <authorEmail />
  <message>{"type":"Deployment","commitId":"033795ae8eae7158dcd7807a36c17eac90d287a3","buildId":"1936","buildNumber":"20200619.1","repoProvider":"TfsGit","repoName":"DS-DocumentMining","collectionUrl":"https://av*********udionl.visualstudio.com/","teamProject":"ce91777b-9d7b-4f5a-bf58-75deec665954","buildProjectUrl":"https://av*********udionl.visualstudio.com/ce91777b-9d7b-4f5a-bf58-75deec665954","repositoryUrl":"https://av*********udionl.visualstudio.com/DS-DocumentMining/_git/DS-DocumentMining","branch":"master","teamProjectName":"DS-DocumentMining","slotName":"production"}</message>
  <progress />
  <status>Failed</status>
  <statusText />
  <lastSuccessEndTime>2020-06-19T13:28:32.6339965Z</lastSuccessEndTime>
  <receivedTime>2020-06-19T13:28:32.6339965Z</receivedTime>
  <startTime>2020-06-19T13:28:32.6339965Z</startTime>
  <endTime>2020-06-19T13:28:32.6339965Z</endTime>
  <complete>True</complete>
  <is_temp>False</is_temp>
  <is_readonly>True</is_readonly>
</deployment>

Blob Funtion

Azure DevOps log

2020-06-19T13:24:16.4775598Z ##[section]Starting: Azure functions app deploy
2020-06-19T13:24:16.4792242Z ==============================================================================
2020-06-19T13:24:16.4792679Z Task         : Azure Functions
2020-06-19T13:24:16.4793147Z Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
2020-06-19T13:24:16.4794484Z Version      : 1.163.7
2020-06-19T13:24:16.4794797Z Author       : Microsoft Corporation
2020-06-19T13:24:16.4795168Z Help         : https://aka.ms/azurefunctiontroubleshooting
2020-06-19T13:24:16.4795598Z ==============================================================================
2020-06-19T13:24:17.3330995Z Got service connection details for Azure App Service:'DSDocumentMining-azureblobfunction-dev'
2020-06-19T13:24:19.4556309Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2020-06-19T13:24:19.4556789Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
2020-06-19T13:24:20.5104776Z Updated App Service Application settings and Kudu Application settings.
2020-06-19T13:24:20.7980850Z Package deployment using ZIP Deploy initiated.
2020-06-19T13:28:08.6634244Z ##[error]Failed to deploy web package to App Service.
2020-06-19T13:28:08.6653584Z ##[error]To debug further please check Kudu stack trace URL : https://$DSDocumentMining-azureblobfunction-dev:***@dsdocumentmining-azureblobfunction-dev.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
2020-06-19T13:28:08.6655945Z ##[error]Error: Error: Failed to deploy web package to App Service. Bad Request (CODE: 400)
2020-06-19T13:28:08.8818976Z App Service Application URL: https://dsdocumentmining-azureblobfunction-dev.azurewebsites.net
2020-06-19T13:28:27.9510299Z ##[section]Finishing: Azure functions app deploy

So here there's no deployment history URL.

so I could not really find out what else is going wrong here.

2nd deployment

Blob now suddenly worked

Python:

Starting: Azure functions app deploy
==============================================================================
Task         : Azure Functions
Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
Version      : 1.163.7
Author       : Microsoft Corporation
Help         : https://aka.ms/azurefunctiontroubleshooting
==============================================================================
Got service connection details for Azure App Service:'DSDocumentMining-azurefunction-python-skill-dev'
Trying to update App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true"}
Updated App Service Application settings and Kudu Application settings.
Package deployment using ZIP Deploy initiated.
##[error]Failed to deploy web package to App Service.
##[error]To debug further please check Kudu stack trace URL : https://$DSDocumentMining-azurefunction-python-skill-dev:***@dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Error: Failed to deploy web package to App Service. Bad Gateway (CODE: 502)
Successfully updated deployment History at https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19391592575494317
App Service Application URL: https://dsdocumentmining-azurefunction-python-skill-dev.azurewebsites.net
Finishing: Azure functions app deploy

3rd deployment

Blob worked

Python function:

Starting: Azure functions app deploy
==============================================================================
Task         : Azure Functions
Description  : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
Version      : 1.163.7
Author       : Microsoft Corporation
Help         : https://aka.ms/azurefunctiontroubleshooting
==============================================================================
Got service connection details for Azure App Service:'DSDocumentMining-azurefunction-python-skill-dev'
Trying to update App Service Application settings. Data: {"WEBSITES_ENABLE_APP_SERVICE_STORAGE":"true"}
Updated App Service Application settings and Kudu Application settings.
Package deployment using ZIP Deploy initiated.
Fetching changes.
Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/22261984-906a-4137-a717-0b95ae2c267d.zip (263.59 MB) to /tmp/zipdeploy/extracted
An unknown error has occurred. Check the diagnostic log for details.
##[error]Failed to deploy web package to App Service.
##[error]To debug further please check Kudu stack trace URL : https://$DSDocumentMining-azurefunction-python-skill-dev:***@dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
Successfully updated deployment History at https://dsdocumentmining-azurefunction-python-skill-dev.scm.azurewebsites.net/api/deployments/19421592579400552
App Service Application URL: https://dsdocumentmining-azurefunction-python-skill-dev.azurewebsites.net
Finishing: Azure functions app deploy

Sometimes the functions do succeed in deployment, but then obtaining the keys using the PS1 script in the Release template fails:

Debug your Azure website remotely.

N/A

Mention any other details that might be useful.

I know this is a ton of things to process, so if someone wants to have a direct look at what's going on, we can basically do teams / screen sharing sessions.

[Rufus1123]

I am working with Devedse on this, and I would like to add one more thing I have noticed in the deployment, which may or may not be relevant:
Besides the two Azure functions, we have 2 app services that deploy into the same app service plan. Therefore, the kind of the plan becomes linux when deployed to Azure, even if I set it to FunctionApp in tf. The Terraform documentation on azure functions however, states the following:

NOTE: This value will be linux for Linux Derivatives or an empty string for Windows (default). When set to linux you must also set azurerm_app_service_plan arguments as kind = "FunctionApp" and reserved = true

Another thing I don't understand is the following line in the Blob function (which does successfully deploy):
2020-06-19T13:24:19.4556789Z Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP"]
To my understanding, this setting has been renamed to "WEBSITE_RUN_FROM_PACKAGE" more than a year ago. We are using the latest azurerm provider in terraform

[Rufus1123]

Finally managed to re-deploy successfully after adding

    "WEBSITES_ENABLE_APP_SERVICE_STORAGE"   = "true"
    "WEBSITE_ENABLE_SYNC_UPDATE_SITE"       = "true"

and solving a deprecation warning by replacing storage_connection_string by storage_account_name and storage_account_access_key. Not sure which of these did the trick, but let's at least hope the deployments keep succeeding
I think it had something to do with certain settings putting the function app in readonly mode