Azure-functions-core-tools: "Permission denied" when publishing python codebase to linux host from ubuntu build agent

[andreasbotsikas]

The Azure Function is a python worker based on the codebase described in this Microsoft doc.
Deployment target running on consumption Y1 SKU, functionapp,linux website kind, python 3.7.

The codebase is hosted in Azure DevOps git. The following Azure DevOps build pipeline is used (currently configured to run on ubuntu-latest):

trigger:
- master

variables:
  # Azure Resource Manager connection created during pipeline creation
  azureSubscription: 'myAzureConnection'
  # Function app name
  functionAppName: 'myAzure-azfn'
  # Agent VM image name. Change the following to 'windows-latest' to work
  vmImageName: 'ubuntu-latest' # 'windows-latest'
  # Working Directory
  workingDirectory: '$(System.DefaultWorkingDirectory)/__app__'
  # Python Version
  pythonVersion: 3.7

stages:
- stage: Package
  displayName: Package stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(workingDirectory)'
        artifact: 'drop'
        publishLocation: 'pipeline'

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Package
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: 'development'
    pool:
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:
          - task: FuncToolsInstaller@0
            inputs:
              version: 'latest'
          - download: current
            artifact: drop
          - task: AzureCLI@2
            displayName: 'Verify that we can read existing deployed functions'
            inputs:
              azureSubscription: $(azureSubscription)
              scriptType: 'pscore'
              scriptLocation: 'inlineScript'
              inlineScript: 'func azure functionapp list-functions  $(functionAppName)'
              workingDirectory: '$(Pipeline.Workspace)/drop/'
          - task: AzureCLI@2
            displayName: 'Deploy new version of codebase'
            inputs:
              azureSubscription: $(azureSubscription)
              scriptType: 'pscore'
              scriptLocation: 'inlineScript'
              inlineScript: 'func azure functionapp publish $(functionAppName) --python'
              workingDirectory: '$(Pipeline.Workspace)/drop/'

This pipeline fails. The pipeline runs using Hosted Agent: Ubuntu16. From the logs of the installing latest func tools:

Finding latest func tools version...
Downloading: https://api.github.com/repos/Azure/azure-functions-core-tools/releases/latest
Latest version is 3.0.2358

The error occurs in the last command func azure functionapp publish $(functionAppName) --python with the following error log:

/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/home/vsts/work/_temp/azureclitaskscript1584291143449.ps1'
Getting site publishing info...
Creating archive for current directory...
Performing remote build for functions project.
Permission denied
##[error]Script failed with error: Error: The process '/usr/bin/pwsh' failed with exit code 1

If I change the agent to Hosted Agent: windows-2019-vs2019 (using windows-latest), everything works as expected.

On the ubuntu worker, I tried upgrading azure CLI to latest but there was no difference.

Is it a possible regression of #1779 ?

[andreasbotsikas]

You can easily reproduce the "Permission denied" issue using the codebase and the yaml pipeline from https://github.com/rndazurescript/python-ml-azure-function repository.

[MichielBijland]

It seems that gozip doesn't have executable permissions on linux.

This was my debug session and fix on a jenkins agent running centos.

`CLI_DEBUG=1 func pack --build-native-deps --python
Directory .python_packages already in sync with requirements.txt. Skipping restoring dependencies...

/usr/local/bin/azure-functions-cli/gozip -base-dir /tmp/jenkins-9c53665c/workspace/ommerce_example-component_master -input-file /tmp/tmpDhJrFL.tmp -output /tmp/tmp0lfu9w.tmp
System.ComponentModel.Win32Exception (13): Permission denied
at Azure.Functions.Cli.Common.Executable.RunAsync(Action1 outputCallback, Action1 errorCallback, Nullable1 timeout, String stdIn) in D:\a\1\s\src\Azure.Functions.Cli\Common\Executable.cs:line 99 at Azure.Functions.Cli.Helpers.ZipHelper.CreateGoZip(IEnumerable1 files, String rootPath, String zipFilePath, String goZipLocation) in D:\a\1\s\src\Azure.Functions.Cli\Helpers\ZipHelper.cs:line 107
at Azure.Functions.Cli.Helpers.ZipHelper.CreateZip(IEnumerable1 files, String rootPath) in D:\a\1\s\src\Azure.Functions.Cli\Helpers\ZipHelper.cs:line 59 at Azure.Functions.Cli.Helpers.PythonHelpers.GetPythonDeploymentPackage(IEnumerable1 files, String functionAppRoot, Boolean buildNativeDeps, BuildOption buildOption, String additionalPackages) in D:\a\1\s\src\Azure.Functions.Cli\Helpers\PythonHelpers.cs:line 303
at Azure.Functions.Cli.Helpers.ZipHelper.GetAppZipFile(String functionAppRoot, Boolean buildNativeDeps, BuildOption buildOption, Boolean noBuild, GitIgnoreParser ignoreParser, String additionalPackages, Boolean ignoreDotNetCheck) in D:\a\1\s\src\Azure.Functions.Cli\Helpers\ZipHelper.cs:line 37
at Azure.Functions.Cli.Actions.LocalActions.PackAction.RunAsync() in D:\a\1\s\src\Azure.Functions.Cli\Actions\LocalActions\PackAction.cs:line 108
at Azure.Functions.Cli.ConsoleApp.RunAsyncT in D:\a\1\s\src\Azure.Functions.Cli\ConsoleApp.cs:line 66`

/usr/local/bin/azure-functions-cli/gozip --help bash: /usr/local/bin/azure-functions-cli/gozip: Permission denied chmod +x /usr/local/bin/azure-functions-cli/gozip /usr/local/bin/azure-functions-cli/gozip --help Usage of /usr/local/bin/azure-functions-cli/gozip: -base-dir string Base dir for the zip archive -dir string dir to zip -input-file string a file containing a list of files to archive -output string Output file name -quiet no output

[rndazurescript]

Thanks @MichielBijland for the suggestion! Function tools are installed in Agent.ToolsDirectory so I had to modify the script to locate the func tool.
As a workaround to this issue, adding the following task in the deployment phase, right after installing the functools seems to solve the specific issue:

          - task: Bash@3
            displayName: 'Allow gozip to execute in non windows environments #1899'
            condition: ne(variables['Agent.OS'], 'Windows_NT')
            env:
              toolsDir: $(Agent.ToolsDirectory)
              architecture: $(Agent.OSArchitecture)
            inputs:
              targetType: 'inline'
              script: |
                funcVersion=$(func --version)
                funcToolsPath="${toolsDir}/func/${funcVersion}/${architecture,,}/"
                echo "Making executable the following file: ${funcToolsPath}gozip"
                chmod +x "${funcToolsPath}gozip"

Note though that now I run into a different issue which may be environment specific

Getting site publishing info...
Creating archive for current directory...
Performing remote build for functions project.
Uploading 4.48 MB []An error occurred while sending the request.
##[error]Script failed with error: Error: The process '/usr/bin/pwsh' failed with exit code 1

[rndazurescript]

Following up on this, it seems that the error was transient. Restarting the pipeline worked, so the workaround seems legit.

/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/home/vsts/work/_temp/azureclitaskscript1584964452041.ps1'
Getting site publishing info...
Creating archive for current directory...
Performing remote build for functions project.
Uploading 4.48 MB []Remote build in progress, please wait...
Updating submodules.
Preparing deployment for commit id '7298e832e1'.
Repository path is /tmp/zipdeploy/extracted
Running oryx build...
...
Source directory     : /tmp/zipdeploy/extracted
Destination directory: /home/site/wwwroot

Python Version: /opt/python/3.7.4/bin/python3

Running pip install...
...
Copying files to destination directory '/home/site/wwwroot'...
Done in 13 sec(s).

Removing existing manifest file
Creating a manifest file...
Manifest file created.

Done in 56 sec(s).
Running post deployment command(s)...
Triggering recycle (preview mode disabled).
Writing the artifacts to a squashfs file
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on /home/site/deployments/functionappartifact.squashfs, block size 131072.

...
Uploading built content /home/site/deployments/functionappartifact.squashfs -> https://azfnnamestg.blob.core.windows.net/scm-releases/scm-latest-azfnname.zip?sv=2014-02-14&sr=b&sig=xyz&sp=rw
Resetting all workers for azfnname.azurewebsites.net
Deployment successful.
Remote build succeeded!
Syncing triggers...
Functions in azfnname:
    classify - [httpTrigger]
        Invoke url: https://azfnname.azurewebsites.net/api/classify?code=xyz

/usr/bin/az account clear

[deepbass]

Hi this has completely scuppered me - I'm attempting the fix now but its pretty critical that I can't do a release from a linux agent to a python function app. Is this being addressed soon?

[anthonychu]

@ankitkumarr Do you know if this will be fixed in an upcoming release?

[ankitkumarr]

Looks like the issue tagged was assigned recently, I added another comment there. Once that is fixed, this should be resolved.

In the meanwhile, you can manually add a step for chmod a+x /usr/local/bin/azure-functions-cli/gozip in your Devops pipeline before you run func azure functionapp publish command to mitigate the problem.

[zsolt-halo]

The only thing worked for me was to add a new bash task to install the func-tools and remove the other one offered by MS:

curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg

sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-get update

sudo apt-get install azure-functions-core-tools-3

[kensykora]

Thanks @zsolt-halo, that workaround did the trick!