Azure-pipelines-tasks: Stage-scoped Variables Not Respected for Azure CLI Task

Created on 31 Mar 2020 · 8Comments · Source: microsoft/azure-pipelines-tasks

Note

Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo

For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

If you have an issue or request for the Azure Pipelines service, use developer community instead:

https://developercommunity.visualstudio.com/spaces/21/index.html )

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug
Enter Task Name: AzureCLI@2

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

Server - Azure Pipelines
- If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
Agent - Hosted
- If using Hosted agent, provide agent queue name: Azure Pipelines

Issue Description

It doesn't look like stage-scoped variables are respected when running the Azure CLI task. Is there anything on the roadmap for getting variables defined at the stage/job level supported? I'm trying to make our pipeline more dynamic by passing in variable templates for each environnment. Here is a generic overview of what I'm trying to do:

vars.yml

variables:
  serviceConnectionName: <service connection name>

cli task in template.yml

    - task: AzureCLI@2
      displayName: Disable Storage Account Firewall
      inputs:
        azureSubscription: $(serviceConnectionName)
        scriptType: 'bash'
        scriptLocation: 'inlineScript'
        inlineScript: |
          # Temporarily disable storage account firewall for Terraform access
          az storage account update -g $(appStorageAccountResourceGroupName) -n $(appStorageAccountName) --default-action=allow

And finally the main pipeline.yml, which references vars.yml in the stage definition:

stages:
- stage:
  displayName: Terraform Plan
  variables:
    - template: vars.yml
  jobs:
  - template: templates/template.yml

I would expect this to work, but instead the variable isn't picked up by the pipeline at all, so the value remains as the literal string "serviceConnectionName." This is the error in the Azure DevOps UI when running the pipeline:

There was a resource authorization issue: "The pipeline is not valid. Job TFPlan: Step AzureCLI1 input connectedServiceNameARM references service connection $(serviceConnectionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.,Job TFPlan: Step TerraformTaskV11 input backendServiceArm references service connection $(serviceConnectionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

If I move the variable template files to the top of the pipeline.yml, and thus making them global variables, it works fine:

variables:
  - template: vars.yml
stages:
- stage:
  displayName: Terraform Plan
  jobs:
  - template: templates/template.yml

It might also be worth mentioning @chshrikh replied to another user on an issue with the AzFileCopy task and said that stage-scoped variables were not supported at the moment. Is this something that is common across all tasks or only a few?

Error logs

[Insert error from the logs here for a quick overview]

Core bug

Source

rudolphjacksonm

Most helpful comment

Ah, thanks for clarifying. The problem is that for purposes of resolving resources, the system can only see variables at the pipeline level. In principle we could change this to also include stage variables (and it's worth asking about that on Developer Community). Job-scoped variables will not work because we pause at the start of each stage -- before any jobs have run -- and collect up the resources to be authorized.

vtbassmatt on 1 Apr 2020

👍2

All 8 comments

This is not a bug, you're looking for product support on how to use variables.

Variables are not the same as parameters. Try changing your ${{ parameters.serviceConnectionName }} to ${{ variables.serviceConnectionName }}. If that doesn't work, you'll need to file an issue in Developer Community for support.

vtbassmatt on 1 Apr 2020

Hi @vtbassmatt, apologies, that was a typo in my original question. That should have read as $(serviceConnectionName) instead of referencing that as a parameter. I've already tried what you suggested and that doesn't work. Here is the error I get when using $( ) syntax:

"The pipeline is not valid. Job TFPlan: Step AzureCLI1 input connectedServiceNameARM references service connection $(serviceConnectionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.,Job TFPlan: Step TerraformTaskV11 input backendServiceArm references service connection $(serviceConnectionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

And here is the error I get when using ${{ variables.serviceConnectionName }} syntax:
##[error]Script failed with error: Error: Input required: connectedServiceNameARM

Both of the above scenarios were tested by using stage-scoped variables. If I define the serviceConnectionName variable at the top of the pipeline (not using a template), ${{ }} syntax doesn't work and I get the same error above where it thinks a value was never passed for to the azureSubscription property.

So far the only way I've been able to get this to work is by defining the variable at the top of the pipeline and using $( ) syntax to reference it in the template, or by passing the value in as a parameter. According to the docs I should be able to specify variables at the stage or job scope and they'll work the same as global variables, but that doesn't seem to be the case here.