Azure-devops-docs: Dynamic Variable Groups?

[jlhjr]

Is there a way to reference a variable group dynamically so that it can be selected when the build is queued? For instance, in the pipeline YAML:

variables:

• group: '$(VARIABLE_GROUP_NAME)'

The use case is to create a variable group per environment. I realize another approach would be to create a release pipeline and scope each variable group to a different stage, but for a variety of reasons I prefer a single YAML file, and this seems to be the way Azure DevOps is moving anyway.

Is it supported yet? Thanks!

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 4827aa99-ae62-bd63-6f2f-a87a4065ed27
• Version Independent ID: c9e461ee-7f42-3503-9460-18fa3a807bbb
• Content: Variable groups for Azure Pipelines and TFS - Azure Pipelines
• Content Source: docs/pipelines/library/variable-groups.md
• Product: devops
• GitHub Login: @alexhomer1
• Microsoft Alias: ahomer

[BevanG]

I would like this too. I have multiple build files that are simply passing variables through to templates.

If possible, and I assume it is not, it would be good to extend this to be able to define variables in the include and exclude rules for build triggers. Would result in multiple builds with Variables defined all pointing to a single YAML file.

[alexhomer1]

Thank you for your feedback. I have passed this on to the product team for advice.

[ashokirla]

@jlhjr @BevanG : This is not supported today. You need to declare the variable group you want to consume in a YAML pipeline. It is the same case with variables in build trigger definition.

We will look into this feedback.
Do you expect to change the variable group for every pipeline run or do you intend to maintain a single YAML file and set the variable group value once per pipeline definition using that YAML file.

[jlhjr]

Understood, thanks for considering this. If I've understood your question correctly, ideally I would like the option to set the variable group for each run. I would still like to use a single YAML file, but I don't want to hardcode the variable group name in that YAML file.

[danieleusebio]

I have the same need. Please deploy the feature. Thank you.

[alexche8]

Would be helpful to have such functionality. For example for simple pipelines files - I want to have one pipeline file that can be responsible for builds of two branches. And depends on which branch has triggered the build - the associated group of variables can be used for the steps.

Check please my question on stackoverflow. There are syntax errors, but the idea is clean I think https://stackoverflow.com/questions/56224798/can-group-name-variable-be-dynamic-in-azure-pipelines

[ejjpi]

This feature would be extremely useful to avoid code and pipelines duplication. NEED.

[jschrade]

This is also a feature my team would use daily.

[yairhdz]

Same need for our projects.
Please add this feature. It would be very useful!
Thanks!

[deploymentking]

Just to add some more feedback regarding the usefulness of this. look forward to hearing any updates about this

[antonpunith]

I need this too. It is logical to have a group for each environment and use one when pipeline runs.

[NirajCricket]

I was running through the same issue.
need to have sets of build running via different sets of variables.
-my solution
-- create separate VG (variable group)
--create individual Agent Job for every set of build tasks via YAML file, and assign VG per build sets which will be passed looking the VG, which you need to link in your pipeline by going to triggers>variables > link variable
sample :
jobs:

• job: Linux1
  pool:
  vmImage: 'ubuntu-16.04'
  variables:
  
  
  
  • group: api_vg-prod
    
    steps:
  
  
  • task: FileTransform@1xx
  
  

[mjalas]

This would be a very useful feature, since could also enable using the strategy matrix for a job to specify the groups to use for different environments. Looking forward to hopefully seeing this feature in the near future.

[mafiaazul]

I would love to have this feature as well...

[Csmithson-qh]

Throwing in my vote for this feature as well!

[thafters]

My team would definitely use this feature.

[bourquep]

Does anybody here have an elegant workaround?

[mjalas]

The closest workaround I was able to do was using a combination of variables and job templates. The end result was close enough, at least for now, to what dynamic variable groups with strategy matrix would have provided. However, it is not as explicit and far as I understood it required an extra file for the template.

[davoodharun]

+1 on this

[batorskik]

+1 It would be extremely useful.

[davoodharun]

I actually found somewhat of a work around for this if you use a script to hit the azure devops rest api for variable groups.

https://docs.microsoft.com/en-us/rest/api/azure/devops/distributedtask/variablegroups/get%20variable%20groups?view=azure-devops-rest-5.1

its not ideal but I think this should work

[jlhjr]

For what it's worth, our workaround is to create a new build definition per "environment" with the variable values embedded in that definition.

Not elegant by any means, and the DevOps UI doesn't really enable this (you can't "clone" a build definition, for instance), but you can automate some of it through the REST API, as others have pointed out in their solutions.

[abrignola]

+1

[atbagga]

For those who are blocked on this and using REST-APIs I wrote a util script using Azure-Devops CLI extension to achieve this. You can take a look at it here - https://github.com/atbagga/PsSamples/tree/master/DevopsVarGroup

This will make it a little easier to consume in your automation.

[x10an14]

+1 We would also be in favor of the utmost celerity (hehe) in the addition of this feature! =)

[craigwillard]

+1 yes please, a thousand times yes.

[UmairSyed]

+1

[winstonhenke]

+1 on this

[mortenlerudjordet]

You can also set a shared variable using powershell and the vsTeam module.
Did a small write-up here:
http://blog.lerun.info/2019/09/28/azure-devops-pass-variable-from-build-to-release/

[x10an14]

@mortenlerudjordet
Awesome =) Unfortunately I myself (and maybe others too) find the azure-pipelines.yaml pipelines-as-code feature to be more comfortable/fitting with our workflows^^

Additionally, in your write-up, I didn't catch how you would handle identical variable names with differing values (useful when deploying the same application to different environments like dev/test/prod)?

[mortenlerudjordet]

@x10an14, I too would prefer to be able to both target -group by using a variable name $(VariableGroupName), and be able to update/create shared group variables in AzP YAML.

I believe identical variable names are not supported inside the same shared variable group, though one could have different variable groups that has the same named variables. The issue here is that the Release pipeline does not support it. Would be nice if the Release pipeline could understand something like: $(VariableGroupName\VariableName) to select from different linked variable groups.

One workaround to this could be to use the same code in Build for interacting with variable groups, but now read instead. Would need to have a robust way of passing in variable names, and setting local ones that the pipeline can use in other tasks.

[kerenkeren]

I would love to have this feature as well...

[mboutet]

+1 we also need this.

[nanderson-gy]

+1 we need this as well

[WimDiepgrond]

This would save a lot of duplication in pipelines +1

[CousinOfDeath]

+1 we also need this.

[JG127]

Same for me. Because I'd like to have a single azure-pipelines.yml file people can reuse as is.

[beasamu1]

+1 on this!

[Bhanu88]

+1 for the mentioned feature

[ssorokoski]

+1 yes

[fd-emil]

+1 please !

[NateB2]

We could also really use this feature.

[ozgunakalin]

+1 here

[KKnapik]

:) 1 up

[jedthornock]

+1 here too!

[jeganbaskar]

• 1 its been a long time the request is pending any road map on the request atleast

[enamba]

+1 here !! pls

[michwii]

Hi guys,

Here is how I have selected dynamically the name of my variable group :

variables:
- ${{ if eq(variables['build.SourceBranchName'], 'prod') }}:
  - group: prod
- ${{ if eq(variables['build.SourceBranchName'], 'staging') }}:
  - group: staging
- ${{ if not(and(eq(variables['build.SourceBranchName'], 'staging'), eq(variables['build.SourceBranchName'], 'prod'))) }}:
  - group: test 

Hope it will help you.

Regards

[lzhwxy]

+1
have been looking for solution for the whole day... we do need this feature if all possible.

[svaenn]

+1!

[ppawiggers]

+1

[ffalorjr]

+1 I have a highly reusable pipeline template that relies on variable groups and pipeline variables to make maintaining easier. Gonna try @michwii work around and base my IF statements on the pipeline name. This should give me what I want in the meantime.

[MikaMaroqui]

+1

[JesseStolwijk]

+1

[Marco10101]

+1

[MikeSzulik]

+1

[dushyantovyas]

+1

[wolszakp]

Any news in that topic?
It would be also useful in my setup.
I really like feature to have most variables in repository.
And only few of them assigned dynamically from DevOps that will control whole solution.

[svaenn]

@jlhjr I posted your question as a suggestion here
@all: Maybe this will get some more attention if we vote it up there.

[mszadziul87]

+1 on this!

[pmbrull]

+1

[motowilliams]

This is a fairly basic thing that, in this case, sets Octopus Deploy apart from this.

This needs to be implemented.

[worldspawn]

Hi folks, just wanted to share my revolting solution for this.

Define username and password variables for your pipeline. Password should be a devops token with read access to variables.

steps:
  - checkout: none

  - task: PowerShell@2
    name: setvars
    inputs: 
      pwsh: true
      script: |
        $region = "";
        $env = ""
        $workspace = ""

        switch ("$(envtarget)") {
          "test" {
            $region = "au"
            $env = "dev"
            $workspace = "test"
            break;
          }
          #...
        }

        $username = $env:USERNAME
        $password = "$(password)"

        $groupName = "processing_$($region)_$($env)_$($workspace)"

        $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$($userName):$password"))
        $variableGroups = Invoke-WebRequest -Uri "https://dev.azure.com/yourtenant/yourproject/_apis/distributedtask/variablegroups?api-version=5.1-preview.1" -Headers @{"Authorization"="Basic $auth"} -UseBasicParsing | ConvertFrom-Json
        $input = ($variableGroups.value | ? { $_.name -eq $groupName } | Select-Object -first 1).id
        if ($input -eq $null) { exit 1 }
        $group = Invoke-WebRequest -Uri "https://dev.azure.com/yourtenant/yourproject/_apis/distributedtask/variablegroups/$($input)?api-version=5.1-preview.1" -Headers @{"Authorization"="Basic $auth"} -UseBasicParsing | ConvertFrom-Json
        $variables = $group.variables

        $variables.psobject.properties | %{ 
          Write-Host "##vso[task.setvariable variable=$($_.name);isOutput=true]$($_.value.value)"
        }

        Write-Host "##vso[task.setvariable variable=registry;isOutput=true]drawboard$($region)-registry"
        Write-Host "##vso[task.setvariable variable=region;isOutput=true]$region"
        Write-Host "##vso[task.setvariable variable=env;isOutput=true]$env"
        Write-Host "##vso[task.setvariable variable=workspace;isOutput=true]$workspace"

Sub out yourtenant/yourproject as appropriate.

[worldspawn]

Of course when you try to use it it doesn't work

variables:
        registry: $[ dependencies.DefineVariables.outputs['setvars.registry']]

- task: Docker@2
        displayName: Login to ACR
        inputs:
          command: login
          containerRegistry: $[variables.registry]

Step Docker1 input containerRegistry references service connection $[variables.registry] which could not be found.

Truly awful product.

[vtbassmatt]

We're introducing runtime parameters next sprint (ETA is early March) which will solve most of these use cases.

Also, for future reference, product feedback should be registered with Developer Community. That's the area we check for requested new features, not the docs repo.

[NickSpag]

If you're just looking to change based on environments, and your environment can be resolved via the built in conditionals, this is valid yaml:

variables:
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
  - group: 'AzureDevopsLibraryName-PROD'

There's probably a lot more you can do with this while we await the runtime parameters that are coming, but as an example to resolve different environments for a git flow style repo:

variables:
  - name: 'AppSettingsPath'
    value: '$(System.DefaultWorkingDirectory)/functions/MyFunction/' 
- ${{ if or(eq(variables['Build.SourceBranch'], 'refs/heads/develop'), startswith(variables['Build.SourceBranch'], 'refs/heads/feature'), startswith(variables['Build.SourceBranch'], 'refs/heads/chore'), startswith(variables['Build.SourceBranch'], 'refs/heads/hotfix')) }}:
  - group: 'AzureDevopsLibraryName-DEV'
- ${{ if startswith(variables['Build.SourceBranch'], 'refs/heads/release') }}:
  - group: 'AzureDevopsLibraryName-QA'
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
  - group: 'AzureDevopsLibraryName-PROD'

[Planche95]

Anybody knows how can I dynamically assign variable groups for release pipelines? not YAML build pipelines

[vtbassmatt]

@Planche95 I don't believe that is supported in classic RM.

[davejhahn]

Not what is being asked, definitely a workaround, but you could store variables in an environment specific keyvault, and based on something (e.g. a passed in variable or looking at a system variable), within a PowerShell task, set a KeyVault variable, then have a task after that to pull from the Key Vault. Provided the variable names are the same for each environment.

e.g.

1. PowerShell: if "development" (however you are determining that), set KeyVault variable name to use to Dev keyvault
2. Azure KeyVault Task, loading from above keyvault varaible. Variable names are the keyvault keys.
3. Use keys.

Obviously that means storing the values in keyvault. For my use case, it's just a small subset from the full environment variable group so it's a reasonable workaround.

[DotNetWebAppDeveloper]

We are trying to reference a variable group dynamically with if condition. We get a error message

Is there a different or new syntax to write in Azure DevOps?

[michwii]

We are trying to reference a variable group dynamically with if condition. We get a error message

Error message:

Is there a different or new syntax to write in Azure DevOps?

Try the post I have written below :

Hi guys,

Here is how I have selected dynamically the name of my variable group :

variables:
- ${{ if eq(variables['build.SourceBranchName'], 'prod') }}:
  - group: prod
- ${{ if eq(variables['build.SourceBranchName'], 'staging') }}:
  - group: staging
- ${{ if not(and(eq(variables['build.SourceBranchName'], 'staging'), eq(variables['build.SourceBranchName'], 'prod'))) }}:
  - group: test 

Hope it will help you.

Regards

Don't worry about the syntax error highlighting, I also have it in my editor.

Wish it will help you :)

[DotNetWebAppDeveloper]

We are trying to reference a variable group dynamically with if condition. We get a error message

Is there a different or new syntax to write in Azure DevOps?

Try the post I have written below :

Hi guys,
Here is how I have selected dynamically the name of my variable group :

variables:
- ${{ if eq(variables['build.SourceBranchName'], 'prod') }}:
  - group: prod
- ${{ if eq(variables['build.SourceBranchName'], 'staging') }}:
  - group: staging
- ${{ if not(and(eq(variables['build.SourceBranchName'], 'staging'), eq(variables['build.SourceBranchName'], 'prod'))) }}:
  - group: test 

Hope it will help you.
Regards

Don't worry about the syntax error highlighting, I also have it in my editor.

Wish it will help you :)

It works
Thanks

[zhangpengchen]

We are trying to reference a variable group dynamically with if condition. We get a error message

Is there a different or new syntax to write in Azure DevOps?

Try the post I have written below :

Hi guys,
Here is how I have selected dynamically the name of my variable group :

variables:
- ${{ if eq(variables['build.SourceBranchName'], 'prod') }}:
  - group: prod
- ${{ if eq(variables['build.SourceBranchName'], 'staging') }}:
  - group: staging
- ${{ if not(and(eq(variables['build.SourceBranchName'], 'staging'), eq(variables['build.SourceBranchName'], 'prod'))) }}:
  - group: test 

Hope it will help you.
Regards

Don't worry about the syntax error highlighting, I also have it in my editor.
Wish it will help you :)

It works
Thanks

When you have many variables, by using this setup you would like to have sub groups in place which is not possible at this moment.

[cayodiebe]

+1!