Azure-pipelines-agent: Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file

[A141133]

Agent version and platform

Version of your agent? 2.120.1

OS of the machine running the agent? Linux and Windows

VSTS type and version

VisualStudio.com

If VisualStudio.com, what is your account name? https://aglthundercats.visualstudio.com

What's not working?

When using the new build definition YAML format with the AzureCLI@1 and Docker@0 tasks, we cannot specify an Azure subscription ID connection using a VSTS variable. For example, this step definition works fine:

- task: AzureCLI@1
  displayName: Upload Image Archive
  inputs:
    connectedServiceNameARM: ********-****-****-****-************
    scriptLocation: inlineScript
    failOnStandardError: false
    inlineScript: pwsh build/Build.ps1

But this one does not:

- task: AzureCLI@1
  displayName: Upload Image Archive
  inputs:
    connectedServiceNameARM: $(AzureSubscriptionEndpointId)
    scriptLocation: inlineScript
    failOnStandardError: false
    inlineScript: pwsh build/Build.ps1

When I try using that step definition, I get this error:

An error occurred while loading the YAML build definition. A service endpoint with name $(AzureSubscriptionEndpointId) could not be found. The service endpoint does not exist or has not been authorized for use.

This suggests that the variable substitution doesn't work for these YAML files. I would have expected to be able to provide my Azure subscription ID through a variable, to avoid hard-coding it into my YAML file.

[stephenmichaelf]

@A141133 Could you try updating to the latest Agent?

[A141133]

@stephenmichaelf I am using a hosted agent pool, so I don't think I can update the agents? Whenever I try, I get an error saying Access Denied. Hosted Agent Pools cannot be modified.

[stephenmichaelf]

@A141133 Thanks for the clarification. Let me see if I can recreate the issue. Using variables like this should work.

[mathieu-benoit]

Hi there (FYI @stephenmichaelf), on my side with task: Docker@0 when setting the endpoint's name or id on the azureSubscriptionEndpoint input, I got this message while trying to queue a new build:

An error occurred while loading the YAML build definition. A service endpoint with name AzureSubscription could not be found. The service endpoint does not exist or has not been authorized for use.

Notes:

• I'm using the Hosted Linux Preview agent
• I'm using GitHub as source control
• I'm doing some test on a branch, not in master
• My .yml file is not in the root folder
• It's working fine via the Build Definition UI
• I'm not using variable for now, but that's the goal after working with hard-coded value ;)

[stephenmichaelf]

@ericsciple Any idea why these two issues could be happening?

[ericsciple]

Endpoint authorization for yaml definitions is performed on-push of the file and on-edit of the definition. It will not work if you use a variable.

If you want to be variable, you could add two tasks, and use conditions to drive the runtime-decision which one is executed. You can use queue-time variables in your condition.

[mathieu-benoit]

Thank you @stephenmichaelf and @ericsciple, for your prompt follow up.
@ericsciple, just to make sure, I'm not using variables in my case. After reading the documentation you referenced, could you confirm it's not working for me (see above) because I'm using GitHub? And/or a .yml file not in the root folder? And/or not in the master branch?

[ericsciple]

@mathieu-benoit if you make a trivial edit to the web definition and save, the act of saving the web definition will trigger resource authorization to run again (loads file from master)

[mathieu-benoit]

Thanks @ericsciple, by doing the trivial edit/save on the web definition it worked. Thx!

[A141133]

Hi all, I think there are two separate issues here. It's great @mathieu-benoit's issue has been worked around though.

@ericsciple - your suggestion of using a condition to switch between two versions of the same task, but with different Azure service ID references, isn't really a long-term solution. It'll get us through for now, but ultimately we'd surely want the build.yaml file to be agnostic to the GUIDs used within VSTS - and for these to be able to be provided in variables. This will become very important when we get VSTS release definitions as YAML.

I understand that the endpoints are authorised on push of the definition, but it feels like this is a legitimate use case. It may not be a bug per se, but could this be tracked as a feature improvement request?

[ericsciple]

I agree. We are thinking about multiple types of process-reuse scenarios and ways to pass inputs to a re-usable process.

[mike10010100]

@ericsciple Have been working with a MS rep on this for a few days now with the same issue as @mathieu-benoit , and the trivial update solution worked! I'd definitely suggest putting this into the Authentication documentation as a possible troubleshooting step, as it wasn't clear why the endpoint wasn't authorized.

[bryanmacfarlane]

Closing this specific issue. The re-use scenarios are on our backlog.

[ggirard07]

@bryanmacfarlane or @ericsciple Any github issue or entry on the roadmap we can refer to in order to know when this will be available? Is it plan for GA?

[lqueryvg]

So when is this likely to get fixed, and where is this being tracked ?
Also, if this is not considered to be a bug, where is this behaviour documented ?

Forcing users to hard-code configuration information like this is always BAD.

HERE IS MY USE CASE:

1. We have 2 organisations (with separate VSTS domains / projects) pointing at YAML in git in a shared repo. I.e. two VSTS, one git repo.
2. Each organisation wants to use it's own names for the service endpoints.
3. But, since Microsoft are forcing us to hard-code the service endpoint name into our YAML code (because we can't use variables), both organisations are forced to use the same names - which in some case breaks our naming standards.

[4c74356b41]

@ericsciple hey, any ideas why the same issue could happen when I'm trying to use templates? when I put everything in 1 file it is working with the above-mentioned hack. thanks

[ericsciple]

@ggirard07 GA is the plan. I don't have a specific date

[ericsciple]

@lqueryvg templates is the answer

[ericsciple]

@4c74356b41 I would expect this to work, even when using templates:

navigate to the build definition in the web, switch the default branch to your branch that includes the service endpoint reference, save, revert back to your desired settings, save again. The act of saving the definition loads the file (from the default branch) and authorizes discovered resources.

If that doesn't work, I'll see if I can get a repro.

We have a better experience in the web landing soon to authorize resources. Sometime in the next month or so I hope it will be rolled out everywhere.

[4c74356b41]

@ericsciple i can share details if you need, like is there an error guid i can send you? i saw that document and it helped me with this issue when everything was in the same yaml, but not with templates. funny enough if I edit the working definition (so the one with resources approved already) to use the templated approach - it breaks.

you do realize a month is an eternity? :)

[4c74356b41]

@ericsciple

I'm using the Hosted Linux Preview agent
I'm using GitHub as source control
I'm doing test on develop and master, similar behavior.
My .yml file is not in the root folder
It's working fine via the Build Definition UI (so not using yaml)
It's working fine if its not using templates (so everything in 1 yaml).

Sample yaml:

name: $(GITVERSION_NuGetVersion)
resources:
- repo: self
  fetchDepth: 1
  clean: true

phases:
- template: vsts/buildApplication.yml

- template: vsts/buildContainer.yml
  parameters:
    appName: 'webApplication'
    appFolder: 'UI'

in the buildContainer.yml I have this:

azureSubscription: yyy
azureContainerRegistry: '{"loginServer":"xxx.azurecr.io", "id":"/subscriptions/guid/resourceGroups/name/providers/Microsoft.ContainerRegistry/registries/name"}'

[lqueryvg]

@ericsciple

@lqueryvg templates is the answer

How can templates be the answer ?

I hope you are not suggesting that people write
a separate template for each possible service endpoint.

This goes against the whole idea of using variables.

This issue quite clearly needs to be fixed.

[ericsciple]

@lqueryvg for the scenario you specified:

We have 2 organisations (with separate VSTS domains / projects) pointing at YAML in git in a shared repo. I.e. two VSTS, one git repo. Each organisation wants to use it's own names for the service endpoints.

you can put the template in the shared repo, and use parameters to pass the endpoint name to the template.

[lqueryvg]

@ericsciple

Right, so you are suggesting that we write a separate template for each service endpoint.
As I said, this goes against the whole idea of using variables and the problem still stands.

As as DevOps engineer I understand the importance of being able to separate code from configuration. Creating separate templates (code) for every service endpoint (configuration) is obviously not a proper solution - it's a compromise in order to work around a feature which is missing from VSTS.

I get the impression (ie this ticket is "Closed") that Microsoft does not see this as a problem ?

[ericsciple]

No, not a template per service endpoint. Here is the template:

# file: my-reusable-template.yml
parameters:
  endpoint: ''
steps:
- task: someTask@1
  inputs:
    theEndpoint: ${{ parameters.endpoint }}

Then multiple entry files can consume the resuable template, and specify the service endpoint as a parameter:

# file: entry-file-1.yml
steps:
- template: my-reusable-template.yml
  parameters:
    endpoint: endpointA

# file: entry-file-2.yml
steps:
- template: my-reusable-template.yml
  parameters:
    endpoint: endpointB

[4c74356b41]

@ericsciple hey, not being rude or something, but can you at least acknowledge that you saw my posts?

[ericsciple]

@4c74356b41 thanks I was able to reproduce the issue. I did not expect this behavior:

if I edit the working definition (so the one with resources approved already) to use the templated approach - it breaks

Thanks I was able to reproduce the issue. As a workaround I specified the guid instead of the endpoint name. I'll look into why referencing the endpoint by name isn't working.

[ericsciple]

@4c74356b41 nevermind I take that back. By name works.

There is a pretty crummy regression I just found, that is causing the list of authorized resources to be cleared in some cases. If by name doesn't work for you, open a separate issue and I can help you work through it. This thread is too long and the discussion is too broad. Let's work through your specific problem on a separate issue.

[lqueryvg]

@ericsciple , I'm sorry my wording was incorrect. Your solution is not multiple templates, it's one. But it's still multiple yaml files - one for each service endpoint (this is what I meant). I.e. you are moving the values (i.e. the endpoint name) into code, when it should really be in variables.

My point still stands.

[4c74356b41]

@ericsciple https://github.com/Microsoft/vsts-agent/issues/1681 this is the issue, thank you very much!

[4c74356b41]

@ericsciple another question, how to use a guid instead of a name? where to find that guid? I was unable to identify it anywhere in endpoint settings. thanks!

[4c74356b41]

ok, I've installed vsts PowerShell module and found AzureRM endpoint guid and used that, but it didn't help. so at this point, I'm extremely skeptical about the idea that this thing works in any fashion.

[ericsciple]

@4c74356b41 the best way to find the guid is save it on a designer build, then browse to the History tab, and diff the history of the designer definition. The task input will contain the guid.

I'm not surprised by your skepticism. I just left a comment on the other issue, detailing the bug that is causing a very confusing experience.

[ericsciple]

@lqueryvg thanks for bearing with me. I think we are on the same page now.

Can you help me understand your specific scenario?

Do you want to centrally manage which service endpoint is used? In other words, are you trying to avoid updating each relevant branch in your repo, if you want to change to use a different service endpoint? The service endpoint itself is centralled managed. So would updating the service endpoint solve this problem?

Or do you want to have the ability to choose the service endpoint at queue-time (i.e. in the dialog that pops up at queue time)? We have plans for this scenario.

[lqueryvg]

@ericsciple, I think specific scenarios are confusing the issue.

It's as simple as this...

I want to be able to pass the service endpoint as a variable.
This is exactly what this ticket is supposed to address.

Without this, I am unable to separate code (in git) from configuration in VSTS which in turn makes it impossible to re-use my code for different configurations.

I should not have to add or change any yaml code (in git) every time I change or add a new service endpoint.

Without this feature, VSTS does not allow me to follow the DevOps mantra of "code all the things" and is not a viable tool for true DevOps.

[markusdresch]

any news on this? the template approach does not work, it still does not resolve variables.

variable-docker-login.yml:

parameters:
  azureSubscriptionEndpoint: ''
  azureContainerRegistry: ''

steps:
- task: Docker@1
  displayName: Container registry login
  inputs:
    command: login
    azureSubscriptionEndpoint: ${{ parameters.azureSubscriptionEndpoint }}
    azureContainerRegistry: ${{ parameters.azureContainerRegistry }}.azurecr.io

in the build pipeline i add a variable group, then

  - template: steps/variable-docker-login.yml@pipelines
    parameters:
      azureSubscriptionEndpoint: $(AZURE_SUBSCRIPTION)
      azureContainerRegistry: $(AZURE_CONTAINER_REGISTRY)

the pipeline fails immediately with:

The pipeline is not valid. Job Build: Step Docker input azureSubscriptionEndpoint references service connection $(AZURE_SUBSCRIPTION) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.

same behavior inline without templates.

[4c74356b41]

did you auth the connection? also, you need to pass in actual connection name. it works for about a year already for me on god knows how many different setups @markusdresch

[markusdresch]

the connection is authenticated, but the variable is not expanded. i'm trying to use the same repository in different projects and push an image to different container registries. i expected that all i'd have to do is use a different set of variables, but seems like that's not possible.

[bryanmacfarlane]

He's referring to resource authorization, not authentication.

The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz

That error is coming from the service and not an agent issue.

Creating a developer community feedback ticket for the service on the specific issue would be best (rather than iterating on this old issue). You can reference this issue from there.

[markusdresch]

that error is not coming from the service because the variable is not expanded, so the agent literally tries to connect to $(AZURE_SUBSCRIPTION), not to the service endpoint it would resolve to. the service it would resolve to is authorized. i.e. if i use the service name directly, it works. but this way the yaml pipeline can't be shared between different projects/environments.

[markusdresch]

i worked around the issue by removing the Docker@1 task for logging in, using scripted docker login with credentials from key vault instead.

[fraserdarwent]

Issue still exists

Input

  - job: ARMDeployment
    displayName: Create ARM Deployment
    pool:
      vmImage: ubuntu-latest
    variables:
      service_connection: My Service Connection
    steps:
      - task: AzureResourceGroupDeployment@2
        inputs:
          azureSubscription: $(service_connection)

Expected:
Task AzureResourceGroupDeployment@2 uses service connection name My Service Connection

Actual:
Task AzureResourceGroupDeployment@2 fails as variable is not expanded

The pipeline is not valid. Job ARMDeployment: Step AzureResourceGroupDeployment input ConnectedServiceName references service connection $(service_connection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

[DeroSavage]

Update on this? I got the same issue.

[Heimidall]

Does anyone know if this got fixed?

[Burnsy2023]

I'm also running into this issue, so not fixed :(

[MichaKnoe]

With the UI release pipelines you could just assign subscription variables and it just worked. To me it looks like a regression in yaml which at least should be properly documented. Or maybe those issue pages are even enough documentation, right ;-)
Regards.

[ggirard07]

Not sure what you are all doing wrong guys, but this is working as expected for me when using the service connection name I configured in my project settings and have it properly authorized...

For the YAML, I am using the same approach as mentiond by @ericsciple in #1307 (comment).
I am using it with AzureCLI, AzureResourceGroupDeployment, AzureWebApp, AzureFunctionApp, ...

parameters:
  subscription: ''

steps:
- task: AzureCLI@2
  inputs:
    azureSubscription: ${{ parameters.subscription }}
    scriptType: pscore
    scriptPath: $(Build.SourcesDirectory)/script.ps1

[MichaKnoe]

Using template parameters is not the issue, but using e.g. variables from variable groups with macro syntax $(serviceConnection)

[ggirard07]

I use a variable (so a macro) defined as

variables:
  subscription: 'my-sub'

jobs:
- template: ./job-template.yml
  parameters:
    subscription: $(subscription)

and pass it to my template parameters without any issue...

[4c74356b41]

I've posted over half a year ago that its working, if its not for you - you misconfigured something.

[MichaKnoe]

To be more precise. What is not working are variables referenced in variable groups not hard coded in yaml files but in Azure DevsOps variable groups. This is the recommended way to reuse variables over pipelines.

[ggirard07]

@MichaKnoe sorry missed that point. You are probably better opening a new issue related to variable group explicitly in that case to gain more traction instead of recycling this closed issue....

[helmerm]

Is there any update on this regarding variables coming from variable groups? Running into the same problem.

[rdbartram]

i am also experiencing this issue. Especially when the value being passed comes from a variable group. variables defined on the pipeline in the UI or in yaml directly work fine

[mcalnd70]

Who's here in 2020 and still can't believe this is a thing?

[onkobu]

This gets even weirder with current multi stage pipelines and environments. Conditions can only be expressed on stage, step or job-level. So I decided to work around this issue by having a template containing 3 jobs with a single task only. Each job is not a plain job-definition but a deployment-definition with an environment:

jobs:
- deployment: 'onDev'
  environment: 'dev'
  condition: eq('${{ parameters.environmentName }}', 'dev')
  strategy:
    runOnce:
      deploy:
       steps:
       - task: AzurePowerShell@3
         inputs:
           azureSubscription: 'azure-dev' # neither $(Environment.Name) nor ${{ parameters. }} does work here
- deployment: 'onInt'
  environment: 'int'
  condition: eq('${{ parameters.environmentName }}', 'int')
  strategy:
  # same job as above
  # and again for prod

It works and runs only the job matching the condition but requires approval according to all 3 environments, no matter which environment it runs on. Executing for dev needs approval for the two others, same in int and prod.

My intention was only to have 3 stages, each for its own environment running the single PowerShell task only switching azureSubscription based on ${{ parameters.environmentName }} which would have been set to $(Environment.Name).

[mcalnd70]

There is an obsession here that closing issues because they don't fit a timeline potentially makes GitHub almost a trawl of items that are closed prematurely and not being addressed or actioned, or are just closed items that refer to other closed items to other closed item etc, like a version of inception with no answer.

Very frustrating

[M0ns1gn0r]

@damccorm Maybe you can re-open this hot prematurely closed issue?

[damccorm]

Hey @M0ns1gn0r - I haven't seen this issue before and am still trying to wrap my head around it (there's been a lot of activity here). At a glance, I tend to agree that closing it when we did was probably not the right thing to do

With that said, I think that the most effective way to make sure we get the right eyes on this is actually going to be opening a ticket at https://developercommunity.visualstudio.com/spaces/21/index.html. This looks to me like an issue with the service, not the agent itself, and I think the investigation into doing this work should at least start there. We'd definitely want to link to that ticket from here. Would you be willing to do that? Does that seem fair?

[onkobu]

Just for tracking: https://developercommunity.visualstudio.com/content/problem/1073702/azure-subscription-endpoint-id-cannot-be-provided.html

[samsmithnz]

It's pretty frustrating that the developer community ticket was closed quickly too... :(

I suspect most people are just making a template for each environment and moving on, but it's not exactly the best experience. How can we find the right people to talk to and solve this? Do we need to open a support request?

[onkobu]

[…] How can we find the right people to talk to and solve this? Do we need to open a support request?
It's software like all software written by average developers. I assume they didn't even get the intention. And the few who didn't chime in know about the quirks for the two-pass parsing. They also know there's no budget to rework things. There's no reason for me to favor any of the cloud providers.

[didaskein]

Hi agree with @ericsciple

You need to extract the ServiceName variable inside a Config.General.yaml in order to inject the service name.
It will not work if you set a variable ServiceName inside the file Config.Development.yaml because Azure Devops need to check all permissions at the beginning of the run of the pipeline.

My Root Yaml file :

trigger:
  branches:
    include:
    - master
    - release/*

pool:
  vmImage: "windows-latest"
variables:
  - template: ../variables/Config.General.yaml

stages:
  ###################################################
  # Build
  ###################################################
  - stage: build_stage
    displayName: "Build"
    jobs:
      - template: build.yaml

  ###################################################
  # Release
  ###################################################
  # Playground
  - stage: deploy_Stage_Playground
    displayName: "Playground Deploy"
    dependsOn: build_stage
    condition: succeeded()
    variables:
      - template: ../variables/Config.Playground.yaml
    jobs:
      - deployment: deploy_Job_Playground
        displayName: "Deploy API & Products"
        environment: "Playground"
        strategy:
            runOnce:
                deploy:
                    steps:
                       - template: deploy-stage.yaml 
                         parameters:
                            ServiceConnectionName: $(ServiceConnectionNamePlayground)


  # Development
  - stage: deploy_Stage_Development
    displayName: "Development Deploy"
    dependsOn: 
      - build_stage
      - deploy_Stage_Playground
    condition: succeeded()
    variables:
      - template: ../variables/Config.Development.yaml
    jobs:
      - deployment: deploy_Job_Development
        displayName: "Deploy API & Products"
        environment: "Development"
        strategy:
            runOnce:
                deploy:
                    steps:
                      - template: deploy-stage.yaml 
                        parameters:
                            ServiceConnectionName: $(ServiceConnectionNameDevelopment)

Inside the file Config.General.yaml you need to have :

variables:
    - name: ServiceConnectionNamePlayground
      value: 'MyServiceConnectionPlayground'
      readonly: true

    - name: ServiceConnectionNameDevelopment
      value: 'MyServiceConnectionDevelopment'
      readonly: true

I know it doesn't seem intuitive not to be able to put the ServiceName in a variable file by environment.
But unfortunately it's designed that way from an access rights validation point of view.
By using yaml variable files and yaml templates youcan factorize all the variables in the file.

Kind regards,

[onkobu]

Azure Devops need to check all permissions at the beginning of the run of the pipeline.

…and could add evaluation of variable expressions to first pass.

Nevertheless I'll give the global variable definition a try, if I can convince the others to avoid typos in variable names or copy&paste-induced issues because of wrong variable. My intention is to avoid both issues by using/ incoporating the EnvironmentName.

Main issue with complex (=real world) pipelines is typos/ wrong environment. And it is not very comfortable since undefined variables evaluate to empty string instead of warning about an undefined variable.

Does this feel like the revival of Visual Basic with all its »features«? I waste my time fixing errors I didn't cause but otherwise couldn't proceed as fast as I supposed to. Why is this YAML-thing an advantage over autotools/ make or why do I need minutes to let Jenkins parse existing pom.xml and derive build graphs from dependencies but hours to figure out how to ingest Environment.Name in a variable?

[pkpaul5]

This issue is still present in Azure DevOps YAML pipeline. When the service connection name is maintained as a variable in variable group and the variable is referred in the YAML pipeline then while running the pipeline below error comes 'There was a resource authorization issue: "The pipeline is not valid. Job Deploy: Step AzurePowerShell2 input ConnectedServiceNameARM references service connection $(azureSubscription) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."'

[didaskein]

@pkpaul5 Do you import the variable at the root level of the yaml (it's working for me on azure devops for multiple projects).
variables:

• template: ../variables/Config.General.yaml

If you import the variable at a lower level (below stages) it will not work by design & security check...

[pkpaul5]

Hi didaskein,
The requirement is to maintain the service connection name in variable groups specific to DEV,PROD environment. Then use the variable in YAML pipeline based on the target environment for deployment. Single pipeline for deployment into DEV and PROD environment. Currently we are not looking to maintain the service connection name in separate template. If you have any idea how this can be addressed with variable groups then let me know.
Thanks
pkpaul5

[didaskein]

@pkpaul5 it should work also with variables groups, you just need to declare them at the beginning of the yaml file not at a lower level (below stages)

[onkobu]

@pkpaul5 it should work also with variables groups, you just need to declare them at the beginning of the yaml file not at a lower level (below stages)

Variable groups don't exist in YAML pipelines. And »should« is not »I checked that it works«. Mind the change in paradigms from interconnected Build- and Release Pipelines to independent multi-stage (and multi-repository) YAML pipelines.

A common azure-pipelines.yaml to build and deploy nowadays turns into this graph:

Build (w. Docker image) +-> Create/ Modify Resources DEV -> Deploy DEV
                        |-> Copy Docker Image to Registry INT --> Create/ Modify Resources INT -> Deploy INT
                        |-> same for Testing
                        |-> same for PROD

Common steps like copying a docker image between registries, getting secrets from a KeyStore into the current stage/ set of following tasks or invoking Az through AzureCLI explode. In Classic Pipelines task groups could be used. Now it is templates. But templates fail during compile phase of the YAML pipeline, since it does not recursively evaluate ${{ parameters.Environment }} which was set as Environment: $(Environment.Name).

If you do so the pipeline complains, that it was unable to find the resource …$(Environment.Name). This is not obvious to the average pipeline user, since Environment.Name is a system variable and the two-phase approach (compile+runtime) of pipelines a hidden convention.

In the Visual Studio Issue I also outlined, that condition-statement in templates is also broken, suffering from a different problem during the same phase. The variables/ parameters don't get evaluated during compile phase so it is not clear what the condition evaluates to unless you use $[]-notation which breaks with the rest of variable evaluations and parameters.

Try not to think of flat pipelines from the text books/ docs compiling something followed by a deployment into one of three environments. Add KeyVaults/ secret replacement, move Docker images between registries, roll passwords through the pipelines, use Helm to fill your templates mounted into the containers, derive routing and policies from Environment.Name. Finally Dont-Repeat-Yourself with the YAMLs necessary for that.

(With YAML pipelines it is still possible to chain pipelines through pipeline triggers. But they are independent now and chaining based on build artifacts into Release Pipelines is not possible anymore. Thus YAML-pipelines are more like God-objects knowing everything from preparation over building until the stages and beyond. I admire nerds taming >2k lines of YAML. I prefer re-usable and easy to maintain snippets.)

[jae-work]

This is still an issue for me as well.

tried passing the subscription name from the root template as a variable, as a template parameter and from the variable in Azure DevOps all with errors in the effect of:

There was a resource authorization issue: "The pipeline is not valid. Job JOBNAME: Step input azureSubscription references service connection $() which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job JOBNAME: Step input azureSubscription references service connection $() which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Are there any work arounds known working without creating multiple copies of the temp[lates?

thanks

[IanMoroney]

For everyone struggling with this issue in YAML, I have a solution.

I am using a pipeline which uses a template, so i can standardise the deployment process, and pass parameters into the template. One of those parameters i desired to pass in is the service connection.

I tried all sorts of $() syntax, and ran into the same issue you did. A former colleague of mine provided the solution. I'll be buying them a beer.

The reason why this works is the difference between compile time parameters and run time parameters. The former (i believe represented by $() syntax) does not work, whereas the latter (i believe represented by ${{ }} syntax) does. I don't know the reason for this, it was just explained to me like this.

Example:

Set:

azureSubscription: ${{ parameters.serviceConnectionName }}

Define:

- name: 'serviceConnectionName' 
  type: string
  default: false

Then in your pipelne,
Set:

        parameters:
          serviceConnectionName: ${{ variables.serviceConnectionName }}

and
Define:

variables:
  serviceConnectionName: <MyAzureDevopsServiceConnectionName>

[onkobu]

Thanks for chiming in but this is only a 40% solution. I'll give this a try later but need a bit of clarification:

1. You tried this with task templates? (There may be differences with job- and task-templates.)
2. the first two Set: and Define: belong into the template?
3. the second two are part of the pipeline using the (task-) template?

Following your suggestion I'd have to manage 3 variables – one per environment. I'd still have to pay attention to not use the wrong variable or need to review all changes of others for copy&paste issues = tedious/ error prone.

It always burns down to: variables are not allowed in compile time expressions. And the latter is not obvious if there are special attributes of certain tasks that are compile-time-only. I see absolutely no other reason for: not possible during compile time except that it is a matter of finding »resources«.

Taking all the comments and hassle into consideration from top of the issue 'til here: there are plenty of characters that could be glued together as a patch. (And mind all those not jumping the hurdle to vote or comment here.)

[IanMoroney]

1. This is currently operational with a task template.
2. First two belong in the template
3. Second two are part of the pipeline.

Think of it as a variable per stage, if you have different subscriptions for each stage.
Different variables for different stages are pretty commonplace, but through further interpolation could also be made simpler.

Below is a full example:

Pipeline:

trigger: none



variables:
- group: Global-Variables
- name: buildName
  value: Build1
- name: customerName
  value: MyCustomer
- name: customerNameAbbreviation
  value: myc
- name: pipelinenumber
  value: 15

name: $(Myapp-Version)

stages:
- stage: Dev
  displayName: Deploy To Dev
  jobs:  
  - deployment: Deploy_to_Dev
    displayName: Deploy To Dev
    variables:
      regionPrefix: eg
      environment: dev
      environmentAbbreviation: dev
      serviceConnectionName: SubscriptionA # Service Connection Name
    pool:
      name: 'Windows'
    environment: Dev
    workspace:
      clean: all
    strategy: 
      runOnce:
        deploy:
          steps:
          - template: ../templates/deploy-webapp.yaml  # Template reference
            parameters:
              regionPrefix: '$(regionPrefix)'
              customerName: '$(customerName)'
              customerNameAbbreviation: '$(customerNameAbbreviation)'
              environment: '$(environment)'
              environmentAbbreviation: '$(environmentAbbreviation)'
              pipelinenumber: '$(pipelinenumber)'
              serviceConnectionName: ${{ variables.serviceConnectionName }} # Passing in service connection name here

Template:

parameters:
- name: 'regionPrefix'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'customerName'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'customerNameAbbreviation'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'environment'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'environmentAbbreviation'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'pipelinenumber'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'serviceConnectionName'  # defaults for any parameters that aren't specified
  type: string
  default: false

steps:
- task: PowerShell@2
  displayName: List Directory
  enabled: true
  inputs:
    targetType: 'inline'
    script: |
      cd $(Pipeline.Workspace)
      pwd
      dir
      tree /F
- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy App'
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: ${{ parameters.serviceConnectionName }}
    appType: 'webApp'
    WebAppName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)myapp-wa'
    deployToSlotOrASE: true
    ResourceGroupName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)-rg'
    SlotName: 'staging'
    packageForLinux: '$(Pipeline.Workspace)\release-$(customerName)-$(environment)-app.zip'
    enableCustomDeployment: true
    DeploymentType: 'webDeploy'
    RemoveAdditionalFilesFlag: true
- task: AzureAppServiceManage@0
  displayName: 'Swap Slots'
  inputs:
    azureSubscription: ${{ parameters.serviceConnectionName }}
    Action: 'Swap Slots'
    WebAppName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)myapp-wa'
    ResourceGroupName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)-rg'
    SourceSlot: 'staging'
    TakeAppOfflineFlag: true

[helmerm]

I am not sure if this is only caused by permissions.

My service connection has no restrictions and I can see a run show up in the usage history of the service connection. However, the task (in my case Pulumi Task) seems to not get the necessary variables from the template expression (azureSubscription: '${{ parameters.azureSubscription }}').

As far as my understanding goes after taking a look into the Azure Task SDK: Tasks are supposed to get access to the service connection via ENDPOINT_AUTH_ environment variables passed by the agent. Looking at the diagnostics output, I do not see any ENDPOINT_AUTH variables passed for my service connection.

I did not find out yet, how the agent determines when to pass these variables. From a security perspective, it only makes sense to pass them, if they are needed. But how does the agent find out if they are needed?

[onkobu]

Think of it as a variable per stage, if you have different subscriptions for each stage.

It's actually exactly this, a variable per stage. I'll give this a try. Nevertheless it is a humble fix. Typo can only happen in one place – the variable definition. Still not perfect by deriving it from ${Environment.Name} but looks manageable. THX.

[jannikbuschke]

Im also hitting this issue.

I only have a half solution now, by parameterising a template file (using a compile time expression in it) and passing the connection as a statically typed parameter.
Ideally I would like to reference the connection from a variable group, but this seems to be not possible.

[jannikbuschke]

@IanMoroney
In your example the value 'SubscriotionA' is still statically typed into the template file, and not a variable right?

stages:

• stage: Dev
  displayName: Deploy To Dev
  jobs:
  
  
  
  • deployment: Deploy_to_Dev
    
    displayName: Deploy To Dev
    
    variables:
    
    regionPrefix: eg
    
    environment: dev
    
    environmentAbbreviation: dev
    
    serviceConnectionName: SubscriptionA # Service Connection Name (statically typed)
    
    pool:
    
    name: 'Windows'
    
    environment: Dev
    
    workspace:
    
    clean: all
    
    strategy:
    
    runOnce:
    
    deploy:
    
    steps:
    
    
    
    
    
    • template: ../templates/deploy-webapp.yaml # Template reference
      
      
      parameters:
      
      
      regionPrefix: '$(regionPrefix)'
      
      
      customerName: '$(customerName)'
      
      
      customerNameAbbreviation: '$(customerNameAbbreviation)'
      
      
      environment: '$(environment)'
      
      
      environmentAbbreviation: '$(environmentAbbreviation)'
      
      
      pipelinenumber: '$(pipelinenumber)'
      
      
      serviceConnectionName: ${{ variables.serviceConnectionName }} # Passing in service connection name here
      
      
      ```
    
    
    
  
  

[IanMoroney]

The value for "SubscriptionA" in my example is listed in my pipeline as a variable.
The pipeline is calling the template "deploy-webapp.yaml" as one of its steps, and passing the parameter into it.

[trietnguyen267]

This issue still persists, please reopen the issue.

[IanMoroney]

@trietnguyen267 Can you post an example of the code you're trying to use
and where it's failing?

You also may want to open a separate issue for better support.