Azure-devops-cli-extension: azure devops extension to support --service-principal login

[swapnild2111]

Describe the bug
az login --service-principal --username xxx --password yyy --tenant zzz
--> Login goes successful.

az devops configure --defaults organization=https://xyz project=xyz
--> This works fine.

az pipelines build list -o table --verbose --debug
--> Error:

Current cloud config:
AzureCloud
trying to get token (temp) for tenant 05d75c05-fa1a-42e7-9cf1-eb416c396f2d and user fc7bc748-404e-4f72-84e2-95cb6f2edb2f
attempting to read file /root/.azure/accessTokens.json as utf-8-sig
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - Authority:Performing instance discovery: ...
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - Authority:Performing static instance discovery
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - Authority:Authority validated via static instance discovery
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - TokenRequest:Getting token from cache with refresh if necessary.
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - CacheDriver:finding with query keys: {'_clientId': '...', 'userId': '...'}
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - CacheDriver:Looking for potential cache entries: {'_clientId': '...', 'userId': '...'}
adal-python : 2e75d6f5-b4a6-46da-b9ca-0a2b5cea63eb - CacheDriver:Found 0 potential entries.
not able to get token from az login
Could not retrieve token from local cache. Please run 'az login'.
Traceback (most recent call last):
  File "/root/.azure/cliextensions/azure-devops/azext_devops/dev/common/services.py", line 157, in get_token_from_az_login
    auth_token = profile.get_access_token_for_resource(user, tenant, '499b84ac-1321-427f-aa17-267ca6975798')
  File "/usr/lib/python2.7/site-packages/azure/cli/core/_profile.py", line 506, in get_access_token_for_resource
    username, tenant, resource)
  File "/usr/lib/python2.7/site-packages/azure/cli/core/_profile.py", line 916, in retrieve_token_for_user
    " Please run 'az login'." if not in_cloud_console() else ''))
CLIError: Could not retrieve token from local cache. Please run 'az login'.

As per ERROR, when service principal is used for az login, then token is not retrieved. However it works fine if normal user is used. I have these steps in automation pipeline, where there is no human interaction.

To Reproduce
Azure Cli Version: 2.0.76

Azure-Devops extension version:
Use az --version
Look for something like-
Extensions:
azure-devops (0.14.0)

Steps to reproduce the behavior:

1. az login
2. az devops configure defaults.
3. Try any az pipelines command
4. See error

Expected behavior
It should return data

[geverghe]

Hey @swapnild2111 - azure devops, which is an extension to azure cli, does not support service principal. If you are using service principal for automated login, you can use az devops login with PAT to access azure devops. Here is a link to documentation that explains more - https://docs.microsoft.com/en-us/azure/devops/cli/log-in-via-pat?view=azure-devops&tabs=windows. I am changing this from bug to feature request.

[swapnild2111]

Ohh ok, got it. Feature request makes sense then.
Currently I am using PAT login approach and it works fine.
However I want to move to Service Principal Approach since I do not want to keep my personal token linked to CI/CD anywhere. I will watch update for feature request.
Thank you :)

[PhilipSkinner]

Using PAT tokens is not acceptable to us (as a company) or the clients we develop software for.

Can you please make your tools use service principals consistently? Its very hard to explain to a client why this doesn't work when it should be working.

[mboutet]

We need this please Microsoft.

[tlentine]

Hey @swapnild2111 - azure devops, which is an extension to azure cli, does not support service principal. If you are using service principal for automated login, you can use az devops login with PAT to access azure devops. Here is a link to documentation that explains more - https://docs.microsoft.com/en-us/azure/devops/cli/log-in-via-pat?view=azure-devops&tabs=windows. I am changing this from bug to feature request.

I have wasted a TON of time on this issue because the referenced link explicitly calls out the fact that if you have authenticated with az login you do not need to use az devops login.

It would be fantastic if:

• The documentation was accurate;
• az devops extension adds support for service principals

[geverghe]

@tlentine - Apologies for the lack of accuracy in the documentation. I have initiated a PR to get the error in documentation fixed.

@PhilipSkinner, @mboutet, @tlentine There are no near term plans to support az login via service principals to enable access to devops commands. I will keep this on the backlog.

[tlentine]

@geverghe Thanks for the update, and please do keep it on the backlog. Using a Service Principal would be much preferred.

[PhilipSkinner]

Ok so we have to have a PAT token that gives (potentially) elevated access to a project set in our environmental variables (secure or otherwise).

Have you ensured that anybody who has read access to the logs from builds/deployments has the same level of access to modify a build/deployment and their variables? Otherwise this is a security concern. Thanks for that.

[rajakesar]

Is there any plan in the works to offer something other than a Personal Access Token for automation? This issue has been open since November.

[langecode]

We have a similar case using FluxCD for GitOps. The FluxCD process (running in our Kubernetes cluster) needs to be able to do a Git pull against Azure DevOps. Currently we configure FluxCD with a PAT, however, it is really not very convenient to store a real users token as part of the CI/CD setup - what if this person leaves the company? Would much prefer to be able to do such things using a service principal.

[PhilipSkinner]

Not only that, its inconsistent with how the tools work on your local machine.

Make one set of CLI tools and use them everywhere, don't make several versions - how on earth are we supposed to gain any confidence in our pipeline definitions when the commands we run locally function differently to those on the CI? Basic stuff guys.

[las3r]

Why was this put "on the backlog"? In my honest opinion this is one of the basic authentication flows that are supported all through Azure, with Azure DevOps being an outlier.

For reference, we are trying to create Azure Repos from a CI-pipeline to automate our "request a repo" feature. Without an SPN we will need to generate a PAT. This PAT will need to be an Administrator PAT, which leads to a host of security issues, let alone issues when the administrators "leave" the organisation, rendering our pipelines broken. The alternative is just as bad, a service accounts - something we are trying to move away from.

When can we expect Service principal-based authentication in the Azure CLI for DevOps?

[PhilipSkinner]

Had to raise this with another client again today, its been documented as an issue with the azure devops offering and this ticket has been linked to.

[kchri-nets]

voting for this too. Very inconvenient to use PAT when automizing devops resource creation!

[las3r]

Any news on this @geverghe

[ThomasDetemmerman]

I was having the same issue. As a workaround, I use the REST API. The solution uses the system access token. To make this work, the "Build Service" needs "Queue build" permissions (https://docs.microsoft.com/en-us/azure/devops/pipelines/policies/permissions?view=azure-devops#pipeline-permissions)

For example, trigger annother pipeline:

    - task: PowerShell@2
      inputs:
        targetType: 'inline'
        script: |
          $body = @(     @{  "definition"= @{ "id"= 91 } } )
          $body = $($body | ConvertTo-Json)
          $url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/builds?api-version=6.1-preview.6"
                  Write-Host "URL: $url"
                  $pipeline = Invoke-RestMethod -Uri $url `
                     -Headers @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"} `
                     -method POST `
                     -ContentType 'application/json' `
                     -Body $body
                  Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
      env:
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)

[PhilipSkinner]

I have a more permanent work around, I now no longer work on anything azure based :).

[las3r]

@ThomasDetemmerman That will only work in Azure DevOps context. We have some automatic provisioning scripts that run outside of azure devops, so this won't help us here. Meanwhile we're having all kinds of problems with AD Conditional access blocking Azure DevOps "user" pats, because we are unable to use service principals. I urge you, @geverghe to take this in as a feature request.

[jaalsh]

We also have provisioning scripts running outside DevOps, just bumping to highlight it's not an isolated case.

The problem is exacerbated using GitOps (e.g. FluxCD mentioned above) as your entire infrastructure is dependent on the PAT being valid so it can pull in your IaC repo.

[KseniaIV]

We need this for scaling build servers, this is a basic requirement for security. Please implement.