Azure-devops-cli-extension: [Feature Request] service-endpoint create needs option to specify pipeline policies

[jfpanisset]

When you create a service connection via the GUI, you get a check box to specific "Allow all pipelines to use this connection". This causes not only a POST connection to be sent to the _apis/serviceendpoint/endpoints REST API, but also a PATCH request to _apis/build/authorizedresources containing:

[{"authorized":true,"id":"the-resource-id","name":"the-resource-name","type":"endpoint"}]

If instead you use the CLI to create a service connection using:

az devops service-endpoint create ----service-endpoint-configuration foo.json

the PATCH request is not sent, and by default the service connection cannot be accessed by the pipeline until you enable that manually in the GUI. There may be a way to manipulate this permission using the az CLI (and if there is I'd love to know about it), it would be very convenience to add a command line option named something like --all-pipelines-use which would default the access to the service connection to all pipelines in the project. Even better would be explicit CLI support to manipulate access from specific pipelines if required.

[atbagga]

@jfpanisset

The policy cannot be at pipeline level since that is not supported. It is either all pipelines or none.

Till the command is enhanced to support the option you can use this command to change that setting from the CLI (complete your automation flow).

az devops invoke --http-method patch --area build --resource authorizedresources --debug --route-parameters project=atbagga --api-version 5.0-preview --in-file .\a.txt --encoding ascii

Where a.txt will contain the json that you mentioned in the issue.

[{"authorized":true,"id":"the-resource-id","name":"the-resource-name","type":"endpoint"}]

[jfpanisset]

Thank you very much, az devops invoke seems to work and correctly sets the "Allow all pipelines to use this service connection" property for my newly created Docker Hub service connection. Before I got your answer I was trying to use az rest --method patch ... but wasn't getting anywhere.

Until az devops service-endpoint create can be enhanced with this functionality, would it make sense to add this recommendation to the documentation?

[geverghe]

Can we investigate what it means to enable this in the command? @jfpanisset - Feel free to chime in based on your expectation.

1. az devops service-endpoint create --enable-for-all

Input - service end point name and other required inputs based on service endpoint type
Output - json output of the create service endpoint connection,
Advantages

1. One command to achieve objective

Drawback

1. Cannot understand what happened to the status of enable for all; did it successfully happen? was there an error?
   [Potentially follow up with the team to include this detail in the json output to understand status]
2. What if I want to enforce the policy to an existing pipeline - can't create again.

2. az devops service-endpoint create followed by az devops service-endpoint enable-for-all

Advantages

1. Modular and therefore available for even existing pipelines
2. Each command corresponds to an API and therefore JSON outputs are reflective of the actions undertaken.

3. az devops service-endpoint create --enable-for-all and az devops service-endpoint enable-for-all

Advantages

1. Available as both a command and as a parameter enabling users to avoid two separate commands when creating a service connection.
2. The separate command allows users to execute the same for existing service connections.

Disadvantages:

1. Cannot understand what happened to the status of enable for all when bundled with create; did it successfully happen? was there an error?
   [Potentially follow up with the team to include this detail in the json output to understand status]

My proposal would be Option 2 as the starting point and evaluate if 3 is possible by checking if the current API has this policy enforcement data in the API output and if not whether it is possible to get it included in the data.

[atbagga]

@geverghe we already handle this is variable-groups.
Check create and update commands - - authorize parameter is what we use.

So we have taken approach 1 in that case.

Drawbacks are not applicable there...
Status of authorize is appended to the output and you can manipulate it using update command.

[geverghe]

That's awesome! However, I see that we have not enabled update service endpoint. So then, this would involve a few command updates and potentially new commands across all service end point command groups

Command updates

az devops service-endpoint create --enable-for-all

az devops service-endpoint github create --enable-for-all

az devops service-endpoint azurerm create --enable-for-all

New commands

az devops service-endpoint update --enable-for-all

az devops service-endpoint github update --enable-for-all

az devops service-endpoint azurerm update --enable-for-all

[jfpanisset]

Interestingly I've been using az devops service-endpoint github create followed by az pipelines create ... --service-connection and have not had any authorization errors with the created pipeline having access to the GitHub service connection, even though I haven't made any explicit calls to manipulate its permissions. In the AZ DevOps UI it shows up as having "Authorized Pipelines" set to the pipeline I created, so it seems the act of creating a pipeline and setting its --service-connection is in fact authorizing that service connection?

I don't have a strong opinion on the exact syntax to be used to authorize service connections, as long as it is well documented: whatever makes the most sense from an implementation and maintainability point of view.

In order to expose all of the GUI options to the CLI, would it make sense to also support options to authorize service connections only for certain pipelines in the project? For instance if I have separate pipelines for "build" and "push packages to a registry", it would make sense to only have the service connection authorized for the "push packages to a registry" pipeline.

[atbagga]

@jfpanisset I am not exactly sure of this.. but this is a probable explanation of the behavior you see.

When you are the owner of the service connection and you execute the first run of the pipeline that service connection is authorized to be used by the pipeline implicitly.

Problem happens when you try to use someone else's service connection in your pipeline. The pipeline run will fail until either that service connection is marked to be authorized for all pipelines OR the owner of the service connection authorizes this one pipeline to be authorized by triggering the first run.

Using APIs this functionality of authorizing only a single pipeline for using a service connection is not exposed.

[atbagga]

@geverghe Would it be good enough to just have one command - az devops service-endpoint update --id <SERVICE_ENDPOINT_ID> --enable-for-all for now.

It will be two steps for all users but that way we do not break compat by changing json response of non-preview commands.

This command will be released in preview for now.

[geverghe]

Got it - makes sense. Let's target only update then to start with.

[atbagga]

Fixed in version 0.16.0.
ETA for release end of this week.

[jfpanisset]

I can confirm that:

az devops service-endpoint update --id $DOCKER_CONNECTION_ID --enable-for-all

correctly sets the "Grant access permission to all pipelines" property for my recently created Docker Hub service connection, and I no longer need to use "az devops invoke --http-method patch ...".

Thank you very much!

[manu-kanwarpal]

Hi Team,

I'm also facing this issue and the suggested workaround doesn't seem to work. Here's the error I get when I try it:

Starting: Update endpoint

Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.182.0
Author : Microsoft Corporation

Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash

Generating script.
Script contents:
az devops service-endpoint update --id $DOCKER_CONNECTION_ID --enable-for-all
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/f6cb7b4a-8332-4eb1-8a20-639a0e142ec3.sh
ERROR: argument --id: expected one argument

TRY THIS:
https://aka.ms/cli_ref
Read more about the command in reference docs

[error]Bash exited with code '2'.

Finishing: Update endpoint

[gauravsaralMs]

@manu-kanwarpal

• are you sure that $DOCKER_CONNECTION_ID has some value
• have you tried running the command locally, if it is working locally and not in your pipeline then that is an issue with your pipeline and not azure devops cli
• can you add --debug to the command and then share the output

[manu-kanwarpal]

Hi @gauravsaralMs

Thanks for the prompt response. I don't think $DOCKER_CONNECTION_ID has a value, but I used it because that was the workaround suggested here. I also tried $GITHUB_CONNECTION_ID as I'm creating a Github service connection.

Here's the result of --debug.

Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'devops': ['azext_devops']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: Total (0) 0.000 0 0
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
DEBUG: cli.azure.cli.core: azure-devops 0.127 60 191 /opt/az/azcliextensions/azure-devops
DEBUG: cli.azure.cli.core: Total (1) 0.127 60 191
DEBUG: cli.azure.cli.core: Loaded 60 groups, 191 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : devops service-endpoint update
DEBUG: cli.azure.cli.core: Command table: devops service-endpoint update
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/vsts/.azure/commands/2021-06-08.14-27-07.devops_service-endpoint_update.1765.log'.
INFO: az_command_data_logger: command args: devops service-endpoint update --id --enable-for-all --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x7f1cb6b87ea0>, .register_query_examples at 0x7f1cb6ae0f28>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x7f1cb6aef048>, .add_cache_arguments at 0x7f1cb6aef158>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): app.aladdin.microsoft.com:443
DEBUG: urllib3.connectionpool: https://app.aladdin.microsoft.com:443 "GET /api/v1.0/suggestions?query=%7B%22command%22%3A+%22devops+service-endpoint+update%22%2C+%22parameters%22%3A+%22--enable-for-all%2C--id%22%7D&clientType=AzureCli&context=%7B%22versionNumber%22%3A+%222.24.0%22%2C+%22errorType%22%3A+%22ExpectedArgument%22%2C+%22correlationId%22%3A+%22276ab2fa-4f54-4312-9ed9-0ebde2d63651%22%7D HTTP/1.1" 200 None
ERROR: cli.azure.cli.core.azclierror: argument --id: expected one argument
ERROR: az_command_data_logger: argument --id: expected one argument

TRY THIS:
https://aka.ms/cli_ref
Read more about the command in reference docs
DEBUG: cli.knack.cli: Event: Cli.PostExecute []
INFO: az_command_data_logger: exit code: 2
INFO: cli.__main__: Command ran in 0.621 seconds (init: 0.133, invoke: 0.488)
INFO: telemetry.save: Save telemetry record of length 2946 in cache
WARNING: telemetry.check: Negative: The /home/vsts/.azure/telemetry.txt was modified at 2021-06-08 14:26:53.155118, which in less than 600.000000 s

[error]Bash exited with code '2'.

Finishing: Update endpoint

I can try locally, but I'm using a simple bash step in Azure DevOps pipelines and this is a part of the same product group I imagine.

Cheers,
Manu

[gauravsaralMs]

don't think $DOCKER_CONNECTION_ID has a value, but I used it because that was the workaround suggested here

Please spend some more time going through issue and suggestions

I can try locally, but I'm using a simple bash step in Azure DevOps pipelines and this is a part of the same product group I imagine.

@manu-kanwarpal : I will highly recommend to try it first locally.

You should try to break your problem and solve it step by step

1. Articulate your problem clearly
2. Write bash script locally to solve your problem
   please read documentation of CLI as and when needed
3. Run your bash script in ADO pipeline

[manu-kanwarpal]

Hi @gauravsaralMs

1. The problem statement is that when I create a new service-endpoint for Git, I'd like to specify the pipeline policy such that --enable-for-all is also set at the time of creation. This is exactly what the original thread is about. It looks like this suggestion was not implemented and the suggested workaround was to run the following command after creating the service endpoint:

az devops service-endpoint update --id $DOCKER_CONNECTION_ID --enable-for-all

When I run this, I get the errors I have shared above, because as far as I can tell $DOCKER_CONNECTION_ID doesn't have any value.

1. I just tried to run a bash script locally and I see the same issue, $DOCKER_CONNECTION_ID doesn't have any value
2. The bash script is not running locally, so no point in running it in ADO as I was running the exact same steps there already.

I'd appreciate if you can test this at your end and see if it works for you.

Cheers,
Manu

[gauravsaralMs]

@manu-kanwarpal

az devops service-endpoint update --id $DOCKER_CONNECTION_ID --enable-for-all

here $DOCKER_CONNECTION_ID is the variable for passing service endpoint ID

see documentation of this command for more details
https://docs.microsoft.com/en-us/cli/azure/devops/service-endpoint?view=azure-cli-latest#az_devops_service_endpoint_update

I tested this locally (not using bash script or ADO pipeline) and it is working fine for me