Terraform-provider-azuredevops: Bad request "InvalidApiVersionParameter" after service endpoint creation

[ac-obair]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and Azure DevOps Provider) Version

Terraform v0.12.28

• provider.azuread v0.10.0
• provider.azuredevops v0.0.1
• provider.azurerm v2.16.0
• provider.kubernetes v1.11.3
• provider.random v2.2.1

Affected Resource(s)

• azuredevops_serviceendpoint_kubernetes

Terraform Configuration Files

resource "azuredevops_serviceendpoint_kubernetes" "example" {
  project_id            = data.azuredevops_project.project.id
  service_endpoint_name = "${data.azurerm_kubernetes_cluster.aks-cluster.name}-${module.k8s-namespace.k8s-namespace-name}"
  apiserver_url         = "https://${data.azurerm_kubernetes_cluster.aks-cluster.fqdn}"
  description           = "test"
  authorization_type    = "AzureSubscription"

  azure_subscription {
    subscription_id   = "c6dXXX"
    subscription_name = "SUB"
    tenant_id         = "698XXX"
    resourcegroup_id  = data.azurerm_resource_group.aks-cluster.id
    namespace         = module.k8s-namespace.k8s-namespace-name
    cluster_name      = data.azurerm_kubernetes_cluster.aks-cluster.name
  }
}

Debug Output

Panic Output

Expected Behavior

Create service endpoint which can be consumed by release

Actual Behavior

Created service endpoint which can't be consumed by release. Attempting to edit the connection in the portal UI returns

An error occured while fetching the AKS cluster. Internal error: Failed to query service connection API: 'https://management.azure.com/subscriptions/c6dXXX/resourcegroups/subscriptions/c6dXXX/resourceGroups/aks-cluster/providers/Microsoft.ContainerService/managedClusters/aks-cluster?api-version=2018-03-31'. Status Code: 'BadRequest', Response from server: '{"error":{"code":"InvalidApiVersionParameter","message":"The api-version '2018-03-31' is invalid. The supported versions are '2020-06-01,2020-05-01,2020-01-01,2019-11-01,2019-10-01,2019-09-01,2019-08-01,2019-07-01,2019-06-01,2019-05-10,2019-05-01,2019-03-01,2018-11-01,2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'."}}'

Steps to Reproduce

1. terraform apply

Important Factoids

• Same issue creating service connections to aks clusters is uksouth and in ukwest.
• The "Cluster" field in the UI is created as "aks-cluster ()" where it shows as "aks-cluster (aks-cluster)" when manually created
• The connection can't associate with the cluster and therefore can't be consumed by the release
• The created connection resource doesn't show up in the list of connection when using az devops service-endpoints list

References

• #0000

[xuzhang3]

Hi @ac-obair Thanks for open this issue. Can you provider more logs or operation details? I cannot reproduce your error.
I can create k8s service connection and consumed by release. az devops service-endpoint show works for me too.

Test script

data "azuredevops_project" "project" {
  project_name = "ADOTest"
}

resource "azuredevops_serviceendpoint_kubernetes" "se_azure_sub" {
  project_id = data.azuredevops_project.project.id
  service_endpoint_name = "Sample Kubernetes"
  apiserver_url = "https://xxxxx.hcp.eastus.azmk8s.io"
  authorization_type = "AzureSubscription"

  azure_subscription {
    subscription_id = "xxxxx"
    subscription_name = "Test Sub"
    tenant_id = "xxxxx"
    resourcegroup_id = "resouregroup"
    namespace = "default"
    cluster_name = "testCluster"
  }
}

Screen shot

k8s service connection

consume by release

release result

CLI show service connections

[ac-obair]

Hey, sure. Following the same steps you've taken:

Screen shots

AZDO_PROVIDER_TEST or any name created using the provider isn't seachable.

Querying a service connection I know already exists vs the one created by the provider.

Attempting to edit the service connection from the portal UI

The same result when using these provider versions instead of the latest

Terraform v0.12.28
+ provider.azuread v0.7.0
+ provider.azuredevops v0.0.1
+ provider.azurerm v1.40.0
+ provider.kubernetes v1.9.0
+ provider.random v2.2.1
+ provider.vault v2.0.0

Same result when creating a service connection for clusters with service principal rbac or with managed indentities which is what we use. I have a sanitised trace output of the build process is there any piece of it specifically you'd like to see?

[xuzhang3]

@ac-obair Can check your SPN id and secret is correct in your Terraform script? Azure DevOps can create the new service connection with fake configs, but will result int authentication fail when we need it. Some times additional space will added if we copy the secret or id from another file.

[ac-obair]

Interesting. I can't query the service connection via cli since it looks like it doesn't exist. I also can't find any other way to query what credentials it was created with beyond my PAT.

Which SPN id and secret are you referring to? I'm using a PAT to create devops resources. I'm using az login to authenticate myself and from the trace output I can see terraform using azure cli to obtain a token. My az account is set to the correct tenant and subscription. If you're talking about the SP used by az devops over the scope of the resource group of the cluster, that is in place and working. Since we have close to 30 service connections currently deployed and running using the same credentials.

I would like to point out that even though the top part of the error message mentions an auth error, the lower part shows a bad request error based on invalid api version parameter. Maybe I'm missing something but I'd expect an authentication error to return some sort of bad client details error message.

I'd also like to mention outwith using the portal UI to create service connections, the following tf works for creating a service connection that is consumable by the release:

resource "local_file" "service-connection" {
     content               = templatefile("${path.module}/connection.tmpl", {
     clusterid             = data.azurerm_kubernetes_cluster.aks-cluster.id
     namespace           = module.k8s-namespace.k8s-namespace-name
     description          = "created via terraform"
     name                  = "${data.azurerm_kubernetes_cluster.aks-cluster.name}-${module.k8s-namespace.k8s-namespace-name}"
     url                   = "https://${data.azurerm_kubernetes_cluster.aks-cluster.fqdn}"
     })
     filename = "${path.module}/connection.json"
 }
resource "null_resource" "servier-connection" { 
  provisioner "local-exec" {
   command = "az devops service-endpoint create --organization https://dev.azure.com/xxx --project xxx --service-endpoint-configuration connection.json"
  }
}

This setup should be using the same token/authentication mechanism, az cli version and extensions etc for putting resources in place as the provider might use. Where the output connection.json, with values redacted looks like:

{
    "administratorsGroup": null,
    "authorization": {
        "parameters": {
            "azureEnvironment": "AzureCloud",
            "azureTenantId": redacted
        },
        "scheme": "Kubernetes"
    },
    "createdBy": {},
    "data": {
        "authorizationType": "AzureSubscription",
        "azureSubscriptionId": redacted
        "azureSubscriptionName": redacted,
        "clusterId": redacted"
        "namespace":  redacted
    },
    "description": "another test",
    "groupScopeId": null,
    "isShared": false,
    "name": redacted,
    "operationStatus": null,
    "owner": "library",
    "readersGroup": null,
    "serviceEndpointProjectReferences": [
        {
            "description": "another test",
            "name": redacted,
            "projectReference": {
                "id": "id",
                "name": "dpd"
            }
        }
    ],
    "type": "kubernetes",
    "url": redacted
}

These values mirror 1:1 the values that are fed into the provider. I can't seem to find a difference.

[xuzhang3]

@ac-obair Sorry I cannot reproduce your error. So I can only guess some potential operations may help you.

1. Check you PAT, make sure this PAT is bind with the organization that you try to use k8s service connections.
2. Use az account show --output table check which subscription you current use in CLI context. Make sure the service principle used to create the service connection is bind with the same subscription that Azure CLI context used.

Authentication error

This error is returned by Azure. Make sure the config you use is correct. Azure DevOps API version is some kind like api-version=5.1, api-version=5.1-preview

[ac-obair]

@xuzhang3 Yea, I'm aware what the api version should be from the ms docs unfortunately. This isn't a test box or test environment running a test setup for test purposes. My environment is set up for all the azure automation for the company, hundreds of resources.

I have a set of data that when passed into a template and applied using az devops cli extension builds a service connection that I can use. This operation runs using specific dependency versions installed cli components, my az aad user and my PAT etc. What I'm getting at is, if it were a problem with my organisation, my user, my pat, my aad account or my environment then it should produce the same or similar results from the cli. If it were a problem with the values I'm feeding into terraform then it should produce the same result when added to the template. The only thing I'm changing is I'm adding the azure devops provider and service connection block instead of the template block, which _does not_ fail per se. It creates a resource that's only visible in the portal UI and not queryable via the cli.

Unless the tf is _not_ using the same dependencies that the cli is using, or my values are wrong, which they aren't. I'm working on sanitising a debug log if it'd help.

[ac-obair]

I had another user run the same terraform from a different environment that they use using their own credentials that work as they're an owner of pretty much everything. Same error.

[ac-obair]

Got this working turns out

resourcegroup_id = data.azurerm_resource_group.aks-cluster.id

Shouldn't be the resource group id despite the name resourcegroup_id. It should be the name of the resource group...

resourcegroup_id = data.azurerm_resource_group.aks-cluster.name. This can be closed but would be nice if the key was more descriptive.

[xuzhang3]

@ac-obair Sorry for reply late. The azuredevops_serviceendpoint_kubernetes did the wrong description, this must misguide you:
resourcegroup_id - (Required) The resource group id, to which the Kubernetes cluster is deployed.
I will update the document.