Cluster-api-provider-azure: Use azure managed identities to authenticate

[rbitia]

/kind feature

Move from using service principals to azure managed identities to authenticate.
https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

More detail coming

[justaugustus]

/milestone v0.3
/priority important-soon
/help

[k8s-ci-robot]

@justaugustus:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/milestone v0.3
/priority important-soon
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

Needs to be scoped.
@juan-lee suggesting User Assigned MSI instead of System-assigned, due to ease of mgmt for the Kubernetes use case.

We need to document this recommendation, before coding it up.

[jadarsie]

Azure Stack does not have managed identities, I would selfishly suggest not to remove support for SPNs.

[alexeldeib]

Some thoughts:

• It's easy to figure out whether the user has provided any valid authentication credentials, or if MSI is available.
• We really only need two types of auth: msi, and client app/secret.
• @jadarsie it looks like the recommended way of picking up the stack environment is by setting AZURE_ENVIRONMENT_FILEPATH and loading the file?
• Is IMDS available in Stack environment? I know it's in all cloud envs. Would be nice to wire this up so the user doesn't even need to provide the details. Otherwise, we can probably add something like AZURE_CLOUD_ENVIRONMENT which should be mapped to these values.
• We should probably use env specific authorizers by default
• It seems like the preference has been for env vars over sdk auth file or manually parsed flags. Okay by me.

To stay as compatible as possible with existing code, I suggest what I consider the least surprising behavior:

• If user specifies client app or secret, we expect both, and fail to validate.
• If user specifies cloud environment without both client app and secret, we fail to validate (MSI does not require this variable, so we could ignore it, but I prefer explicitly failing the user for providing this combination).
• If user provides all of client app, secret, and environment, we use the environment to load the authorization details via the SDK
  
  
  
  • fail in Stack if AZURE_ENVIRONMENT_FILEPATH is unspecified
  
  
  • fail in general if AZURE_CLOUD_ENVIRONMENT is non-empty and invalid.
  
  
  • default to AZUREPUBLICCLOUD
  
  
• If user provides both client app and secret, we auth to AZUREPUBLICCLOUD
• If user specifies nothing at all, we default to MSI.

Examples:

# default, public cloud cluster with managed identity
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT=""
AZURE_CLIENT_ID=""
AZURE_CLIENT_SECRET=""

# gov cloud with managed app
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT="AZUREUSGOVERNMENTCLOUD"
AZURE_CLIENT_ID=""
AZURE_CLIENT_SECRET=""

# public cloud cluster with manual client app
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT=""
AZURE_CLIENT_ID="foo"
AZURE_CLIENT_SECRET="bar"

# works for Stack environment
AZURE_ENVIRONMENT_FILEPATH="/foo/bar"
AZURE_CLOUD_ENVIRONMENT="AZURESTACKCLOUD"
AZURE_CLIENT_ID="foo"
AZURE_CLIENT_SECRET="bar"

# FAILS, Stack requires filepath
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT="AZURESTACKCLOUD"
AZURE_CLIENT_ID="foo"
AZURE_CLIENT_SECRET="bar"

# FAILS, env provided with nothing else (or explicitly fail stack without sp)
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT="AZURESTACKCLOUD"
AZURE_CLIENT_ID=""
AZURE_CLIENT_SECRET=""

# FAILS, public cloud cluster missing manual app
AZURE_ENVIRONMENT_FILEPATH=""
AZURE_CLOUD_ENVIRONMENT=""
AZURE_CLIENT_ID=""
AZURE_CLIENT_SECRET="bar"

[jadarsie]

Sorry @alexeldeib, I missed this.

@jadarsie it looks like the recommended way of picking up the stack environment is by setting AZURE_ENVIRONMENT_FILEPATH and loading the file?

Correct, that's how AKSe works today

Is IMDS available in Stack environment?

Not really, limited and it does not include MSI.

We should probably use env specific authorizers by default

In Stack's case, we also need to set AZURE_AD_RESOURCE using the audience returned by this https://management.azure.com/metadata/endpoints?api-version=1.0 (azure example, each stack instance returns its own set of endpoints)

I think AZURE_CLOUD_ENVIRONMENT should be AZURE_ENVIRONMENT

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[CecileRobertMichon]

keep open

[CecileRobertMichon]

/remove-lifecycle stale

[nader-ziada]

/assign

[devigned]

I'm happy to pair on this. I've spent a fair amount of time on Managed Identities, AAD and the SDKs.

[CecileRobertMichon]

@nader-ziada do you still want to try looking into this one?

[nader-ziada]

I was going to work on it later this week or next week, but I'm okay if someone else is ready to get started on it

[nader-ziada]

@devigned feel free to take over if you would like, I can pick up something else. Thanks

[devigned]

WIP doc that @nader-ziada and I have been working on. Probably get it fleshed out today.

[nader-ziada]

@devigned for the user managed identity, would the user have to create the identity first and pass it along in the cluster yaml? or would the we actually create it?

[devigned]

For user assigned identities, there are a couple routes.

1. The identity could be created out of band, possibly using Azure CLI or other flow, and assigned the desired roles. The identity or identities would then need to be specified by name in the AzureMachine or AzureMachinePool specs.
2. It would be possible to have the manager create the user assigned identity. I could imagine creating a new CRD which would represent this identity and the roles requested for the identity. Instances of this resource could referenced from AzureMachine and AzureMachinePool specs.

Note: VM and VMSS support assigning multiple user assigned identities to machines. The AZURE_CLIENT_ID env var should contain the CLIENT_ID of the identity to be used via the Azure SDK for Go. The CLIENT_ID will be used by the SDK to request a token from the IMDS service running on the VM.

[jseely]

@devigned are the Managed Identities only used for the cloud controller manager?

They aren't being used for the management cluster to provision workload clusters, are they?

Also, will we also support Service Principals for environments like Azure Stack, per @jadarsie's comment?

[devigned]

Managed Identities are not currently used in the cloud controller manager, but they definitely could be. I would go as far as to say they should be the preferred identity for cloud controller manager in environments that support managed identities.

We _must_ also support Service Principal credentials.

[ncdc]

Will we be able to test all the various auth options in CI?

[devigned]

@ncdc, we will be able to test for workload clusters no problem. Testing a control plane cluster would be a little odd. I think we would want to create a workload cluster, then pivot control plane to that cluster since managed identity would only be available in the Azure provisioned machines.

Does that make sense?

[ncdc]

Yes, that makes sense. Thanks. I'd imagine we'd want to test all the relevant scenarios for:

• Bootstrap cluster (not running in Azure)
• Workload cluster
• Management cluster running in Azure

[devigned]

After discussing with @CecileRobertMichon, we would like to propose the following:

• scope user assigned identities to only "bring your own identity"
• add an array of identities to AzureMachineSpec

Something like:

type Identity struct {
    ProviderID string `json:"providerID"`
}

type AzureMachineSpec struct {
    // ....
    UserAssignedIdentities []Identity `json:"userAssignedIdentities,omitempty"`
    // ....
}

AzureMachine should validate that if VMIdentity == "UserAssigned", the UserAssignedIdentities array is populated with at least one identity.

The ProviderID for the identity should be set by a user to indicate the resource is already created and should not be created or otherwise managed by the controller. The list of ProviderIDs can be validated and transformed into Azure resource IDs, which are required in the API request body.

In the future, we will likely change the ProviderID to be optional, and add to the Identity struct to allow for creation of identities.

[CecileRobertMichon]

@devigned s/AzureVirtualMachine/AzureMachineSpec no?

[devigned]

@CecileRobertMichon what do you mean?

[devigned]

Also, if those field / struct names offend any senses, please offer changes. They were off the cuff.