Aad-pod-identity: Windows support for AAD Pod Identity

[PatrickLang]

Describe the request
Support Windows pods running on Windows nodes with AAD identity

Describe the solution you'd like

Parity between Windows & Linux for AAD pod identity

Please feel free to +1 this issue and help explain your use cases :)

Describe alternatives you've considered

Another alternative is to use an Azure service principal stored in a kubernetes secret, and use that to obtain other secrets from Azure KeyVault as needed.

Additional context

This would need some changes to how things are deployed:

1. The service used to intercept and respond to IMDS calls would need to run on the Windows node directly, not in a container. Host networking and privileged containers are not supported for Windows containers.
2. DNAT rules need to use the Windows host networking service instead of IPTables. It may make sense to use an annotation in the CNI configuration to automatically enable this as NICs are added to pods.

[PatrickLang]

cc @kkmsft @dineshgovindasamy - opened this for tracking the investigation

[PatrickLang]

Here's what it would take to get a preview. It would only work for availabilitysets, not node pools, so it wouldn't work with AKS.

1. Build the NMI (https://github.com/Azure/aad-pod-identity/tree/master/cmd/nmi) as a Windows binary
2. Add service control manager support (see golang.org/x/sys/windows/svc)
3. Write an AKS-Engine extension to
   
   
   
   • install & register it as a service on the Windows host
   
   
   • Modify the CNI config to include a proxy from the IMDS IP to the node's IP/port
   
   

"Name":  "EndpointPolicy",
    "Value":  {
        "Type":  "PROXY",
        "IP":  "169.254.169.254",
        "Port": "80",
        "Destination":  "10.240.0.4:8090"
    }

To get to beta and stable:

1. Publish nmi Windows build to a trusted CDN
2. Move the setup steps into AKS-Engine for all Windows nodes to support availabilityset & vmss
3. Add a flag to agentPools to opt in/out of deploying NMI

[david-jarman]

Thanks for opening this! Our team uses Windows containers and we would like to move use AAD Pod Identity. Currently, we use MSI inside our containers by running a prestartup script in the entrypoint of our containers. The script is based on the answer of this StackOverflow question, which just adds a route to the MSI endpoint in Azure:

$instanceMetadataServiceEndpoint = '169.254.169.254/32';
$gateway = Get-NetRoute -DestinationPrefix '0.0.0.0/0';
if ($gateway -ne $null -and $gateway -isnot [array])
{
    New-NetRoute -DestinationPrefix $instanceMetadataServiceEndpoint -InterfaceIndex $gateway.ifIndex -NextHop $gateway.NextHop | Out-Null;
}

[rajarshi-singh]

Our use case also requires pod identity on Win containers. We are dependent on AKS, so I am not sure when this feature would land.

[wasker]

@PatrickLang Is there any ETA for this work?

[wasker]

As it is now, Windows containers can be unblocked by having managed identities assigned to VMSS, but there's always a risk that AKS will not re-apply them to the VMSS whenever nodepool is updated/deleted or newly added.

@PatrickLang If implementing solution for this issue is too much work, would it be possible to extend az aks create command to take a list of managed identities to be assigned to VMSS (not kubelet identities, but just identities that are applied to VMSS whenever AKS changes VMSS during mgmt operations)?

[PatrickLang]

It's still in progress - some folks at Microsoft are working on a prototype. I don't have an ETA that I can share at this point

[hh23485]

It's still in progress - some folks at Microsoft are working on a prototype. I don't have an ETA that I can share at this point

Is this support for windows container now? or any other work around ways to use it in Windows node pool?

[hh23485]

It's still in progress - some folks at Microsoft are working on a prototype. I don't have an ETA that I can share at this point

Is this support for windows container now? or any other work around ways to use it in Windows node pool?

Seems It's ok to get msi in windows container from linux node pool for me.

[asumner]

Seems It's ok to get msi in windows container from linux node pool for me.

Hi, did you get pod identity working with windows containers?

[hh23485]

Seems It's ok to get msi in windows container from linux node pool for me.

Hi, did you get pod identity working with windows containers?

Yes, It's working for me. these repo's container running in Linux node pool, but it's ok to login with MSI in windows containers.

[ceilingfish]

We're assigning managed identities to the VMSS, can we get clarification on whether that's going to be supported going forward as a means of obtaining a managed identity?

[aramase]

We're currently working on a MVP solution for aad-pod-identity on windows. There is currently no ETA that I can share at this point.

cc @phillipgibson for more details.

[harikumarks]

Hi @aramase any ETA on this that you can share

[phillipgibson]

We are in the late phases of testing out a cut of an MVP release. More details to come soon.

[zhiweiv]

Any update?

[phillipgibson]

We're still in testing. Unfortunately there's not a specific date I can provide as we are ensuring that all of our tests pass.

[jlteodoro]

Is this supported now?

[phillipgibson]

@jlteodoro we're in the final stages of testing. These test require a lot of coordination with the core Windows group testing.

[RobLSDev]

Is there a timeline available for when this might go to preview or GA?

[Tankatronic]

@jlteodoro we're in the final stages of testing. These test require a lot of coordination with the core Windows group testing.

Is there an update on the status of testing?

[chewong]

We won't be supporting Windows for aad-pod-identity v1 going forward. We are actively working on aad-pod-identity v2 internally, which will support both Linux and Windows workload out of the box. We will open-source it once it's ready for consumption.