Aad-pod-identity: Limitation "A maximum of 50 pod identities are allowed for a cluster." for real or just in Preview?

[lucasmaj]

The limitation "A maximum of 50 pod identities are allowed for a cluster" from here: https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity

... is it just in Preview or is it going to stay?

[chewong]

The limitation only applies when you use the managed experience of AAD Pod Identity (creating AzureIdentity & AzureIdentityBinding with azure-cli). @bcho do you know why the maximum number of pod identities is 50?

If you use the open-source solution, there is no limitation on the number of pod identities you can create. For more questions on the managed experience, you can open an issue at https://github.com/Azure/AKS/issues

[bcho]

hi @lucasmaj, yes, we set this limit for preview stage. Do you have a use case that needs more than 50 pod identities? We want to check with some real world use cases and set a proper limit here (due to VMSS side limitation). cc @miwithro

[lucasmaj]

Thank you for consideration on this topic. @bcho Do you have a use case that needs more than 50 pod identities? No, right now I don't. But I do have a SaaS solution made of 30 microservices, each microservice is a k8s Deployment. (mostly to allow for scaling of each microservice individually) Does each microservice/deployment need unique identity? Hopefully not. Maybe yes.

It is just weird that while a cluster has much higher (> (30 * 100)) limits on number of pods, you would be limited by unique identities? As your cluster grows, namespaces are added, more apps and therefore pods gets added, you might run out of pod identities, and the only answer would be "Sorry, get a new cluster. While you could have more pods, we can't give them new identities." Security considerations would directly limit your cluster size?

Am I missing something? Please advise.

[miwithro]

No, it was just set as a base limit for the preview. We will do some testing to introduce higher levels going forward.

[lucasmaj]

Thanks. Earlier I was trying to say that having any limit would feel weird IMHO. Ideally you would not want situation where "Security considerations/limits would put limit on your cluster growth."

[kwaazaar]

Isn't that a limit per nodepool (VMSS)?

Regarding the limit: we use a release pipeline that creates an identity for every single solution that is deployed to the cluster. So every pod (deployment/job) has its own identity. Functionally they could share identities, but since there was no limit before, we simply create separate ones for each pod. Rights are also assigned to these identities through a Azure Devops release pipeline, so maintenance of the identities is no issue. I doubt we are the only ones taking this approach.

[jeanpbond]

@bcho Do you have a use case that needs more than 50 pod identities?

Yes, we have one identity per microservice and we are planning to deploy between 30-50 microservices per environnement. We have clusters in which we deploy up to 4 environments, so we are planning to use 120-200 identities in some clusters. We use one identity per service to secure access to certain resources that are specific to each microservice (Ex: Key Vault).

[bcho]

@kwaazaar @jeanpbond thanks for the info! Cool use cases, we will re/evaluate these usages in our end.

[dhirschfeld]

Just chiming in here that our deployment model assigns an individual identity per app so that the permissions for each app can be assigned, using the principle of least privilege, only what it actually requires.

Having a single identity for every app which is given privileges to all resources that any app may possibly need is less than ideal from a security PoV. It might be possible to share an identity for apps which require access to exactly resources A, B & C and another for apps which need access to A & D but that would be a bit of a logistical nightmare. It's a very straightforward model to create an identity per app and assign only the permissions it requires.

That said we wouldn't hit the 50 identity limit managed service just yet, but we're only beginning our deployment to AKS so we might eventually require a couple of hundred identities per cluster with the deployment model we're using at the moment.

[ericxw]

Adding to the conversation here as well. We have a number of applications planned to get moved into AKS. I'd like to get away from the model where multiple applications and environments share an identity (seems people do it for convenience, but is terrible for security and diagnostics). In such a case, a limit of 50 at the current base setup of 3 environments per application would limit us to 16 applications total, but in the case we have additional environments, we'll start to feel the pain rather soon. Once we open the flood gates up more to automated testing, things are going to get even more hairy.

This won't be realistic for the long run and so we're hoping to see this limit increased. Curious, are there performance considerations, let's say beyond 1000 or something? Any hard limits should be set based on performance/functionality limitations. An optional soft limit can be added which can be overwritten by customer.

[AvigdorLevy]

Shouldn't it be at least 1 identity per namespace (w/o limitation as namespace r not limited)?