Amazon-vpc-cni-k8s: cannot get AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true tags applied on node after following the instruction on the url

[hetpats]

What happened: we enabled per pod eni on the cluster by follwoing url https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html.
but we cannot get the self managed nodes to update the tags to AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true hence when we deploy a pod with SG it fails to schedule with following error..
ERROR while deploying pod or deployments:
32m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6s093 0/2 nodes are available: 2 Insufficient vpc.amazonaws.com/pod-eni.
32m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6s093 skip schedule deleting pod: dev-jenkins/dev-tfdeployer-kbbm8-6s093
41m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6zzl7 no nodes available to schedule pods
41m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6zzl7 no nodes available to schedule pods
41m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6zzl7 0/2 nodes are available: 2 Insufficient vpc.amazonaws.com/pod-eni.
40m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-6zzl7 skip schedule deleting pod: dev-jenkins/dev-tfdeployer-kbbm8-6zzl7
45m Warning FailedScheduling pod/dev-tfdeployer-kbbm8-78d1w 0/2 nodes are available: 1 Insufficient pods, 1 Insufficient vpc.amazonaws.com/pod-eni.
45m Normal NotTriggerScaleUp pod/dev-tfdeployer-kbbm8-78d1w pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 Insufficient vpc.amazonaws.com/pod-eni

Environment:dev

• Kubernetes version (use kubectl version):1.17
• CNI Version 1.7.8
• OS (e.g: cat /etc/os-release):amznlinux2
• Kernel (e.g. uname -a):4.14.203-156.332.amzn2.x86_64

[jayanthvn]

Hi @hetpats

Just to understand, you enabled per pod and configured custom networking on aws-node DS, then you are trying to schedule pods on it right?

[hetpats]

yes correct , but the host do not update the tag to true hence the pod is in pending state

[jayanthvn]

Can you please confirm if you followed the below steps to enable custom networking?

https://aws.amazon.com/premiumsupport/knowledge-center/eks-multiple-cidr-ranges/

[abhipth]

@hetpats thanks for reaching out. It looks like you are trying to launch pods using security groups but there are no nodes with capacity to schedule these pods. Could you please ensure that you have AmazonEKSVPCResourceController attached to your cluster role. The above policy allows the vpc-resource-controller to attach Trunk ENI and advertise the vpc.amazonaws.com/pod-eni capacity on the given node.

The detailed steps are present in the following document - https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html

Please let us know if this resolves your issue.

[hetpats]

Looks like we had a deny policy attached to cluster role that was not allowing the tags on subnet hence the error.

kubernetes.io/cluster/ shared was missing on the subnets.
Please Mark this issue as resolved
Thanks