Terraform-aws-eks: When using the irsa getting an issue with OpenIDConnect

Created on 31 Jul 2020 · 6Comments · Source: terraform-aws-modules/terraform-aws-eks

I have issues

I'm submitting a...

[ ] bug report
[ ] feature request
[x] support request - read the FAQ first!
[ ] kudos, thank you, warm fuzzy

What is the current behavior?

Using the irsa with the module has been giving me an error where there is no OpenIDConnect provider found in your account.
If I add an OpenIDConnect by hand it works. the only thing i changed was adding my account number and changing the name of the role.

If this is a bug, how to reproduce? Please include a code sample if relevant.

What's the expected behavior?

Are you able to fix this problem and submit a PR? Link here if you have already.

Environment details

Affected module version:
OS:
Terraform version:

Any other relevant info

Source

LouisDinatale

Most helpful comment

The module will create the irsa resource if you set enable_irsa = true

dpiddockcmp on 18 Sep 2020

👍3

All 6 comments

What is the error?

What is the code you're trying to run?

dpiddockcmp on 3 Aug 2020

Think I have been having the very same issue, deployed eks with austoscaling enabled and created IRSA but after deploying autoscaler on the gluster I get the following

E0813 16:43:55.729097       1 aws_manager.go:261] Failed to regenerate ASG cache: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.eu-west-1.amazonaws.com/id/<REDACTED>
        status code: 400, request id: d3971115-8cf9-4478-866c-034a2b27afe4
F0813 16:43:55.729117       1 aws_cloud_provider.go:376] Failed to create AWS Manager: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.eu-west-1.amazonaws.com/id/<REDACTED>
        status code: 400, request id: d3971115-8cf9-4478-866c-034a2b27afe4

The code I used:

locals {
  cluster_name                  = "${var.environment}-eks-cluster-${random_string.suffix.result}"
  k8s_service_account_namespace = "kube-system"                               # for IRSA cluster autoscaler
  k8s_service_account_name      = "cluster-autoscaler-aws-cluster-autoscaler" # for IRSA cluster autoscaler
}

# Iam roles for service accounts - to enable cluster austoscaling
module "iam_assumable_role_admin" {
  source                        = "../../modules/iam-assumable-role-with-oidc"
  create_role                   = true
  role_name                     = "cluster-autoscaler"
  provider_url                  = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
  role_policy_arns              = [aws_iam_policy.cluster_autoscaler.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:${local.k8s_service_account_namespace}:${local.k8s_service_account_name}"]
}

resource "aws_iam_policy" "cluster_autoscaler" {
  name_prefix = "cluster-autoscaler"
  description = "EKS cluster-autoscaler policy for cluster ${module.eks.cluster_id}"
  policy      = data.aws_iam_policy_document.cluster_autoscaler.json
}

data "aws_iam_policy_document" "cluster_autoscaler" {
  statement {
    sid    = "clusterAutoscalerAll"
    effect = "Allow"

    actions = [
      "autoscaling:DescribeAutoScalingGroups",
      "autoscaling:DescribeAutoScalingInstances",
      "autoscaling:DescribeLaunchConfigurations",
      "autoscaling:DescribeTags",
      "ec2:DescribeLaunchTemplateVersions",
    ]

    resources = ["*"]
  }

  statement {
    sid    = "clusterAutoscalerOwn"
    effect = "Allow"

    actions = [
      "autoscaling:SetDesiredCapacity",
      "autoscaling:TerminateInstanceInAutoScalingGroup",
      "autoscaling:UpdateAutoScalingGroup",
    ]

    resources = ["*"]

    condition {
      test     = "StringEquals"
      variable = "autoscaling:ResourceTag/kubernetes.io/cluster/${module.eks.cluster_id}"
      values   = ["owned"]
    }

    condition {
      test     = "StringEquals"
      variable = "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled"
      values   = ["true"]
    }
  }
}

module "eks" {
  source       = "../../modules/terraform-aws-eks"
  cluster_name = local.cluster_name
  subnets      = module.vpc-eks.private_subnets
  vpc_id       = module.vpc-eks.vpc_id

  cluster_endpoint_private_access      = true
  cluster_endpoint_public_access_cidrs = ["REDACTED"]

  worker_groups = [
    {
      name                = "on-demand-1"
      instance_type       = "m5.large"
      asg_max_size        = 10
      kubelet_extra_args  = "--node-labels=spot=false"
      suspended_processes = ["AZRebalance"]
      tags = [
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "propagate_at_launch" = "false"
          "value"               = "true"
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${local.cluster_name}"
          "propagate_at_launch" = "false"
          "value"               = "true"
        }
      ]
    }
  ]
  worker_groups_launch_template = [
    {
      name                    = "spot-1"
      override_instance_types = ["m5.large", "m5a.large", "m5d.large", "m5ad.large"]
      asg_desired_capacity    = 2
      asg_max_size            = 10
      kubelet_extra_args      = "--node-labels=node.kubernetes.io/lifecycle=spot"
      tags = [
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "propagate_at_launch" = "false"
          "value"               = "true"
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${local.cluster_name}"
          "propagate_at_launch" = "false"
          "value"               = "true"
        }
      ]
    },
  ]
}

resource "helm_release" "cluster_autoscaler" {
  name       = "cluster-autoscaler"
  repository = "https://kubernetes-charts.storage.googleapis.com"
  chart      = "cluster-autoscaler"
  version    = "7.3.4"
  namespace  = "kube-system"

  set {
    name  = "repository"
    value = "us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler"
  }

  set {
    name  = "imageTag"
    value = "v1.16.5"
  }

  set {
    name  = "cloudProvider"
    value = "aws"
  }

  set {
    name  = "replicaCount"
    value = "3"
  }

  set {
    name  = "awsRegion"
    value = "eu-west-1"
  }

  set {
    name  = "rbac.create"
    value = "true"
  }

  set {
    name  = "rbac.serviceAccountAnnotations.eks\\.amazonaws\\.com/role-arn"
    value = "arn:aws:iam::${var.aws_account_id}:role/cluster-autoscaler"
    type  = "string"
  }

  set {
    name  = "autoDiscovery.enabled"
    value = "true"
  }

  set {
    name  = "autoDiscovery.clusterName"
    value = "<REDACTED>"
  }
}

Service account on the cluster exists:

kubectl get serviceaccount --all-namespaces | grep cluster-autoscaler-aws-cluster-autoscaler
kube-system       cluster-autoscaler-aws-cluster-autoscaler   1         40m

kubectl describe sa cluster-autoscaler-aws-cluster-autoscaler -n kube-system
Name:                cluster-autoscaler-aws-cluster-autoscaler
Namespace:           kube-system
Labels:              app.kubernetes.io/instance=cluster-autoscaler
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=aws-cluster-autoscaler
                     helm.sh/chart=cluster-autoscaler-7.3.4
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<REDACTED>:role/cluster-autoscaler
                     meta.helm.sh/release-name: cluster-autoscaler
                     meta.helm.sh/release-namespace: kube-system
Image pull secrets:  <none>
Mountable secrets:   cluster-autoscaler-aws-cluster-autoscaler-token-k2chc
Tokens:              cluster-autoscaler-aws-cluster-autoscaler-token-k2chc
Events:              <none>

Can you advise please?