External-dns: AWS cross account access with OIDC provider in EKS & externalDNS not working

[michazt]

Hello to everyone,

I hope this is the right place to ask my question. If not please point me to the correct place or documentation.

We are in the process of setting up an AWS EKS cluster that uses the OIDC provider for IAM access control to AWS resources. So far this works fine for the ALB ingresss controler. Now we want to also add externalDNS to create DNS records in Route53 for zones hosted in another AWS account.

We have a MGMT AWS account (ID: 222233334444) that is hosting the Route53 DNS zones and a test stage AWS account (ID: 999988887777) where we run the EKS cluster. Now we want to setup externalDNS for a cross account access to add/change the DNS entries hosted in the MGMT account. See below for the config files and error message we get.

Please let me know if I missed something or you need more information.

Michael

====== config details ======
When I try to setup externalDNS I get the following error:
```kubectl apply -f external-dns_testversion.yaml
deployment.apps/external-dns created
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created

kubectl logs -n kube-system external-dns-XYZ
time="2020-05-28T16:14:23Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[] ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[test.local] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:private AWSZoneTagFilter:[] AWSAssumeRole:\"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\" AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:\"test account\" TXTPrefix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
time="2020-05-28T16:14:23Z" level=info msg="Instantiating new Kubernetes client"
time="2020-05-28T16:14:23Z" level=debug msg="kubeMaster: "
time="2020-05-28T16:14:23Z" level=debug msg="kubeConfig: "
time="2020-05-28T16:14:23Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2020-05-28T16:14:23Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2020-05-28T16:14:25Z" level=info msg="Assuming role: \"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\""
time="2020-05-28T16:14:26Z" level=error msg="AccessDenied: User: arn:aws:sts::999988887777:assumed-role/externalDNSEKSrole/${ID} is not authorized to perform: sts:AssumeRole on resource: \"arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest\"n\tstatus code: 403, request id: ${REQUESt_ID}"

IAM Role for the test account:
```arn:aws:iam::999988887777:role/externalDNSEKSrole

Trust relationships
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::999988887777:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}:sub": "system:serviceaccount:kube-system:external-dns",
          "oidc.eks.eu-central-1.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999988887777:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attached IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest
            ]
        }
    ]
}

IAM Role for the MGMT account:
```arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest

Trust relationships
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::999988887777:root"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}

Attached IAM policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets"
],
"Resource": [
"arn:aws:route53:::hostedzone/"
]
},
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Resource": [
""
]
}
]
}

Kubernetes YAML configuration for externalDNS:
```---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      automountServiceAccountToken: true
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.7.1
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=test.local
        - --provider=aws
        - --aws-assume-role="arn:aws:iam::222233334444:role/externalDNSAccessRoute53CrossAccountTest"
        - --policy=upsert-only
        - --aws-zone-type=private
        - --registry=txt
        - --txt-owner-id="test account"
        - --log-level=debug
      securityContext:
        fsGroup: 65534
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::999988887777:role/externalDNSEKSrole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system

[aidan-melen]

yes, I have run into the exact same issue. all roles are correctly configured for cross-account trust; however, the external-dns operators fails to assume the trusting role.

[michazt]

A short update on this: we also tested it with the latest release with the same error messages.

Can anyone tell me if this is a bug or we are doing something wrong? For me it looks like a bug but maybe I am mistaken. And if it is a bug, I will of course file a bug report.

THX for any hint on this because it is preventing us from using externalDNS. Which would be very helpful if we could use it in our Setup.

[linki]

I would be interested in how ALB ingress controller handles this. We can probably do the same here.

[michazt]

@linki the ALB basically has the same IAM setup. The difference between the ALB ingress controller and externalDNS is that externalDNS needs to do a cross account access to create the DNS record. The resources the ALB creates are in the same AWS account.

From the various documentations and hints I have read externalDNS can do a cross account access or can use the OIDC provider. But I never found something where people use both. That's why I think there is a problem with the combination cross account access and OIDC provider. Or we simply do not understand how to set this up.

As AWS is promoting the OIDC provider for EKS more and more people will use this to authenticate a Kubernetes Cluster against AWS IAM. And I think it will be helpful for others to have a working example in the docs for this.

[djschnei21]

Same issue here!

[richrepko]

Same issue here as well. Have yet to find a configuration that supports cross account hosted zones.

[rust84]

I tried following https://aws.amazon.com/blogs/containers/cross-account-iam-roles-for-kubernetes-service-accounts/

1. MAIN ACCOUNT

data "aws_iam_policy_document" "externaldns-assume-role-policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::$ACCOUNT2_ID:root"]
    }
  }

  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::$ACCOUNT2_ID:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/$ID"]
    }

    condition {
      test     = "StringEquals"
      variable = "oidc.eks.us-east-2.amazonaws.com/id/$ID:aud"

      values = [
        "sts.amazonaws.com",
      ]
    }
  }
}

data "aws_iam_policy_document" "externaldns" {
  statement {
    sid = "1"

    actions = [
      "route53:ChangeResourceRecordSets"
    ]

    resources = ["arn:aws:route53:::hostedzone/*"]
  }

  statement {
    sid = "2"

    actions = [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets"
    ]

    resources = ["*"]
  }
}

1. EKS ACCOUNT

resource "aws_iam_policy" "external-dns-policy" {
  name_prefix = "external-dns"
  description = "Route53 policy for External DNS ${module.eks.cluster_id}"
  policy      = data.aws_iam_policy_document.external-dns.json
}

data "aws_iam_policy_document" "external-dns" {
  statement {
    sid = "1"

    actions = [
      "sts:AssumeRole"
    ]

    resources = ["arn:aws:iam::$MAIN_ACCOUNT_ID:role/external-dns"]
  }
}

And I get:

time="2020-07-20T16:04:10Z" level=error msg="WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your  │
│ account for https://oidc.eks.us-east-2.amazonaws.com/id/$ID\n\tstatus code: 400, request id: xxx-xxx-xxx-xxx"

[aidan-melen]

FYI related to https://github.com/kubernetes-sigs/external-dns/pull/524

[michazt]

@aidan-melen THX for the info but what does this mean for us? I see this is a commit from 2018 that was merged in 2018. So for me this would mean we are doing something wrong if the AWS cross account access really works. Then I would like to know what is not correct in the described setup above? And what would be a working example?
A side note: with the setup from above I can use the AWS CLI with "aws sts get-caller-identity" successfully. So from my perspective we have a correct AWS IAM setup and I am really lost here.

[aidan-melen]

I was going to jump into the GO code this weekend if I have time.

[michazt]

@aidan-melen ah now I see that you have committed some code for this problem. I missed your entry at the end. Thanks a lot and I hope that it will fix our problem. As I am not a programmer I am not helpful at coding but willing to test it on our setup. And I will write some documentation when it works.

[linki]

I tested it in our setup and it seems to work fine. Here's what I did.

Let's assume we want to run ExternalDNS in account A managing records in account B. We want to use the IAM role in account A arn:aws:iam::A:role/external-dns to assume the IAM role in account B arn:aws:iam::B:role/external-dns-cross-account.

Head over to account B (the target account) and create the IAM role you want to assume. In the following this target IAM role is denoted with the ARN arn:aws:iam::B:role/external-dns-cross-account.

Then go back to AWS account A and annotate ExternalDNS' ServiceAccount in cluster A with:

eks.amazonaws.com/role-arn: arn:aws:iam::A:role/external-dns

Note, this should already be the case in your setup.

Then start ExternalDNS in cluster A with:

- --aws-assume-role=arn:aws:iam::B:role/external-dns-cross-account

For the IAM role for ExternalDNS in cluster A, add the permission to assume the role in account B:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::B:role/external-dns-cross-account"
        }
    ]
}

Normally this role would have Route53 permissions to manage DNS records in account A but that wasn't needed. For the OIDC stuff to work you need to keep the Trust Relationship to the OIDC provider and the StringsEqual stuff. This should already be there and can be left unchanged.

Then go to the IAM role in account B that you created at the beginning and that you want to assume. As a reminder this is denoted as ARN arn:aws:iam::B:role/external-dns-cross-account.

Give it permission to manage DNS records just like the IAM role for ExternalDNS in account A normally has, e.g.:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "route53:*",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Then add a Trust Relation to the IAM role in account A with:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::A:role/external-dns"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Note that I removed the Trust Relationship to The identity provider(s) ec2.amazonaws.com as well. It works either way and I guess it's better to just keep it then.

Note, the IAM role in account B can be a "plain" IAM role. It must have permissions to modify DNS records as well as a Trust Relationship back to account A. If you want to assume another ServiceAccount's IAM role, such as ExternalDNS' role in a different account, then the role will also have the Trust Relationship to the OIDC provider. Both cases worked fine for me.

In my setup the target account has its own Kubernetes cluster with its own OIDC provider. I'm certain that this doesn't interfere with the experiment but who knows. Unfortunately, I can't delete it at the moment without causing some disruption.

To summarize you basically need to:

• Make sure that IAM role A works fine with your OIDC setup within account A
• add a permission to IAM role A to allow to assume IAM role B
• add a trust relationship on IAM role B stating IAM role A
• configure ExternalDNS' ServiceAccount to use IAM role A
• start ExternalDNS and tell it to --aws-assume-role IAM role B

This was tested with v0.7.1 and v0.7.2.

Let me know if that helped you or if you still get stuck.

[rust84]

Thanks @linki @aidan-melen the --aws-assume-role parameter was the piece I was missing. For anyone else who is using the Bitnami helm chart this can be set with .Values.aws.assumeRoleArn.

Annotated the service account with the role on account A which grants assume role permission for the account B role which is granting route53 access.
Set --aws-assume-role to the role on account B.

The trust policy on account B is set to the root user on account A. I didn't have to use the Federated StringsEqual part, which might not be optimal. I could probably change that over once all clusters are using OIDC but some of them aren't yet I'm leaving it as is for now.

    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::$ACCOUNT_ID:root"]
    }

It's working perfectly. Thanks very much.

[michazt]

@linki THX for this example. I will try this next week and let you know if it succeeded. This week I am busy with other stuff. I think that I am also missing the --aws-assume-role=arn:aws:iam::B:role/external-dns-cross-account start parameter. Sounds promising.

[djschnei21]

@rust84 or @linki any chance one of you can paste your working deployment yaml here??? We're definitely missing something simple... still can't get our pod which is using an OIDC provider to assume a cross account role.

[ericrdgz]

@linki we are facing similar issues. Trying to follow your guidance. How are you creating your IAM Roles. CLI, Terraform, or Console? We are using Terraform. Also are you on EKS non-fargate or EKS Fargate?

[hetpats]

IS there a way to pass External-Id for assuming the role in cross account?

[michazt]

So far we had no time to test this. Hopefully a colleague of mine will have time soon to test it.

[mgrecar]

I followed these directions to set up an identity provider and attach it to the role (following pretty much the same steps for the policy and role in the docs), and I was able to get cross-account ExternalDNS working for me. The ExternalDNS service account gets deployed with the annotation pointing to that role and it just works for us. Just needed to rollout the deployment again after making the update.

[yves-vogl]

To me it looks like the author of this issue did set up the trust relationship incorrectly.
When using OIDC with an annotated service account (which linkes to an IAM role) a pod runs under a temporarly session which impersonates the role as its identity. Therefore this particular role should be trusted.

Also the error messages points out that the initial role has no permission to assume the cross-account role (which is required beside the trust relationship on the target role).

[yves-vogl]

What @linki says. That's how I do this (and did yesterday the last time) and how it's supposed to work.

[chapitos]

To add my 5 cents here.

I have done exactly the same steps as @linki described and run into the problem anyway (access denied, cannot assume role, as above). The cause was that I have provided the role ARN to be assumed between quotation marks (e.g. --aws-assume-role="arn:aws:iam::123456789012:role/role-to-be-assumed"). So basically, there is an error in args in the comment above, which cause the problem @michazt reported.

After removing the quotation masks, everything works fine.

[michazt]

I closed this issue because we resolved it somehow. We could not resolve the original problem described by me here but we did the latest approach recommended by AWS described here: https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/
With this starting new every things works perfectly with externalDNS doing a cross account Route53 access. Unfortunately we could not find out what was the problem in our other setup but my guess is that we did something wrong in the trust relationship setup and it has nothing to do with externalDNS.