External-dns: Pod setup error when using AWS: "CredentialRequiresARNError"

[phobos-dthorga]

Apologies if this has been posted already, but I couldn't find much on this particular error condition, if anything. But I'm having such a difficult time setting up external-dns via Rancher within Kubernetes and I've tried pretty much everything I know at this point.

As you can see below [ 1, 2 ], I keep getting this error with my external-dns pods when trying to setup a configuration to make use of AWS Route 53.

[ 1 ] - https://paste.gekkofyre.io/view/8e550193
[ 2 ] - https://imgur.com/a/5NYkR5R

You can also find my configuration at the link just below, with obvious private information omitted for security purposes.

[ 3 ] - https://paste.gekkofyre.io/view/5dbf5822#k1Qz8yTVKFOJL1s6y16uONn9l2fyikMh

Please note that we're making use of Rancher v2.3.2 to orchestrate our Kubernetes cluster, which consists of three nodes currently, plus the Rancher controller. There is plenty of RAM, vCPUs, and storage to go around so I can't see any of that being an issue.

Our Kubernetes version is v1.16.2-00 across all nodes and Docker itself is v5:18.09.9~3-0 as well. This is all running on the latest updated version of Ubuntu 18.04 LTS, which again is the same for all nodes and the Rancher controller itself. If anyone can offer assistance then that would be dearly appreciated, thank you.

--
https://gekkofyre.io/
GekkoFyre Networks

[frittenlab]

I am seeing exactly the same problem. This worked last week without a glitch. I am installing external dns via the helm chart. After downgrading the helm chart to the image 0.5.16-debian-9-r8 everything works again. Before I was also seeing this error:

time="2019-11-08T16:23:39Z" level=fatal msg="CredentialRequiresARNError: credential type source_profile requires role_arn, profile default"

[cliedeman]

Ran into the same issue.

@frittenlab the downgrade worked for me too thanks.

Its likely on of these PR's
https://github.com/kubernetes-sigs/external-dns/pull/1182
https://github.com/kubernetes-sigs/external-dns/pull/1185
https://github.com/kubernetes-sigs/external-dns/pull/1172

Will do a bisect if I get time

[rivernews]

Ran into same issue when installing external-dns chart by helm. Looking at external-dns pod log gives msg="CredentialRequiresARNError: credential type source_profile requires role_arn, profile default"

Can confirm version by @ffledgling worked for me too. Kubernetes version is 1.14.8. For those using terraform you can lock the version to chart v2.6.1 which is the latest version using app version 0.5.16:

resource "helm_release" "project-external-dns" {
  name      = "external-dns"
  chart     = "stable/external-dns"
  version   = "v2.6.1"
  ...
}

[prageethw]

can confirm this appears after helm version 2.10.1.

[acim]

Same here, works with Helm chart 2.10.1 and not with any later. Latest EKS version at the moment.

[Freyert]

Looks like the S3 terraform resource had a similar issue and HashiCorp had to patch their aws-sdk wrapper to grab credentials differently.

Issue with helpful details: https://github.com/hashicorp/aws-sdk-go-base/issues/4

https://github.com/hashicorp/aws-sdk-go-base/pull/5

Ticket was reopened later: https://github.com/hashicorp/terraform/issues/22732

---

External DNS uses a command line flag to assume the role:

https://github.com/helm/charts/blob/master/stable/external-dns/templates/deployment.yaml#L101-L103

Where the flag is consumed and the role should be assumed: https://github.com/kubernetes-sigs/external-dns/blob/f763d2a4139746abd775c61642cb9e776b387ba6/provider/aws_sd.go#L104-L107

---

Something worth trying:

Add the role ARN to the AWS config :)

https://github.com/helm/charts/blob/master/stable/external-dns/templates/_helpers.tpl#L113-L117

---

Lol OK so my deployment isn't even trying to assume a role.

---

OK, so there's a second block where it tries to do this which definitely is broken:

https://github.com/helm/charts/blob/master/stable/external-dns/templates/_helpers.tpl#L119-L123

For some reason it doesn't evaluate to the first block which would work. This block tries to use source_profile without a role_arn.

OK, so here's how it happens:

Providing aws.region causes an AWS config to be attached with a [profile default] whose source_profile is default. I think what they're trying to do is refer to the credentials in the credentials file. If you omit aws.region the config won't be attached.

[Freyert]

Fix delivered to stable chart. Please confirm if the newest version 2.13.2 works for you.

[tomerleib]

2.13.2 works!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.