External-dns: ExternalDNS + kube2iam integration problem with default role

[placydo]

Hi,
First of all thanks for Your amazing work!
We would like to run ExternalDNS with kube2iam with --default-role enabled. While it works perfectly while the default role is disabled we encountered issue when we enable it. For some reason it looks like there were wrong role assigned to pod - default instead of the dedicated one. It sometimes resolve to correct role after 20+ minutes.

Log:

time="2017-07-27T07:05:50Z" level=info msg="Connected to cluster at https://10.3.0.1:443"
time="2017-07-27T07:06:25Z" level=error msg="AccessDenied: User: arn:aws:sts::VERY_SECRET_ID:assumed-role/VERY_SECRET_ROLE-assume-def/185b3bd4-VERY_SECRET_ROLE-assume-def is not authorized to perform: route53:ListHostedZones
status code: 403, request id: 1adf6f00-729a-11e7-9cdb-9195f1e9a104"

There is corresponding issue on kube2iam github: https://github.com/jtblin/kube2iam/issues/80

Tried update kube2iam to newest version 0.6.4 - it did not resolve the issue.
Currently using registry.opensource.zalan.do/teapot/external-dns:v0.4.0

[mikkeloscar]

This seem to be a purely kube2iam issue or am I misunderstanding something?

Doesn't sound like it can be resolved in external-dns if the pod is getting the wrong IAM role.

[linki]

@mikkeloscar I told @placydo to open an issue here so that we can keep track of it. It does look like kube2iam though.

[jrnt30]

@mikkeloscar We had a series of issues with Kube2iam which is a caching issue that can rear it's head more readily (for us at least) when using cron jobs.

The next time you have, if you do a kubectl get pods -a check to see if the IP of a "Completed" pod is that of the external-dns pod (or whatever is failing). I have a PR https://github.com/jtblin/kube2iam/pull/92 to address this if you want to try running it, we have been using this for a few weeks and have not experienced this issue again yet.

[placydo]

@linki Thanks to @jrnt30 we do not notice that issue anymore. It was purely Kube2IAM problem and now it is solved. Thanks for your awesome work!

[linki]

Thanks @placydo for reporting the issue and thank you @jrnt30 for fixing kube2iam.