Kops: `kops create cluster` fails with NoCredentialProviders

[exarkun]

Following the getting started guide at http://kubernetes.io/docs/getting-started-guides/kops/ I encountered a fatal error on step 4:

$ kops -v10 --logtostderr create cluster --zones us-east-1a --name useast1.staging.$DOMAIN
I1109 10:04:00.643781    1690 s3context.go:61] Querying S3 for bucket location for "kops-testing.dev.$DOMAIN"

error reading cluster configuration "useast1.staging.$DOMAIN": error reading configuration file s3://kops-testing.dev.$DOMAIN/useast1.staging.$DOMAIN/config: error getting location for S3 bucket "kops-testing.dev.$DOMAIN": NoCredentialProviders: no valid providers in chain. Deprecated. 
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

My AWS credentials are available and - so far as I can tell - properly configured. kubectl is happy with them. Also the AWS CLI:

$ grep $AWS_PROFILE ~/.aws/credentials | wc -l
1
$ aws s3 ls s3://kops-testing.dev.$DOMAIN && echo $?
0

[exarkun]

Putting the credentials from $AWS_PROFILE in ~/.aws/credentials into AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables allows kops to operate properly. So there seems to be a problem with loading credentials from a specified profile in ~/.aws/credentials.

[kris-nova]

Just curious - what do the permissions/ownership on the file look like?

[exarkun]

~/.aws is a symlink to another directory. The directory is 0700. The credentials file inside the directory is 0600.

[kris-nova]

This is a known issue with the go filepath.Walk not following symlinks https://github.com/golang/go/issues/17451

Wondering what happens when you remove the symlink ;)

[exarkun]

Good guess. I moved the symlink aside and copied its target to ~/.aws. In that configuration, kops get cluster succeeds instead of failing with a credentials error.

[kris-nova]

Resolved

[exarkun]

Wait, what was the resolution?

[exarkun]

@kris-nova ping

Sorry, could you help me understand this please? It seems like the implied solution is "don't have ~/.aws as a symlink" but that's not exactly a solution. I don't keep my AWS credentials on every system I want to interact with AWS from. I keep them on removable storage and symlink ~/.aws to the right place.

[kris-nova]

@exarkun

Thanks the ping, and I'm on my phone so bare with me :)

So basically what I was saying is that

The official AWS SDK for Golang uses filepath.Walk and getting the SDK to respect a symlink would fall under the scope of either changing the AWS SDK, or changing Go standard library. Both of which are beyond the scope of the Kops repository unfortunately.

I recommend reading up on AWS best practices for managing security credentials.. Maybe you could export your configuration as environmental variables instead of a link on the file system? More information.

I would particularly note what their supported avenues of configuration are, and the order they are parsed.

Your welcome to open an issue in Kops regarding the parsing of symlinks for AWS config, and in theory we could hack the SDK to support this. Albeit that would be a poor design choice in my mind.

Does this help clarify why the issue was closed?

[exarkun]

The official AWS SDK for Golang uses filepath.Walk and getting the SDK to respect a symlink would fall under the scope of either changing the AWS SDK, or changing Go standard library. Both of which are beyond the scope of the Kops repository unfortunately.

Ah, ok. That was one of the explanations I was expecting. So if the AWS SDK is fixed to follow symlinks then kops will just inherit this fix, right? Or maybe someone will have to update the AWS SDK vendored in kops?

I recommend reading up on AWS best practices for managing security credentials.. Maybe you could export your configuration as environmental variables instead of a link on the file system?

I thought I had figured out best practice. :) I'm avoiding putting the credentials in environment variables because that exposes the secrets much more widely than selecting a profile from a credentials file via the AWS_PROFILE variable.

It looks like something based on temporary credentials and session tokens might be an acceptable replacement... I can use aws sts get-session-token to generate short-lived credentials and put those in the environment, instead. Temporary credentials aren't as a valuable as permanent credentials ... but they're still pretty valuable (they could still be used to compromise services running on the Kubernetes cluster I'm managing, for example) ...

I don't typically operate on a shared system but I'd still rather not adopt a practice that allows someone who somehow gains local non-privileged access to one of my systems leverage that into even partial control of my AWS accounts.

So I suspect getting the AWS SDK fixed is a better option. Since the official AWS CLI supports a symlink here, it seems reasonable for the Golang AWS SDK to support it as well.

Looking at Kops' _vendor directory, it seems that the AWS SDK used is the official library from Amazon - https://github.com/aws/aws-sdk-go - so if I wanted to convince someone to fix the symlink handling, it would be that project, right?

Thanks very much for the further clarification and the additional help!

[aliakhtar]

Does kubectl respect the aws cli profile, or is this issue present there too?

Its OK to export the env variables once for kops, as cluster creation / update is rare.

But having to do it again and again for k8s would be a pain.

[exarkun]

Its OK to export the env variables once for kops, as cluster creation / update is rare.

I disagree with this but it's basically moot at this point.

[kvz]

In my case, it wasn't caused by a symlink but too open permissions 😱 like @kris-nova suggested in https://github.com/kubernetes/kops/issues/855#issuecomment-259449332 - just thought I'd share for those stumbling upon this.

[bsmedberg-xometry]

For the record, I just experienced this issue and I resolved it by exporting AWS_SDK_LOAD_CONFIG=true in my environment.