Kops: AWS Gov Coud Unable to list AWS regions

[nniehoff]

When running kops create cluster in gov cloud I get:

error reading cluster configuration "kube.testing.com": error reading s3://kubernetes.state.store/kube.testing.com/config: Unable to list AWS regions: AuthFailure: AWS was not able to validate the provided access credentials

I have verified I provided the correct AWS credentials and I can list the s3 buckets using the aws cli with the same credentials. I am running the following command to create the cluster:

kops create cluster --zones=us-gov-west-1a,us-gov-west-1b --admin-access="x.x.x.x/24" --cloud=aws --image=ami-xxxxxxxx --master-zones=us-gov-west-1a,us-gov-west-1b --node-count=4 --vpc=vpc-xxxxxxxx --state=s3://kubernetes.state.store --network-cidr="y.y.y.y/24" --name=kube.testing.com -v 5 --logtostderr

From what I can gather it doesn't look like kops is able to validate us-gov-west-1 as a valid region.

Unfortunately I'm really new to kubernetes, kops, and aws so hopefully I'm not doing something stupid.

[TKCen]

I'm getting the same error for any kops operation which wants to retrieve the state of a running cluster:

$ kops delete cluster kops.cluster.com --yes                                                                     

error reading cluster configuration: error reading cluster configuration "kops.cluster.com": error reading s3://kops/kops.cluster.com.
io/config: Unable to list AWS regions: AuthFailure: AWS was not able to validate the provided access credentials                                
        status code: 401, request id: xxx

[geojaz]

@nniehoff
I'd love to try to get this fixed up for you, but as I learned today, I don't have access the government regions, so it makes it quite difficult for me to test.

Have you tried setting the zones like --zones="us-gov-west-1a,us-gove-west-1b"
I tried your original syntax and got a weird error, but when I used syntax like above, I got error populating configuration: Region is not a recognized EC2 region: "us-gov-west-1" (check you have specified valid zones?) back. If you take a look at https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/awsup/aws_utils.go#L36-L70 you can see how that error is generated. It looks like it uses my AWS credentials to list the regions available to me and I don't have us-gov-west-1a/b, so it pushes back. You may be more successful.

If this doesn't help you out, ping me on #kops slack @erichole

[nniehoff]

@geojaz
Thanks for the help. It's been a few days since I looked at this and having now played with kargo and others, which has been good to help me learn AWS and GovCloud, but I don't remember exactly where I was with kops. Based on your code snippet you sent I realized I needed to set the env variable AWS_REGION="us-gov-west-1". Now I am getting further. Unfortunately AWS GovCloud is like AWS but AWS minus about 75% of the features. One of which is route53. So now I'm getting:

error determining default DNS zone: error querying zones: MissingEndpoint: 'Endpoint' configuration is required for this service

I'm pretty sure this is related to the lack of route53 in GovCloud so now I need to go read more on private DNS within GovCloud.

If you've never seen GovCloud it looks like AWS looked in the beggining. I have EC2, S3, VPC, CloudFormation, RDS, and that's about it.

[nniehoff]

Any pointers on how to fix the DNS issue above if I don't have Route53?

[pandeybk]

@nniehoff I was seeing this error when I tried to execute command from ec2 instance which contains iam roles. Some reason kops only trying to use iam roles attached to instance instead of using aws configured credentials.

To fix this I assign following policy to instance role itself.

AmazonRoute53FullAccess
AmazonS3FullAccess
IAMFullAccess
AmazonVPCFullAccess

You can also export variable like this,

export AWS_ACCESS_KEY_ID=<access key>
export AWS_SECRET_ACCESS_KEY=<secret key>

You can find reference here,

https://github.com/kubernetes/kops/blob/master/docs/aws.md

[chrislovecnm]

/assign

Need to check this in iam roles

[ctownsen357]

I would be glad to help sort this out. I'm running a Docker Swarm cluster out on GovCloud right now but would like to migrate to Kubernetes eventually for some of the additional features Kubernetes provides. Currently, requiring Route53 appears to be a show stopper. I can help test running commands on GovCloud and have external DNS through hover.com. I can't find any documentation on bypassing Route53 but saw it mentioned as a possibility on another thread.

[ctownsen357]

I got much further with this after reading @geojaz 's comment and following the documentation to specify .k8s.local for the suffix of the cluster name. The create cluster command started creating resources but can't find the target AMI out on GovCloud. I'm getting this error now:

W0104 21:00:52.165942   63474 executor.go:109] error running task "LaunchConfiguration/master-us-gov-west-1a.masters.cfs-kube-test.k8s.local" (5m47s remaining to succeed): could not find Image for "kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02"

So close! 72/75 tasks reported as complete.

[chrislovecnm]

@ctownsen357 you copy the image over to gov? Or you can use another ami

[ctownsen357]

Thank you for the follow up. I've switched over to kube-aws and that has been a little easier for me to make progress with.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[weisjohn]

setting export AWS_REGION="us-gov-west-1" before doing the kops create cluster worked for me, but then when applying via kops update cluster [...] --yes, kops wasn't able to complete:

I0702 16:01:17.383569   40128 executor.go:91] Tasks: 54 done / 77 total; 19 can run
W0702 16:01:19.302566   40128 executor.go:118] error running task "LaunchConfiguration/master-us-gov-west-1a.masters.test-foo-tech.k8s.local" (9m58s remaining to succeed): could not find Image for "kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11"
W0702 16:01:19.302639   40128 executor.go:118] error running task "LaunchConfiguration/nodes.test-foo-tech.k8s.local" (9m58s remaining to succeed): could not find Image for "kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11"
I0702 16:01:19.302680   40128 executor.go:91] Tasks: 71 done / 77 total; 3 can run

[immesys]

I'm having the same issue but with the latest version of kops: the images don't exist on govcloud

[immesys]

BTW this part is easy to fix by patching kops with the new region if you need us-gov-east-1 (<10 lines changed) and manually copying across the AMI. Afterwards though, you will hit the problems I mentioned in #6325. I mention a solution there too