Installer: error simulating policy: Throttling: Rate exceeded

Created on 15 Mar 2019 · 11Comments · Source: openshift/installer

Version

0.14.0

Platform:

aws

What happened?

When deploying the following error happens:

dev-sandbox-ocp4-0124.log:fatal: [clientvm.0124.internal]: FAILED! => {"changed": true, "cmd": ["openshift-install", "create", "cluster", "--dir=/home/ec2-user/cluster-0124"], "delta": "0:00:12.892995", "end": "2019-03-14 16:36:19.002163", "msg": "non-zero return code", "rc": 1, "start": "2019-03-14 16:36:06.109168", "stderr": "level=fatal msg=\"failed to fetch Terraform Variables: failed to fetch dependency of \\\"Terraform Variables\\\": failed to fetch dependency of \\\"Bootstrap Ignition Config\\\": failed to fetch dependency of \\\"Master Machines\\\": failed to generate asset \\\"Platform Credentials Check\\\": validate AWS credentials: checking install permissions: error simulating policy: Throttling: Rate exceeded\\n\\tstatus code: 400, request id: 4b3be60f-4677-11e9-b604-4fdfe40cdda9\"", "stderr_lines": ["level=fatal msg=\"failed to fetch Terraform Variables: failed to fetch dependency of \\\"Terraform Variables\\\": failed to fetch dependency of \\\"Bootstrap Ignition Config\\\": failed to fetch dependency of \\\"Master Machines\\\": failed to generate asset \\\"Platform Credentials Check\\\": validate AWS credentials: checking install permissions: error simulating policy: Throttling: Rate exceeded\\n\\tstatus code: 400, request id: 4b3be60f-4677-11e9-b604-4fdfe40cdda9\""], "stdout": "", "stdout_lines": []}
Extract from .openshift_install.log:

time="2019-03-14T16:39:19Z" level=debug msg="        Fetching \"Certificate (admin-kubeconfig-signer)\"..."                                                                                   
time="2019-03-14T16:39:19Z" level=debug msg="        Generating \"Certificate (admin-kubeconfig-signer)\"..."                                                                                 
time="2019-03-14T16:39:20Z" level=debug msg="      Generating \"Certificate (admin-kubeconfig-client)\"..."                                                                                   
time="2019-03-14T16:39:20Z" level=debug msg="      Fetching \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."                                                                    
time="2019-03-14T16:39:20Z" level=debug msg="        Fetching \"Certificate (kube-ca)\"..."
time="2019-03-14T16:39:20Z" level=debug msg="        Generating \"Certificate (kube-ca)\"..."
time="2019-03-14T16:39:20Z" level=debug msg="        Fetching \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."                                                                        
time="2019-03-14T16:39:20Z" level=debug msg="          Fetching \"Certificate (kube-apiserver-localhost-signer)\"..."                                                                         
time="2019-03-14T16:39:20Z" level=debug msg="          Generating \"Certificate (kube-apiserver-localhost-signer)\"..."                                                                       
time="2019-03-14T16:39:21Z" level=debug msg="        Generating \"Certificate (kube-apiserver-localhost-ca-bundle)\"..."                                                                      
time="2019-03-14T16:39:21Z" level=debug msg="        Fetching \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."                                                                  
time="2019-03-14T16:39:21Z" level=debug msg="          Fetching \"Certificate (kube-apiserver-service-network-signer)\"..."                                                                   
time="2019-03-14T16:39:21Z" level=debug msg="          Generating \"Certificate (kube-apiserver-service-network-signer)\"..."                                                                 
time="2019-03-14T16:39:21Z" level=debug msg="        Generating \"Certificate (kube-apiserver-service-network-ca-bundle)\"..."                                                                
time="2019-03-14T16:39:21Z" level=debug msg="        Fetching \"Certificate (kube-apiserver-lb-ca-bundle)\"..."                                                                               
time="2019-03-14T16:39:21Z" level=debug msg="          Fetching \"Certificate (kube-apiserver-lb-signer)\"..."                                                                                
time="2019-03-14T16:39:21Z" level=debug msg="          Generating \"Certificate (kube-apiserver-lb-signer)\"..."                                                                              
time="2019-03-14T16:39:22Z" level=debug msg="        Generating \"Certificate (kube-apiserver-lb-ca-bundle)\"..."                                                                             
time="2019-03-14T16:39:22Z" level=debug msg="      Generating \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."                                                                  
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Install Config\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Install Config\""                                                                                             
time="2019-03-14T16:39:22Z" level=debug msg="    Generating \"Kubeconfig Admin Client\"..."
time="2019-03-14T16:39:22Z" level=debug msg="    Fetching \"Kubeconfig Kubelet\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Certificate (kube-ca)\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Certificate (kube-ca)\""                                                                                      
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."                                   
time="2019-03-14T16:39:22Z" level=debug msg="        Fetching \"Certificate (kube-ca)\"..."
time="2019-03-14T16:39:22Z" level=debug msg="        Reusing previously-fetched \"Certificate (kube-ca)\""                                                                                    
time="2019-03-14T16:39:22Z" level=debug msg="      Generating \"Certificate (system:serviceaccount:openshift-machine-config-operator:node-bootstrapper)\"..."                                 
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Install Config\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Install Config\""                                                                                             
time="2019-03-14T16:39:22Z" level=debug msg="    Generating \"Kubeconfig Kubelet\"..."
time="2019-03-14T16:39:22Z" level=debug msg="    Fetching \"Kubeconfig Kubelet Client\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Certificate (kube-apiserver-complete-server-ca-bundle)\"..."                                                                    
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Certificate (kube-apiserver-complete-server-ca-bundle)\""                                                     
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Certificate (kubelet-client)\"..."                                                                                              
time="2019-03-14T16:39:22Z" level=debug msg="        Fetching \"Certificate (kubelet-bootstrap-kubeconfig-signer)\"..."                                                                       
time="2019-03-14T16:39:22Z" level=debug msg="        Generating \"Certificate (kubelet-bootstrap-kubeconfig-signer)\"..."                                                                     
time="2019-03-14T16:39:22Z" level=debug msg="      Generating \"Certificate (kubelet-client)\"..."                                                                                            
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Install Config\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Install Config\""                                                                                             
time="2019-03-14T16:39:22Z" level=debug msg="    Generating \"Kubeconfig Kubelet Client\"..."
time="2019-03-14T16:39:22Z" level=debug msg="    Fetching \"Master Machines\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Cluster ID\"..."
time="2019-03-14T16:39:22Z" level=debug msg="      Reusing previously-fetched \"Cluster ID\""
time="2019-03-14T16:39:22Z" level=debug msg="      Fetching \"Platform Credentials Check\"..."
time="2019-03-14T16:39:22Z" level=debug msg="        Fetching \"Install Config\"..."
time="2019-03-14T16:39:22Z" level=debug msg="        Reusing previously-fetched \"Install Config\""                                                                                           
time="2019-03-14T16:39:22Z" level=debug msg="      Generating \"Platform Credentials Check\"..." 
Shared connection to clientvm.4bc9.sandbox651.opentlc.com closed.aform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Cono fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: mint credentials check: error simulating policy: Throttling: Rate exceeded\n\tstatus code: 400, request id: c08b55a2-4677-11e9-a2ca-953eb853e303"

What you expected to happen?

I expect the installer to implement the exponential backoff with a higher number of retries.

How to reproduce it (as minimally and precisely as possible)?

Not 100% reproducible. It depends on the number of IAM API calls in your account.

AWS support says it's SimulatePrincipalPolicy that has been throttled.

platforaws

Source

fridim

Most helpful comment

One year later, we still hit this issue sometimes.

@abhinavdahiya you closed the issue but i think you're missing the point.

@wking was saying there is exponential backoff but with max retries = 3 so it stays low.

A variable to defined that max_retries would be useful. And it could still default to 3.

Currently the workaround we have is to retry the whole installation, which is even more expensive on the APIs.

fridim on 22 Apr 2020

👍2 🚀1 🎉1

All 11 comments

I expect the installer to implement the exponential backoff with a higher number of retries. Not 100% reproducible. It depends on the number of IAM API calls in your account.

AWS support says it's SimulatePrincipalPolicy that has been throttled.

Did they say that this is specifically from crossing a SimulatePrincipalPolicy rate threshold? Or an IAM-request threshold? My understanding was that throttling buckets depended on whether resources were being queried or created, but that a single action bucket was generally shared across several resources. See previous discussion in #1352.

I expect the installer to implement the exponential backoff with a higher number of retries.

We use AWS's stock SDK for this, which means we get exponential backoff with 3 retries (although maybe there's an IAM-specific retry override that I'm missing). We could certainly bump our number of retries, or even set it to some unimaginably-huge number, but I'm cautious about being more opinionated about this sort of thing than the SDK authors. It looks like their default dates back to 2015, and I don't see the value discussed in the commit message there, but that's an awfully big commit, so it would be hard for them to explicitly motivate all its choices. The drawback to bumping maxRetries is that the deeper clients survive into their exponential backoff, the noisier the account is going to be. So I'd expect more clusters successfully created in a busy account, but all of those clusters to take longer to come up (during busy times). Without doing some experiments, it's hard to know how strongly either metric would be affected.

I don't expect this sort of thing to happen for accounts that aren't used for testing cluster creation. Accounts with long-running clusters that do not emphasize cluster creation should see much lower AWS API consumption.

wking on 15 Mar 2019

Did they say that this is specifically from crossing a SimulatePrincipalPolicy rate threshold?

Yes.

We use AWS's stock SDK for this, which means we get exponential backoff with 3 retries (although maybe there's an IAM-specific retry override that I'm missing). We could certainly bump our number of retries, or even set it to some unimaginably-huge number, but I'm cautious about being more opinionated about this sort of thing than the SDK authors.

As a cloud provider, AWS has different requirements than us.

The drawback to bumping maxRetries is that the deeper clients survive into their exponential backoff, the noisier the account is going to be.

It generates indeed more traffic to APIs but not that much: the deeper clients survive, the more exponential they get. As it is exponential what it does really is to flatten when there are peaks. With retries=3 we don't even get there.
If you have a busy account, you will have to implement the exponential back-off with a high retry number for your infra and apps anyway.

So I'd expect more clusters successfully created in a busy account, but all of those clusters to take longer to come up (during busy times).

Correct. "longer" could also be 10, 20 seconds here, instead of failing. Even if it adds 2-5 minutes, considering it's a ~30-40 minutes deployments, it looks acceptable.

Without doing some experiments, it's hard to know how strongly either metric would be affected.

We deployed 200 clusters for load testing before an event. We are using sand-boxed accounts to avoid rate-limit and service limit issues. Each cluster has its own account, otherwise OCP4 generates too many API calls for a single account and it's currently not even possible to deploy that much.
But even with dedicated accounts per deployment we got this rate-limit error.
(We worked around it by throttling the deployments.)

But more generally, if we don't allow the default value of 3 to be changed, it means you cannot deploy OCP in some busy aws accounts or during a peak.

If we cannot easily guess what is the right value (i think 3 is very low still), at least it could be something we can pass to the installer ? A global api_max_retries to generally allow retries on all APIs would be very nice to have.
The throttling error could also be caught by the installer with an error message like "You can try to increase the api_max_retries value".

fridim on 15 Mar 2019

I also saw a throttling error on a PR overnight:

Installing from release registry.svc.ci.openshift.org/ci-op-fhbk2rdh/release@sha256:ebbccc903228a9164953715cfc6165dba458e93f4e1239cc821679c523de60f9
level=fatal msg="failed to fetch Terraform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Config\": failed to fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: mint credentials check: error simulating policy: Throttling: Rate exceeded\n\tstatus code: 400, request id: fb0639d5-46fd-11e9-905f-b5bfca14856b

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/529/pull-ci-openshift-machine-config-operator-master-e2e-aws/2541/

kikisdeliveryservice on 15 Mar 2019

the installer uses aws-sdk-go and it has exp backoff already.

/close

abhinavdahiya on 25 Jul 2019

@abhinavdahiya: Closing this issue.

In response to this:

the installer uses aws-sdk-go and it has exp backoff already.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot on 25 Jul 2019