Terraform: It's easy to hit Github's rate-limit when getting modules from the Registry in a shared CI

[navaati]

Terraform Version

Terraform v0.12.24

Terraform Configuration Files

module "asg" {
  source  = "terraform-aws-modules/autoscaling/aws"
  version = "~> 3.0"
[...]
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
[...]
}

# and several others, 7 different modules sources in total (some of them invoked more than once)

Debug Output

https://gist.github.com/navaati/990eeadbd5b1eb6051cf98aa3eeed841

Crash Output

N/A

Expected Behavior

I would at least expect Terraform to clearly say that it's a rate-limit error with Github's API, to save some time on the debugging.

Then, there is a variety of things that Terraform could do to alleviate the issue, such as using an app Oauth token to move the limit from 60 requests per hour to 5000, or providing a mechanism to cache modules in a safe way (I haven't been able to find information about that in the documentation).

Another way could be to have the Github API calls done regularly by registry.terraform.io, the result stored, and then the Terraform CLI calls registry.terraform.io instead of Github. But that becomes a major architecture decision then…

I don't really know, that's why I'm opening this issue :).

Actual Behavior

Error: Failed to download module
Could not download module "alb" (alb.tf:1) source code from
"https://api.github.com/repos/terraform-aws-modules/terraform-aws-alb/tarball/v5.1.0//*?archive=tar.gz":
bad response code: 403.

I'm sure that it's a Github rate-limit issue because I ran the command curl -v https://api.github.com/rate_limit from the same context and received a response containing:

{
  "resources": {
    "core": {
      "limit": 60,
      "remaining": 0,
      "reset": 1585047599
    },
    [...]
}

Steps to Reproduce

$ terraform init -backend=false

Additional Context

I'm running Terraform from a CI cluster shared by a number of teams in my company, with all traffic going out on the same IP.

References

There is talks of the Github rate-limit in terraform-providers/terraform-provider-github#5 but it's about the Github provider, not the Registry mechanism in core.

Thank you Terraform folks !

[oscarwest]

Even having 0 remaining on the rate limit gives 403 for github-backed modules on the terraform registry in 0.12.24. Running terraform get -update with 0.12.18 does not give this problem. Maybe this is a bug introduced here? https://github.com/hashicorp/terraform/pull/24260

[oscarwest]

As a workaround you can bypass the terraform registry
e.g.

module "some_module" {
  source = "github.com/terraform-aws-modules/terraform-aws-vpc?ref=v2.31.0"
}

[bogarcia]

Same problem with Terraform v0.12.19, confirmed it happens just when reach rate_limit:
"resources": { "core": { "limit": 60, "remaining": 0, "reset": 1585136114 },

[rahulk94]

As a workaround you can bypass the terraform registry
e.g.

module "some_module" {
  source = "github.com/terraform-aws-modules/terraform-aws-vpc?ref=v2.31.0"
}

@oscarwest thanks mate. You also need to remove version but besides that got my CI working again with:

  # source  = "terraform-aws-modules/acm/aws"
  # version = "~> v2.0"
  source = "github.com/terraform-aws-modules/terraform-aws-acm?ref=v2.5.0"

[justincampbell]

Thanks for the report! We've updated the Terraform Registry so that the URLs returned for module downloads do not get rate-limited by GitHub as easily. Please let me know if this fixes the issue for you, or if any new issues arise from it.

@oscarwest I don't believe the PR linked has any relation to the rate-limiting.

Thanks everyone!

[navaati]

Wait wait wait wait, you mean that it's a server-side change and I don't need to wait for a release, and then for my company to adopt it, and then… yada yada, and my problem is resolved right now ?

/me hammering the stuff and seeing it still works, no rate-limiting

Wow :tada: ! I wouldn't have dared to hope for that !

Thank you :heart: !

[bogarcia]

@justincampbell is possible that your change had been reverted?
I'm starting today to see same problems we had until you did your magic.

[justincampbell]

@bogarcia Could you paste an error message or the relevant ERROR log lines when running TF_LOG=1 terraform init? I'm not able to reproduce the issue we were experiencing last week with Terraform 0.12.24.

[bogarcia]

I'm not able to reproduce it again, as it seems now I have slots into Github API.

 "resources": {
    "core": {
      "limit": 60,
      "remaining": 31,
      "reset": 1585598256
    },

It was zero when failing. So, do it still use API calls? I guess yes., I am in a shared (company Jenkins) environment, so it's not easy to get isolated. Thanks anyway.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.