terraform-aws-modules/rds fails to download

[skyzyx]

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

module "rds" {
  source                = "terraform-aws-modules/rds/aws"
  version               = "1.19.0"
  identifier            = "${var.tags["Application"]}-${var.tags["Environment"]}"
  engine                = "${data.aws_ssm_parameter.rds_engine.value}"
  engine_version        = "${data.aws_ssm_parameter.rds_engine_version.value}"
  instance_class        = "${data.aws_ssm_parameter.rds_instance_class.value}"
  allocated_storage     = "${data.aws_ssm_parameter.rds_allocated_storage.value}"
  storage_encrypted     = "${data.aws_ssm_parameter.rds_storage_encrypted.value}"
  copy_tags_to_snapshot = "${data.aws_ssm_parameter.rds_copy_tags_to_snapshot.value}"
  multi_az              = "${data.aws_ssm_parameter.rds_multi_az.value}"

  #monitoring_interval             = "${data.aws_ssm_parameter.rds_monitoring_interval.value}"
  name                      = "${data.aws_ssm_parameter.rds_name.value}"
  username                  = "${data.aws_ssm_parameter.rds_username.value}"
  password                  = "${data.aws_ssm_parameter.rds_password.value}"
  port                      = "${data.aws_ssm_parameter.rds_port.value}"
  vpc_security_group_ids    = ["${split(",",data.aws_ssm_parameter.rds_security_group_ids.value)}"]
  maintenance_window        = "${data.aws_ssm_parameter.rds_maintenance_window.value}"
  backup_window             = "${data.aws_ssm_parameter.rds_backup_window.value}"
  backup_retention_period   = "${data.aws_ssm_parameter.rds_backup_retention_period.value}"
  subnet_ids                = ["${split(",",data.aws_ssm_parameter.vpc_internal_subnet_ids.value)}"]
  family                    = "${data.aws_ssm_parameter.rds_family.value}"
  license_model             = "${data.aws_ssm_parameter.rds_license_model.value}"
  final_snapshot_identifier = "${var.tags["Application"]}-${var.tags["Environment"]}-final"
  tags                      = "${module.aws_resource_tags.common_tags}"
}

Debug Output

https://gist.github.com/skyzyx/9a509b29130ab07d3e9c91459388ded6

Expected Behavior

The module should have resolved and downloaded correctly.

Actual Behavior

The module failed to download with an HTTP 401.

Steps to Reproduce

1. terraform init -backend=true
2. terraform apply -auto-approve

Additional Context

Following the URLs that Terraform reports using curl -iL, they appear to work as expected.

Attempting slightly older versions of the module (e.g., 1.16.0) have unexpected URL artifacts. Not sure whether the problem is in how the registry resolves the module download URL, or with GitHub's hosting of it.

/cc @SmoshySmosh

[apparentlymart]

Hi @skyzyx! Sorry this isn't working properly.

The only thing that springs to mind here is that you are somehow hitting a GitHub API rate limit. Since Terraform makes unauthenticated requests to download modules, it is subject to GitHub's anonymous request rate limit of 60 requests per hour per IP address.

Is there anything else at your same IP address making unauthenticated requests to the GitHub API such that together these two clients are exhausting the limit?

I am already skeptical about this theory because firstly GitHub documents that rate limit errors are signaled with 403 Forbidden rather than 401 Unauthorized, and secondly because you mentioned being able to make a request with curl immediately afterwards, when presumably you'd still be making the request from the same IP address and thus subject to the same rate limit.

[skyzyx]

The only thing that springs to mind here is that you are somehow hitting a GitHub API rate limit. Since Terraform makes unauthenticated requests to download modules, it is subject to GitHub's anonymous request rate limit of 60 requests per hour per IP address.

_Possibly…_ but wherever possible, I tend to tune software to use the authenticated requests. Not sure how to achieve that with Terraform specifically, however.

Is there anything else at your same IP address making unauthenticated requests to the GitHub API such that together these two clients are exhausting the limit?

I've tried both at work (behind the corp firewall) as well as over the "guest" Wi-Fi. Same issue. Need to try outside of the corp network to see if that's a usable datapoint.

I am already skeptical about this theory because firstly GitHub documents that rate limit errors are signaled with 403 Forbidden rather than 401 Unauthorized, and secondly because you mentioned being able to make a request with curl immediately afterwards, when presumably you'd still be making the request from the same IP address and thus subject to the same rate limit.

Yes, I was able to cURL right afterwards.

Trying again this morning, and I have the same issue.

[skyzyx]

Let's take this snippet from my TRACE log:

$ TF_LOG=TRACE terraform apply -auto-approve
<…/snip…>
2018/07/16 12:03:50 [DEBUG] found available version "1.19.0" for terraform-aws-modules/rds/aws
2018/07/16 12:03:50 [DEBUG] looking up module location from "https://registry.terraform.io/v1/modules/terraform-aws-modules/rds/aws/1.19.0/download"
2018/07/16 12:03:50 [TRACE] detected module source "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
2018/07/16 12:03:50 [DEBUG] fetching module from https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0?archive=tar.gz
2018/07/16 12:03:50 [DEBUG] fetching "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0?archive=tar.gz" with key "1.rds;terraform-aws-modules/rds/aws.1.19.0"
  Found version 1.19.0 of terraform-aws-modules/rds/aws on registry.terraform.io
  Getting source "terraform-aws-modules/rds/aws"
2018/07/16 12:03:50 [DEBUG] plugin: waiting for all plugin processes to complete...
Error downloading modules: Error loading modules: bad response code: 401

And follow the white rabbit — beginning with this line:

2018/07/16 12:03:50 [DEBUG] looking up module location from "https://registry.terraform.io/v1/modules/terraform-aws-modules/rds/aws/1.19.0/download"

We get…

$ curl -i "https://registry.terraform.io/v1/modules/terraform-aws-modules/rds/aws/1.19.0/download"
HTTP/2 204 
server: Cowboy
cache-control: public, max-age=604800, stale-if-error=31536000
last-modified: Wed, 20 Jun 2018 12:47:14 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-terraform-get: https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz
via: 1.1 vegur
via: 1.1 varnish
accept-ranges: bytes
date: Mon, 16 Jul 2018 19:07:28 GMT
via: 1.1 varnish
age: 337
x-served-by: cache-iad2143-IAD, cache-jfk8135-JFK
x-cache: HIT, MISS
x-cache-hits: 1, 0

Then this from x-terraform-get…

$ curl -i "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
HTTP/1.1 302 Found
Server: GitHub.com
Date: Mon, 16 Jul 2018 19:07:37 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Status: 302 Found
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 44
X-RateLimit-Reset: 1531770332
Cache-Control: public, must-revalidate, max-age=0
Expires: Mon, 16 Jul 2018 19:07:37 GMT
Location: https://codeload.github.com/terraform-aws-modules/terraform-aws-rds/legacy.tar.gz/v1.19.0//*
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.028060
X-GitHub-Request-Id: AF88:785E:50822EF:9A3DCA4:5B4CECF9

Then this from Location…

$ curl -i "https://codeload.github.com/terraform-aws-modules/terraform-aws-rds/legacy.tar.gz/v1.19.0//*"
HTTP/1.1 404 Not Found
Content-Length: 15
Access-Control-Allow-Origin: https://render.githubusercontent.com
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
Strict-Transport-Security: max-age=31536000
Vary: Authorization,Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Date: Mon, 16 Jul 2018 19:07:48 GMT
X-GitHub-Request-Id: 1684:660C:9962E1:FEDDF8:5B4CED04

404: Not Found

---

But in the DEBUG log, there appears to be a _little bit_ of URL cleanup.

2018/07/16 12:03:50 [DEBUG] fetching module from https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0?archive=tar.gz

If we follow that…

$ curl -i "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0?archive=tar.gz"
HTTP/1.1 302 Found
Server: GitHub.com
Date: Mon, 16 Jul 2018 19:08:33 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Status: 302 Found
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 44
X-RateLimit-Reset: 1531770332
Cache-Control: public, must-revalidate, max-age=0
Expires: Mon, 16 Jul 2018 19:08:33 GMT
Location: https://codeload.github.com/terraform-aws-modules/terraform-aws-rds/legacy.tar.gz/v1.19.0
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.026126
X-GitHub-Request-Id: 424E:785E:50846BA:9A42298:5B4CED31

Then this from Location…

$ curl -i "https://codeload.github.com/terraform-aws-modules/terraform-aws-rds/legacy.tar.gz/v1.19.0"
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Access-Control-Allow-Origin: https://render.githubusercontent.com
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
Strict-Transport-Security: max-age=31536000
Vary: Authorization,Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
ETag: "f6af441e431ab6f04e7df4ea7c31173b4eaf4d72"
Content-Type: application/x-gzip
Content-Disposition: attachment; filename=terraform-aws-modules-terraform-aws-rds-v1.19.0-0-gf6af441.tar.gz
X-Geo-Block-List: 
Date: Mon, 16 Jul 2018 19:08:47 GMT
X-GitHub-Request-Id: 3E27:660B:5D63C5:990968:5B4CED3F

?=is?F???_1+??R??x?J??Tђ?bI^Qvj?傇????????u?
@??-˻   ?
?LO??w8?̉돨kN?Y??×&?^???B?????oZ?N????ڽcx?????L?.+%#@???G??mz???H9n˟͘=???m?n?EG?q??N?q????j?tFp9??'???ƹ*W"t??]???o?.
<…/snip…>

And we have our file!

I haven't started digging through the source code yet, but there appears to be _something else_ that is responding with a 401 error that isn't showing up in the TRACE logs.

Any ideas?

[skyzyx]

Also, just tested while tethered to my iPhone and disconnected from the corporate network. Same error.

[apparentlymart]

Hmm... very interesting. Thanks for the detailed debug logs, @skyzyx!

One thing to note here is that the URL-like string coming back in X-Terraform-Get there isn't strictly a URL, but rather a string like you could put directly in a module source argument. Terraform does some processing of those before actually making the request, which in particular includes trimming off everything after and including the double-slash // which tells Terraform which subdirectory within the archive to fetch.

Out of curiosity, do you see a different result if you take the registry out of the loop altogether and go directly to the URL-like-string the registry is returning?

module "rds" {
  source                = "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
  # (etc)
}

--

On the subject of making authenticated requests, unfortunately I think that isn't possible right now because it is the registry instructing Terraform to fetch from GitHub, rather than Terraform itself referring directly to GitHub. In that sense, Terraform is just requesting what it's told, without any idea that it happens to be a github URL.

We may be better off here having the registry return a URL to clone from github using git rather than accessing the tarball, since I believe different limits apply there and, crucially, the git client will pick up any credentials you have configured on your system.

[skyzyx]

Out of curiosity, do you see a different result if you take the registry out of the loop altogether and go directly to the URL-like-string the registry is returning?

I've updated my code to this:

module "rds" {
  source                = "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
  version               = "1.19.0"
  identifier            = "${var.tags["Application"]}-${var.tags["Environment"]}"
  engine                = "${data.aws_ssm_parameter.rds_engine.value}"
  engine_version        = "${data.aws_ssm_parameter.rds_engine_version.value}"
  instance_class        = "${data.aws_ssm_parameter.rds_instance_class.value}"
  allocated_storage     = "${data.aws_ssm_parameter.rds_allocated_storage.value}"
  storage_encrypted     = "${data.aws_ssm_parameter.rds_storage_encrypted.value}"
  copy_tags_to_snapshot = "${data.aws_ssm_parameter.rds_copy_tags_to_snapshot.value}"
  multi_az              = "${data.aws_ssm_parameter.rds_multi_az.value}"

  #monitoring_interval             = "${data.aws_ssm_parameter.rds_monitoring_interval.value}"
  name                      = "${data.aws_ssm_parameter.rds_name.value}"
  username                  = "${data.aws_ssm_parameter.rds_username.value}"
  password                  = "${data.aws_ssm_parameter.rds_password.value}"
  port                      = "${data.aws_ssm_parameter.rds_port.value}"
  vpc_security_group_ids    = ["${split(",",data.aws_ssm_parameter.rds_security_group_ids.value)}"]
  maintenance_window        = "${data.aws_ssm_parameter.rds_maintenance_window.value}"
  backup_window             = "${data.aws_ssm_parameter.rds_backup_window.value}"
  backup_retention_period   = "${data.aws_ssm_parameter.rds_backup_retention_period.value}"
  subnet_ids                = ["${split(",",data.aws_ssm_parameter.vpc_internal_subnet_ids.value)}"]
  family                    = "${data.aws_ssm_parameter.rds_family.value}"
  license_model             = "${data.aws_ssm_parameter.rds_license_model.value}"
  final_snapshot_identifier = "${var.tags["Application"]}-${var.tags["Environment"]}-final"
  tags                      = "${module.aws_resource_tags.common_tags}"
}

Trace log is here: https://gist.github.com/skyzyx/9ae6f3fbd4af997c210c57e713e126bd

Still no dice.

[apparentlymart]

Well, on the plus side at least we can remove the registry itself from consideration... it does seem to be an error coming from GitHub, or from some intermediary between you and GitHub.

Unfortunately I don't have any other ideas off the top of my head right now, so I'm going to raise this to the rest of the team and see if anyone else has some ideas where to look next.

In the mean time, since all of the modules on the public registry are backed by github repositories anyway, you may have better luck with forcing Terraform to use the git protocol here instead of just downloading the tarball:

module "rds" {
  source                = "github.com/terraform-aws-modules/terraform-aws-rds"
  # (etc)
}

Terraform's module source resolver should detect that as a shorthand git URL and download the module using git clone instead of a direct HTTP request, which I believe is subject to different auth requirements and rate limits.

[apparentlymart]

Ahh... we have a new avenue to try.

When making HTTP requests here Terraform will honor a ~/.netrc file for credentials if present. I've not looked in detail yet at exactly how that might interact with this codepath, but was wondering if you have such a file on your system, since you mentioned that you tend to prefer to configure auth for things like this.

[skyzyx]

@apparentlymart: That appears to have unblocked me for the time being. Thank you.

[skyzyx]

Yep.

$ cat ~/.netrc
machine api.github.com login skyzyx password {redacted}

[apparentlymart]

Hi again @skyzyx,

I must admit I'm not sure why adding a valid set of credentials to that download request would cause it to fail, but just in the interests of completeness could you try making the curl request you tried earlier (the one to api.github.com that then led you to the final archive URL) with your credentials from netrc included? I believe adding the -n option to the curl call will cause it to also read the .netrc file and add the credentials automatically.

[skyzyx]

curl -n -i "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
HTTP/1.1 401 Unauthorized
Date: Tue, 17 Jul 2018 17:40:50 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 93
Server: GitHub.com
Status: 401 Unauthorized
X-GitHub-Media-Type: github.v3; format=json
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1531852080
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.096038
X-GitHub-Request-Id: DBB4:85AA:871BFBA:AD9BB8D:5B4E2A22

{
  "message": "Bad credentials",
  "documentation_url": "https://developer.github.com/v3"
}

Well, what would you know…. That's some pretty good sleuthing there, I'd have to say. Let me replace my credentials and try again.

curl -n -i "https://api.github.com/repos/terraform-aws-modules/terraform-aws-rds/tarball/v1.19.0//*?archive=tar.gz"
HTTP/1.1 302 Found
Date: Tue, 17 Jul 2018 17:48:46 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Server: GitHub.com
Status: 302 Found
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4996
X-RateLimit-Reset: 1531853119
Cache-Control: public, must-revalidate, max-age=0
Expires: Tue, 17 Jul 2018 17:48:46 GMT
Location: https://codeload.github.com/terraform-aws-modules/terraform-aws-rds/legacy.tar.gz/v1.19.0//*
Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
X-Runtime-rack: 0.033040
Vary: Accept-Encoding
X-GitHub-Request-Id: DC4B:85AA:8725243:ADA7548:5B4E2BFE

The original source URN appears to be working now. Thanks!

[apparentlymart]

Great @skyzyx, I'm glad we got to the bottom of this!

We'll have a think about how we could make it more obvious that .netrc is being consulted here. It's unfortunately happening in the depths of one of the shared libraries we depend on, and so not a place we can easily insert UI-type code, but we may be able to improve on this once we've merged down the development branch for the next major release since it includes some refactoring of how module installation is handled.

I'll make a new issue for that so we don't this idea. Thanks for reporting this!

[skyzyx]

Fantastic to hear. Thank you.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.