Terraform: Don't validate AWS Regions in provider config bootstrap

[dastbe]

Hey,

Wanted to open this issue before I supplied a code change. I noticed that terraform is using a static array of available regions to whitelist terraforming. Of course, new regions are coming all the time and some people are able to make use of them before they're publicly available. For unlaunched services one could create a shim provider until its early access/full release, but here the only solution is to branch and build :(.

Is there a compelling reason to have any validation here, or at most warn that this is not a known public region?

Here's the method in question.

Thanks!

[cjeanneret]

It would be better to ask AWS directly what regions are available I guess? For example, the following awscli command returns the required things:

aws ec2 describe-regions

Pretty sure the aws-sdk for go supports such a thing — might even be cached for some time if needed.

[stack72]

Hi

To quote @radeksimko from another issue (https://github.com/hashicorp/terraform/pull/7383#issuecomment-229169574)

I think this is purely for static/offline validation of the aws provider configuration - so that the user gets a human-friendly error for cases like this:

provider "aws" {
  region = "not-exists"
}
We cannot verify this online, even though it's available from the API, because we'd need to choose a valid region (and hard-coding a single region is not a good idea since that region becomes a single point of failure) + have valid credentials so this is kind of a 🐔 & 🍳 problem when validating provider which is responsible for setting up the connection.

Also the way most SDKs work is that they simply take the given region as a string and make a hostname of it, so the error message user would see if Terraform wouldn't perform this offline validation is something like this:

Could not connect to the endpoint URL: "https://sts.not-exists.amazonaws.com/"
What I think would make sense to do though is to keep such list of valid regions in the SDK and let the SDK perform the validation, but SDK doesn't currently offer such functionality neither a list of regions.

Hope this helps

Paul

[radeksimko]

For unlaunched services one could create a shim provider until its early access/full release, but here the only solution is to branch and build

All AWS tooling would typically also need to be rebuilt using the JSON blueprints to get a new API functionality - i.e. it is non-trivial to use off-the-shelf tools to control private, unreleased APIs anyway and I don't expect this to be easy in Terraform either.

That said this may not be the case for regions that offer the exact same APIs.

I'm thinking we could either expand or add to https://github.com/hashicorp/terraform/pull/6533 to allow the user to temporarily disable this region validation. Suggestions welcomed. 😉

[afoldesi-gds]

Yes. Extending #6533 with an option to skip the validation against the hardcoded list of regions would have been a nice solution (however it's now closed in favour of #7874 which doesn't actually has to do anything with regions).

So, here I am. I'm in private beta of the new eu-west-2 region, finally we have access, but I can not use my terraform scripts.

What's the fastest way to get a change released to solve this problem? What's the best?

PS. To be clear, this is NOT to use unreleased APIs. This is to use a new region, same generic APIs.

[Cinderhaze]

I am in the same boat.. I am in a separate, air-gapped AWS region with different region and endpoints. (similar to govcloud-us) There is a team that has a recompiled version of terraform with a modified endpoints.json file injected into the build process, but otherwise identical.

It would be nice to be able to use the official binary without rebuilding by being able to either point terraform to a config file which has the region/endpoint information in it, or supply it as a part of the aws provider

[radeksimko]

FYI - I just opened https://github.com/hashicorp/terraform/pull/11358 to add skip_region_validation. It comes with a few caveats, but it should hopefully help you overcoming the mentioned issues. 😉

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.