Aws-sdk-go: Add support for custom region/endpoint configuration at runtime

[gregorydulin]

Is this related to a problem?

My organization makes use of non-public AWS regions and/or AWS-compatible private cloud computing environments (e.g. Eucalyptus). When attempting to use tools compiled with aws-sdk-go (e.g. Hashicorp Consul), we've found that these compiled tools don't support custom regions and/or service endpoints at runtime (i.e. the way boto does).

Feature description

I'd like to be able to specify custom AWS regions and their associated service endpoints in a .json config file (e.g. /etc/boto/endpoints.json and/or an environment variable (e.g. BOTO_ENDPOINTS), and expect a go application compiled with a recent aws-sdk-go to honor those settings. This would mimic the custom region/endpoint support in boto and the aws cli, which we use extensively and greatly appreciate.

Describe alternatives you've considered

Currently, we're forking each go project (e.g. Hashicorp Consul), and adding custom service endpoints to vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go

Another solution (I think) would be to fork aws/aws-sdk-go, add the custom endpoints to aws/endpoints/defaults.go, then fork each go project, replacing all references to github.com/aws/aws-sdk-go with our forked repo.

Both of these solutions require us to maintain a fork of any golang project we use. If this feature were implemented, then eventually (once the upstream project updated to the latest aws-sdk-go), we could just use the upstream project directly, and provide our custom endpoints at runtime.

[gregorydulin]

My organization may be willing to devote some resources (developer-hours) to implementing this feature, but only after we get a nod from the aws-sdk-go maintainers that such a pull-request would be well-received (e.g. it's a feature the maintainers are interested in, and there's no duplication of effort).

[jasdel]

Thanks for reaching out @gregorydulin with this feature request. The SDK has support for loading a custom endopints.json file to be used as a endpoints.Resolver and endpoints.Partition.

The endpoints.DecodeModel function takes an io.Reader as output, and returns a endpoints.Resolver.

custEndpointFile, err := os.Open(custEndpointFilename)
defer custEndpointFile.Close()

resolver, err := endpoints.DecodeModel(custEndpointFile)

sess, err := session.NewSession(&aws.Config{
    EndpointResolver: resolver,
})

If you are looking to define a set of endpoints in addition to the SDK's bundled endpoints you can do this by implementing the enpdoints.Resolver interface and delegate to the SDK's built in resolver if the resolver loaded from your custom endpoints.json doesn't service the passed in region.

[gregorydulin]

@jasdel, If I'm understanding things correctly, I could fork each upstream project, and use the endpoints.DecodeModel function to have that project read a custom endpoints.json file.

However, that would still require me to maintain a fork of each upstream project we use. I'd really apreciate some support for a custom endpoints.json in a standard location/variable built in to the SDK itself. That way, any binary compiled using the (latest) SDK would support a custom endpoints.json (either in a specific location, or as an environment variable, or both), and I could just download the latest upstream binary, vs forking and recompiling.

On the other hand, if you're saying "Here's the function you'll want to use in your forthcoming PR to add this support to the SDK", then thanks! I certainly appreciate the help.

[jasdel]

Thanks for the clarification @gregorydulin. Correct, the SDK does not provide an external configuration option for configuring how the SDK's endpoints are resolved. Reviewing the current AWS CLI, boto3 and botocore, I think this behavior has been removed from these versions, or at least not yet found the functionality. I've reached out to the AWS CLI team for more clarification on the CLI and Python SDK's current behavior.

Maintaining a custom fork of the utilities using the SDK is less than idea, but does accomplish the task. Building a custom implementation of the SDK's endpoints.Resolver will allow you to load/use custom endpoint metadata based on your use case. Using this pattern you would not need to fork the SDK, but would still need to maintain a fork of the utilities to use the custom endpoint resolver.

[jasdel]

HI @gregorydulin I synced with the AWS CLI team, and clarify the AWS SDK for Python and CLI's behavior regarding custom endpoints. The CLI and SDK do not support the environment variable BOTO_ENDPOINT, nor /etc/boto/endpoints.json file. The CLI and Python SDK does support loading a custom endpoints.json file from ~/.aws/models/endpoints.json. This functionality is specific to the AWS SDK for Python and AWS CLI, and is not used by any of the other AWS SDKs.

[gregorydulin]

Thanks for checking witht he CLI team, @jasdel. Sorry, I meant BOTO_ENDPOINT and /etc/boto/endpoints.json as arbitrary examples. I couldn't remember off the top of my head where exactly boto expected these things.

The AWS CLI (and boto) (as late as 1.16.158, I haven't looked at changes since then) looks in /etc/boto.cfg for additional boto configuration.
http://boto.cloudhackers.com/en/stable/boto_config_tut.html#details

This configuration file can contain an endpoints_path option, who's value is a path to a json file (e.g. /etc/boto/endpoints.json)
http://boto.cloudhackers.com/en/stable/boto_config_tut.html#boto

We're currently using this functionality in order to use the AWS CLI to access AWS regions which aren't connected to the public internet. We can use the AWS CLI unmodified (tested on 1.16.158), in conjunction with the config files above, to access AWS endpoints on these private regions.

I am painfully aware that the AWS SDK for Python is the only AWS SDK which supports this functionality. We're forced to maintain forks of upstream projects which we wouldn't have to maintain if this functionality was standard across all AWS SDKs. I'm submitting this ticket against the Go SDK because we use a lot of tools written in Go to interact with AWS APIs (e.g Hashicorp's Vault, Consul, and Terraform). If we ever start using Java tools, I'll be submitting a similar ticket against the Java SDK.

Again, I'm more than willing to write the code to add this functionality (and any tests, documentation, etc. required to get it through the approval process); I just want to make sure it'll make it past the maintainers' "this is a good feature" test before I start working on it.

[Promaethius]

@gregorydulin any updates on this?

[gregorydulin]

@Promaethius It looks like the additional endpoints I needed were hard-coded in the SDK a few weeks after my last comment.

That solution temporarily resolves the issue. All currently available regions (that I'm aware of) are now accessible without fork/recompile (assuming the project maintainers are using a recent version of this SDK). However, if a new region comes online, it'll be inaccessible until someone hard-codes the new endpoints in this SDK, and the downstream project integrates the new SDK into their build.

[Promaethius]

@gregorydulin one last question then. If the SDK is passed a region which has FIPS endpoints, will it prioritize those for any given service?

[jasdel]

@Promaethius A region like fips-us-west-2 will be specific to the modeling of a given service's set of endpoints. There isn't strict consistency between the service modeled endpoints. If a service does not have a modeled endpoint for that FIPS pseudo region, the endpoint resolver most likely will resolve an endpoint that is invalid. Using the pseudo region, fips-us-west-2 in the endpoint pattern directly instead of modifying the name portion of the endpoint. e.g. API gateway does not have a modeled FIPS endpoint in the SDK for fips-us-west-2 so the SDK would incorrectly attempt to use apigateway.fips-us-west-2.amazonaws.com, instead of the correct endpoint, apigateway-fips.us-east-2.amazonaws.com.

With the capability of the SDK's current endpoint modeling, In the cases where FIPS endpoints are desired across multiple APIs, it is probably best to specify the specific endpoint explicitly when creating the API client value in your application.

[KapitalEarth]

Del Ponte = The Ponte

On 19 Aug 2020, at 3:42 PM, Jason Del Ponte [e-mail] wrote:


@Promaethius A region like fips-us-west-2 will be specific to the modeling of a given service's set of endpoints. There isn't strict consistency between the service modeled endpoints. If a service does not have a modeled endpoint for that FIPS pseudo region, the endpoint resolver most likely will resolve an endpoint that is invalid. Using the pseudo region, fips-us-west-2 in the endpoint pattern directly instead of modifying the name portion of the endpoint. e.g. API gateway does not have a modeled FIPS endpoint in the SDK for fips-us-west-2 so the SDK would incorrectly attempt to use apigateway.fips-us-west-2.amazonaws.com, instead of the correct endpoint, apigateway-fips.us-east-2.amazonaws.com.

With the capability of the SDK's current endpoint modeling, In the cases where FIPS endpoints are desired across multiple APIs, it is probably best to specify the specific endpoint explicitly when creating the API client value in your application.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.