Botocore: SigV4 sign arbitrary requests

[rhboyd]

Botocore Sigv4 signs all AWS API Requests. It should expose the ability to sign arbitrary https requests.

API Gateway has an AWS_IAM auth mechanism and this results in needed to sigv4 sign https requests to domains not included in the usual Python SDK. As more databases (like Neptune) offer AWS IAM auth to manage database permissions, this problem will appear more frequently. The current "recommended" approach is to use a random third-party library named "aws-requests-auth" but that library isn't maintained by AWS and I'd really like to avoid asking new devs to Google "AWS SIgv4 Python", which returns about a dozen different repos. Should AWS be encouraging people to use 3P libraries for something as important as signing requests?

I propose that botocore expose some functionality to attach SigV4 Auth to any request. I think this can be accomplished by adding a def __call__(self, request) method to the SigV4Auth class and I'd even be willing to implement and test this feature if the botocore team agrees that this feature is needed.

I hacked together a way to perform the signing with the existing functionality, but it's pretty sloppy and requires building two requests in parallel then stripping the auth bits out of one so that we can send the other.

https://gist.github.com/rhboyd/1e01190a6b27ca4ba817bf272d5a5f9a

[benkehoe]

In my ideal world, this is available on the Session object, so I could do something like session = botocore.Session(); response = requests.get(url, auth=session.signer(service='execute-api')) or something similar. And then also made available on the boto3 Session object, so I don't have to muck about getting the botocore session out of it.

[swetashre]

@rhboyd -Thank you for your post. I would mark this as a feature request.

[tyldavis]

I was surprised to find this wasn't already a feature! This would be hugely beneficial for AWS_IAM secured API Gateway endpoints.

Edit: I am especially surprised after finding this is a feature of the Go SDK, and in fact it appears all other SDKs support this already.

[ryansb]

I'd also be happy to do implementation work on this, as long so the botocore team is ok with the new __call__ on SigV4Auth.

[benkehoe]

It could be a new wrapper class with a __call__ method

[richardhboyd]

I've discovered a cleaner way to do this lately

import boto3
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import requests

session = boto3.Session()
credentials = session.get_credentials()
creds = credentials.get_frozen_credentials()

def signed_request(method, url, data=None, params=None, headers=None):
    request = AWSRequest(method=method, url=url, data=data, params=params, headers=headers)
    # "service_name" is generally "execute-api" for signing API Gateway requests
    SigV4Auth(creds, "service_name", REGION).add_auth(request)
    return requests.request(method=method, url=url, headers=dict(request.headers), data=data)

def main():
    url = f"my.url.example.com/path"
    data = {"environmentId": self._environment_id}
    headers = {'Content-Type': 'application/x-amz-json-1.1'}
    response = signed_request(method='POST', url=url, data=data, headers=headers)

if __name__ == "__main__":
    main()

[tyldavis]

I've discovered a cleaner way to do this lately

[...]

Isn't this basically the same thing as your original Gist (I'm assuming @rhboyd and @richardhboyd are the same person... apologies if not), but without calling AWSRequest.prepare()? Looking at the source it seems like it isn't doing much anyway.

[richardhboyd]

yeah, the request.headers part can also be re-used for websockets that need IAM Auth (Neptune Gremlin connections) and the credentials work in a more environment agnostic way as well.