Boto3: Presign URL for AWS IoT https / mqtt client

[cetex]

Hi!

Trying to get a websockets from web-browsers to work through AWS IoT by presigning a URL which is delivered to the web-browser/client but it seems there is no way for boto3 to presign a URL with the method "GET" which is the method used to access IoT Websockets (as far as i understand at least)

Code:

import boto3
sts = boto3.client('sts')
creds = sts.assume_role(RoleArn="arn:aws:iam::000000000000:role/service-role/test", RoleSessionName="web-client-some-ip")
iot = boto3.client(
        'iot',
        aws_access_key_id=creds['Credentials']['AccessKeyId'],
        aws_secret_access_key=creds['Credentials']['SecretAccessKey'],
        aws_session_token=creds['Credentials']['SessionToken'])
print(iot.generate_presigned_url('GET', ExpiresIn=3600))

Traceback:

Traceback (most recent call last):
  <flask traceback dropped>
  File "/home/oskar/Code/project/web.py", line 96, in get_live_feed
    print(iot.generate_presigned_url('GET', ExpiresIn=3600))
  File "/home/oskar/venv/lib/python3.6/site-packages/botocore/signers.py", line 563, in generate_presigned_url
    raise UnknownClientMethodError(method_name=client_method)
botocore.exceptions.UnknownClientMethodError: Client does not have method: GET

According to the docs it is possible to presign url's for https connections to AWS IoT gateway, but there seems be something missing in boto to support it.

[joguSD]

The presigned url functionality exposed on the iot client is for specific API operations.
So rather than the HTTP method it should be an API operation:

operation_name = 'list_authorizers'
iot.generate_presigned_url(operation_name, {}, ExpiresIn=3600)

Let me know if that helps at all.

[cetex]

Thanks for the input, I realize that I wasn't too clear in the problem definition.

What i'd like to achieve is to get a presigned url to the websocket endpoint of the IoT Gateway itself, so we can deliver a presigned url to our users.

I've implemented my own function for this now and this is how i call it:

# In reality this is credentials from boto3 sts
creds = { 'Credentials': { 'AccessKeyId': 'ASIAxxxxxxxx', 'SecretAccessKey': 'ascgscgjahgcjhackhachkajshckljahsc', 'SessionToken': 'acklashckjhasckljhalscjkhklajschakljschklajschkjasch' } }

iot_url = aws_presign(
    access_key=creds['Credentials']['AccessKeyId'],
    secret_key=creds['Credentials']['SecretAccessKey'],
    session_token=creds['Credentials']['SessionToken'],
    host='<iot-endpoint>.iot.eu-west-1.amazonaws.com',
    region=boto3.session.Session().region_name,
    method='GET',
    protocol='wss',
    uri='/mqtt',
    service='iotdevicegateway',
    expires=900)

This returns a URL like:

wss://<iot-endpoint>.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAxxxxxxxx%2F20180130%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-Date=20180130T113606Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=082da5cfd2600f33e81bd18e756348cd1fada1a23b9038035d1b4c6d70350960&X-Amz-Security-Token=acklashckjhasckljhalscjkhklajschakljschklajschkjasch

I'd like this functionality to be available in boto though. :)

[joguSD]

Ah, that makes sense. Marking this as a feature request then! :)

[bbernays]

@cetex - Until this functionality can be built into Boto3 can you post the code for creating the presigned URL?

[cetex]

Sure, just pasting it here.

Of course all 'n' were reinterpreted as actual newlines so i had to fix that manually, I hope i didn't miss something more.

import hmac, datetime, urllib.parse, hashlib
def aws_sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def aws_getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = aws_sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = aws_sign(kDate, regionName)
    kService = aws_sign(kRegion, serviceName)
    kSigning = aws_sign(kService, 'aws4_request')
    return kSigning

def aws_presign(access_key=None, secret_key=None, session_token=None, host=None, region=None, method=None, protocol=None, uri=None, service=None, expires=3600, payload_hash=None):
    # method=GET, protocol=wss, uri=/mqtt service=iotdevicegateway
    assert 604800 >= expires >= 1, "Invalid expire time 604800 >= %s >= 1" % expires;

    # Date stuff, first is datetime, second is just date.
    t = datetime.datetime.utcnow()
    date_time = t.strftime('%Y%m%dT%H%M%SZ')
    date = t.strftime('%Y%m%d')
    # Signing algorithm used
    algorithm = 'AWS4-HMAC-SHA256'

    # Scope of credentials, date + region (eu-west-1) + service (iot gateway hostname) + signature version
    credential_scope = date + '/' + region + '/' + service + '/' + 'aws4_request'
    # Start building the query-string
    canonical_querystring = 'X-Amz-Algorithm=' + algorithm
    canonical_querystring += '&X-Amz-Credential=' + urllib.parse.quote_plus(access_key + '/' + credential_scope)
    canonical_querystring += '&X-Amz-Date=' + date_time
    canonical_querystring += '&X-Amz-Expires=' + str(expires)
    canonical_querystring += '&X-Amz-SignedHeaders=host'

    if payload_hash is None:
        if service == 'iotdevicegateway':
            payload_hash = hashlib.sha256(b'').hexdigest()
        else:
            payload_hash = 'UNSIGNED-PAYLOAD'

    canonical_headers = 'host:' + host + '\n'
    canonical_request = method + '\n' + uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\nhost\n' + payload_hash

    string_to_sign = algorithm + '\n' + date_time + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode()).hexdigest()
    signing_key = aws_getSignatureKey(secret_key, date, region, service)
    signature = hmac.new(signing_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest()

    canonical_querystring += '&X-Amz-Signature=' + signature
    if session_token:
        canonical_querystring += '&X-Amz-Security-Token=' + urllib.parse.quote(session_token)

    return protocol + '://' + host + uri + '?' + canonical_querystring

[DhanashreeChavan]

Did you see the below failure after while connecting to the aws IoT instance ?
UnicodeError: encoding with 'idna' codec failed

[claudiopastorini]

Did you see the below failure after while connecting to the aws IoT instance ?
UnicodeError: encoding with 'idna' codec failed

Did anybody solve this? It seems related to the length of the URI.

UnicodeError: encoding with 'idna' codec failed (UnicodeError: label too long)

This is my really simple example with paho-mqtt:

import paho.mqtt.client as mqtt

import urllib.parse

def on_connect(mqttc, obj, flags, rc):
    print("rc: "+str(rc))

def on_message(mqttc, obj, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

def on_publish(mqttc, obj, mid):
    print("mid: "+str(mid))

def on_subscribe(mqttc, obj, mid, granted_qos):
    print("Subscribed: "+str(mid)+" "+str(granted_qos))

def on_log(mqttc, obj, level, string):
    print(string)

mqttc = mqtt.Client(transport="websockets")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe

url = "YOUR_PRESIGNED_URL_STARTING_WITH_WSS_HERE"

mqttc.connect(url, 443, 60)
mqttc.subscribe("test/iot", 0)

mqttc.loop_forever()

[claudiopastorini]

Thanks @cetex for your snippet.

If there is someone (like me) that struggles with this error:

[ERROR] ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::***:assumed-role/***/*** is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::***:role/***
...

I suggest you reading this post from AWS Knowled-Center that helps to configure in the right way IAM roles for sts:AssumeRole operation.