Boto3: Updating AWS Gamelift Realtime Script doesn't work as expected

[legut2]

Describe the bug
Not able to upload a zip file for an AWS gamelift realtime script easily with local zip file upload.

Steps to reproduce
Please see stack overflow post here: https://stackoverflow.com/questions/64549611/how-do-i-update-an-aws-gamelift-script-with-boto3-in-python

Expected behavior
I expect to be able to specify a zip file with either a path or base64 encoded version of the file to upload a zip that should work. I've tried both and permutations of both but can't figure out what is going wrong exactly.

Debug logs
Full stack trace by adding boto3.set_stream_logger('') to your code.

2020-10-27 03:03:17,774 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-10-27 03:03:17,782 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-10-27 03:03:17,782 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-10-27 03:03:17,788 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-10-27 03:03:17,789 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-10-27 03:03:17,791 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-10-27 03:03:17,794 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-10-27 03:03:17,805 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-10-27 03:03:17,806 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-10-27 03:03:17,809 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-10-27 03:03:17,810 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-10-27 03:03:17,834 botocore.utils [INFO] IMDS ENDPOINT: http://169.254.169.254/
2020-10-27 03:03:17,839 botocore.credentials [DEBUG] Looking for credentials via: env
2020-10-27 03:03:17,842 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-10-27 03:03:17,843 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-10-27 03:03:17,844 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-10-27 03:03:17,845 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-10-27 03:03:17,850 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2020-10-27 03:03:17,852 botocore.loaders [DEBUG] Loading JSON file: C:\Users\danie\Documents\...\buildScript\venv\lib\site-packages\botocore\data\endpoints.json
2020-10-27 03:03:17,866 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000021242AE6700>
2020-10-27 03:03:18,027 botocore.loaders [DEBUG] Loading JSON file: C:\Users\danie\Documents\...\buildScript\venv\lib\site-packages\botocore\data\gamelift\2015-10-01\service-2.json
2020-10-27 03:03:18,056 botocore.hooks [DEBUG] Event creating-client-class.gamelift: calling handler <function add_generate_presigned_url at 0x0000021242A904C0>
2020-10-27 03:03:18,061 botocore.endpoint [DEBUG] Setting gamelift timeout as (60, 60)
2020-10-27 03:03:18,065 botocore.loaders [DEBUG] Loading JSON file: C:\Users\danie\Documents\...\Assets\buildScript\venv\lib\site-packages\botocore\data\_retry.json
2020-10-27 03:03:18,066 botocore.client [DEBUG] Registering retry handlers for service: gamelift
2020-10-27 03:03:18,105 botocore.hooks [DEBUG] Event before-parameter-build.gamelift.UpdateScript: calling handler <function generate_idempotent_uuid at 0x0000021242B0B8B0>
2020-10-27 03:03:18,107 botocore.hooks [DEBUG] Event before-call.gamelift.UpdateScript: calling handler <function inject_api_version_header_if_needed at 0x0000021242B12160>
2020-10-27 03:03:18,108 botocore.endpoint [DEBUG] Making request for OperationModel(name=UpdateScript) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'GameLift.UpdateScript', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.16.5 Python/3.9.0 Windows/10 Botocore/1.19.5'}, 'body': b'{"ScriptId": "SCRIPT_ID_REDACTED_FROM_STACK_TRACE", "Version": "0.4.7", "ZipFile": "LS1_BASE_64_ENCODED_ZIP_REDACTED_FROM_STACK_TRACE_T0="}', 'url': 'https://gamelift.us-east-2.amazonaws.com/', 'context': {'client_region': 'us-east-2', 'client_config': <botocore.config.Config object at 0x0000021242F865B0>, 'has_streaming_input': False, 'auth_type': None}}
2020-10-27 03:03:18,142 botocore.hooks [DEBUG] Event request-created.gamelift.UpdateScript: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000021242F863D0>>
2020-10-27 03:03:18,153 botocore.hooks [DEBUG] Event choose-signer.gamelift.UpdateScript: calling handler <function set_operation_specific_signer at 0x0000021242B0B790>
2020-10-27 03:03:18,155 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-10-27 03:03:18,156 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:gamelift.us-east-2.amazonaws.com
x-amz-date:20201027T080318Z
x-amz-target:GameLift.UpdateScript

content-type;host;x-amz-date;x-amz-target
c9b53d85a52ea990434f2065f0c11b0fa0ca19b867a884ba25df49cf7f100d3a
2020-10-27 03:03:18,158 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20201027T080318Z
20201027/us-east-2/gamelift/aws4_request
fd085a2887d85b152bcc9a3c4d7bb94799b484394e546a3ae8263b9bdb2576b9
2020-10-27 03:03:18,160 botocore.auth [DEBUG] Signature:
abde19f0764a04166fe38d61e722c937a32eb95873dc359bf8ee67e3bf9c6771
2020-10-27 03:03:18,166 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://gamelift.us-east-2.amazonaws.com/, headers={'X-Amz-Target': b'GameLift.UpdateScript', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.16.5 Python/3.9.0 Windows/10 Botocore/1.19.5', 'X-Amz-Date': b'20201027T080318Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20201027/us-east-2/gamelift/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=REDACTED', 'Content-Length': '24286'}>
2020-10-27 03:03:18,168 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): gamelift.us-east-2.amazonaws.com:443
2020-10-27 03:03:18,445 urllib3.connectionpool [DEBUG] https://gamelift.us-east-2.amazonaws.com:443 "POST / HTTP/1.1" 400 81
2020-10-27 03:03:18,459 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': '6c3f7481-a68f-4289-957b-f7eb58966b96', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '81', 'Date': 'Tue, 27 Oct 2020 08:03:19 GMT', 'Connection': 'close'}
2020-10-27 03:03:18,459 botocore.parsers [DEBUG] Response body:
b'{"__type":"InvalidRequestException","Message":"Failed to unzip the zipped file."}'
2020-10-27 03:03:18,464 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': '6c3f7481-a68f-4289-957b-f7eb58966b96', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '81', 'Date': 'Tue, 27 Oct 2020 08:03:19 GMT', 'Connection': 'close'}
2020-10-27 03:03:18,465 botocore.parsers [DEBUG] Response body:
b'{"__type":"InvalidRequestException","Message":"Failed to unzip the zipped file."}'
2020-10-27 03:03:18,468 botocore.hooks [DEBUG] Event needs-retry.gamelift.UpdateScript: calling handler <botocore.retryhandler.RetryHandler object at 0x0000021242F86F10>
2020-10-27 03:03:18,471 botocore.retryhandler [DEBUG] No retry needed.
Traceback (most recent call last):
  File "C:\Users\danie\Documents\...\buildScript\updateScript.py", line 38, in <module>
    main(sys.argv[1])
  File "C:\Users\danie\Documents\...\buildScript\updateScript.py", line 31, in main
    response = client.update_script(
  File "C:\Users\danie\Documents\...\buildScript\venv\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\danie\Documents\...\buildScript\venv\lib\site-packages\botocore\client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the UpdateScript operation: Failed to unzip the zipped file.

[swetashre]

@legut2 - Thank you for your post. With boto3 you don't have to specify --zip-file in the request. Have your tried by specifying only filename in the ZipFile parameter ?

[legut2]

@legut2 - Thank you for your post. With boto3 you don't have to specify --zip-file in the request. Have your tried by specifying only filename in the ZipFile parameter ?

When using ZipFile=b'RealtimeServer.zip'

botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the UpdateScript operation: You may be sending a string instead of an actual binary file, try --zip-file fileb://[pathToZipFile.zip]

When using ZipFile=b'\"fileb://RealtimeServer.zip\"'

botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the UpdateScript operation: Failed to unzip the zipped file.

[legut2]

@swetashre any thoughts?

[swetashre]

@legut2 - I am able to reproduce the behavior. I will investigate more and update.

[legut2]

@swetashre any idea where the relevant code is located? I wanted to take a look but I'm unfamiliar with the repo. I tried a quick search but couldn't find it. I've been meaning to contribute to open source projects anyways. Seems like an opportunity to do so.

[swetashre]

@legut2 - sorry for late reply. Can you try this and see if it works for you ?

In [20]: with open('test.zip','rb') as f:
    ...:     contents = f.read()
In [21]: response = client.update_script(ScriptId='script-*****',ZipFile=contents)

[legut2]

@legut2 - sorry for late reply. Can you try this and see if it works for you ?

In [20]: with open('test.zip','rb') as f:
    ...:     contents = f.read()
In [21]: response = client.update_script(ScriptId='script-*****',ZipFile=contents)

That seems to have done the trick. Thank you so much for your help.
Posting code below for others who get stuck, it will only work up to 5mb zip file. S3 bucket needed for any zipfile that ends up being larger.

from zipfile import ZipFile
import os
from os.path import basename
import boto3
import sys, getopt

def main(argv):
    versInput = sys.argv[1]
    #initializes client for updating script in aws gamelift
    client = boto3.client('gamelift')

    #Where is the directory relative to the script directory. In this case, one folder dir lower and the contents of the RealtimeServer dir
    dirName = '../RealtimeServer'

    # create a ZipFile object
    with ZipFile('RealtimeServer.zip', 'w') as zipObj:
        # Iterate over all the files in directory
        for folderName, subfolders, filenames in os.walk(dirName):
            rootlen = len(dirName) + 1
            for filename in filenames:
                #create complete filepath of file in directory
                filePath = os.path.join(folderName, filename)
                # Add file to zip
                zipObj.write(filePath, filePath[rootlen:])

    with open('RealtimeServer.zip','rb') as f:
        contents = f.read()

    response = client.update_script(
        ScriptId="SCRIPT_ID_GOES_HERE",
        Version=sys.argv[1],
        ZipFile=contents
    )

if __name__ == "__main__":
   main(sys.argv[1])

requirements.txt

boto3==1.16.5
botocore==1.19.5
jmespath==0.10.0
python-dateutil==2.8.1
s3transfer==0.3.3
six==1.15.0
urllib3==1.25.11

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.