Aws-cli: Error: Connection was closed before we received a valid response from endpoint URL

[moldstadt]

I am receiving the above error when trying to upload my code to Lambda using the cli:

aws lambda update-function-code --function-name <api> --zip-file fileb://api.zip

This was working fine a couple of days ago, but today it just keeps failing.

Here is a piece of the debug logs, where the error is thrown:

2019-01-10 15:08:06,708 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002439C15CE10>>
2019-01-10 15:08:06,708 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002439A2B69D8>
2019-01-10 15:08:06,744 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-10 15:08:06,744 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/################################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190110T130806Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-10 15:08:06,745 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190110T130806Z
20190110/us-east-1/lambda/aws4_request
003e24e57b3ce08238427bf808241b76bcd70c30b3cafc1bf5e3c18f92a6f8da
2019-01-10 15:08:06,745 - MainThread - botocore.auth - DEBUG - Signature:
###########################################
2019-01-10 15:08:06,746 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/#############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190110T130806Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=############################/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###################################', 'Content-Length': '15340895'}>
2019-01-10 15:09:08,100 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x000002439AB59E10>
2019-01-10 15:09:08,101 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/##############################/code".
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 273, in send
    endpoint_url=request.url
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###########################/code".
2019-01-10 15:09:08,128 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.23947896318623496
2019-01-10 15:09:08,129 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.23947896318623496 seconds

It retries a few times but continually fails. It finally fails with the following error:

Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/##############################/code".

What is the problem?

[moldstadt]

I am still experiencing this problem. Any suggestions?

[justnance]

@moldstadt - Thank you for reaching out. Since this use to work as expected it is possible the file size has increased and exceeded the Lambda limitation. Please let us know if the file size is greater than 50 MB.

Also, thanks for providing the debug output in your initial response. It appears only part of the debug was provided and I'm not able to determine the CLI version in use. If the file size has not exceeded the Lamdba limits and this issue continues, please provide a more of the debug output.

Also, you can try the following:

1. enable history
2. rerun the command to reproduce the timeout.
3. run aws history show to see if there is any other status codes that may help us understand the problem better.

[moldstadt]

@justnance Thanks for the feedback.

The zip file is only 11MB (67MB uncompressed).

The CLI version is 1.16.17.

Here is a more complete debug log:

call aws lambda update-function-code --function-name ############################### --zip-file fileb://API.zip --debug
2019-01-11 12:41:02,455 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7
2019-01-11 12:41:02,455 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['lambda', 'update-function-code', '--function-name', '###############################', '--zip-file', 'fileb://API.zip', '--debug']
2019-01-11 12:41:02,457 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x000002CF1562E7B8>
2019-01-11 12:41:02,457 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,457 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x000002CF14F877B8>
2019-01-11 12:41:02,458 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,459 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x000002CF14FAB730>
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2019-01-11 12:41:02,461 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,461 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2019-01-11 12:41:02,462 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,462 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2019-01-11 12:41:02,663 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,663 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x000002CF154D0AE8>
2019-01-11 12:41:02,665 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,665 - MainThread - awscli.customizations.history - DEBUG - Enabling CLI history
2019-01-11 12:41:02,675 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,675 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2019-01-11 12:41:02,677 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\lambda\2015-03-31\service-2.json
2019-01-11 12:41:02,680 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.lambda: calling handler <function register_retries_for_service at 0x000002CF14E6C1E0>
2019-01-11 12:41:02,681 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: lambda
2019-01-11 12:41:02,684 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lambda: calling handler <function add_waiters at 0x000002CF156349D8>
2019-01-11 12:41:02,722 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('function-name', <awscli.arguments.CLIArgument object at 0x000002CF15F5CFD0>), ('zip-file', <awscli.arguments.CLIArgument object at 0x000002CF15F58400>), ('s3-bucket', <awscli.arguments.CLIArgument object at 0x000002CF15F625F8>), ('s3-key', <awscli.arguments.CLIArgument object at 0x000002CF15F620B8>), ('s3-object-version', <awscli.arguments.CLIArgument object at 0x000002CF15F62048>), ('publish', <awscli.arguments.BooleanArgument object at 0x000002CF15F620F0>), ('no-publish', <awscli.arguments.BooleanArgument object at 0x000002CF15F62160>), ('dry-run', <awscli.arguments.BooleanArgument object at 0x000002CF15F62278>), ('no-dry-run', <awscli.arguments.BooleanArgument object at 0x000002CF15F62240>), ('revision-id', <awscli.arguments.CLIArgument object at 0x000002CF15F62358>)])
2019-01-11 12:41:02,723 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_streaming_output_arg at 0x000002CF1562ED08>
2019-01-11 12:41:02,724 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function _modify_zipfile_docstring at 0x000002CF14FAB8C8>
2019-01-11 12:41:02,725 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_cli_input_json at 0x000002CF14FABD08>
2019-01-11 12:41:02,727 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function unify_paging_params at 0x000002CF15649BF8>
2019-01-11 12:41:02,767 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\lambda\2015-03-31\paginators-1.json
2019-01-11 12:41:02,768 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_generate_skeleton at 0x000002CF154C9950>
2019-01-11 12:41:02,773 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.lambda.update-function-code: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x000002CF15F624A8>>
2019-01-11 12:41:02,774 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.lambda.update-function-code: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x000002CF15F623C8>>
2019-01-11 12:41:02,776 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.function-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000002CF144BB908>
2019-01-11 12:41:02,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <function validate_is_zip_file at 0x000002CF14FAB6A8>
2019-01-11 12:41:02,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <awscli.argprocess.ParamShorthandParser object at 0x000002CF151F1BE0>
2019-01-11 12:41:02,778 - MainThread - awscli.arguments - DEBUG - Unpacked value of '###############################' for parameter "function_name": '###############################'
2019-01-11 12:41:02,780 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.zip-file: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000002CF144BB908>
2019-01-11 12:41:02,793 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <function validate_is_zip_file at 0x000002CF14FAB6A8>
2019-01-11 12:41:02,885 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <awscli.argprocess.ParamShorthandParser object at 0x000002CF151F1BE0>
2019-01-11 12:41:07,698 - MainThread - awscli.arguments - DEBUG - Unpacked value of b'PK\x03\x04\x14\...'
2019-01-11 12:41:53,427 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-bucket: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,427 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,428 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-object-version: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,431 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.publish: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,432 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.dry-run: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,433 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.revision-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,433 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event calling-command.lambda.update-function-code: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x000001961E3E24A8>>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event calling-command.lambda.update-function-code: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x000001961E3E23C8>>
2019-01-11 12:41:53,434 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,434 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'us-east-1'.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2019-01-11 12:41:53,436 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2019-01-11 12:41:53,436 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2019-01-11 12:41:53,437 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2019-01-11 12:41:53,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\endpoints.json
2019-01-11 12:41:53,447 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,448 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x000001961D257730>
2019-01-11 12:41:53,450 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x000001961D19FF28>
2019-01-11 12:41:53,452 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2019-01-11 12:41:53,455 - MainThread - botocore.endpoint - DEBUG - Setting lambda timeout as (60, 60)
2019-01-11 12:41:53,457 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: lambda
2019-01-11 12:41:53,475 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.lambda.UpdateFunctionCode: calling handler <function generate_idempotent_uuid at 0x000001961D2E6AE8>
2019-01-11 12:41:53,607 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=UpdateFunctionCode) with params: {'url_path': '/2015-03-31/functions/###################################/code', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7'}, 'body': b'{"ZipFile": "UEsDBBQAAAAIABZ2Kk6MMc..."}', 'url': 'https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x000002CF16D57128>, 'has_streaming_input': False, 'auth_type': None}}
2019-01-11 12:43:18,141 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002CF16D57198>>
2019-01-11 12:43:18,148 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002CF14E669D8>
2019-01-11 12:43:18,193 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-11 12:43:18,193 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/###############################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190111T104318Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-11 12:43:18,194 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190111T104318Z
20190111/us-east-1/lambda/aws4_request
92c08cff0013f4216f1867da86748d396d5302483b73049c0a81c6356794ce6c
2019-01-11 12:43:18,195 - MainThread - botocore.auth - DEBUG - Signature:
######################################################
2019-01-11 12:43:18,195 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190111T104318Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=######################################################', 'Content-Length': '15340895'}>
2019-01-11 12:44:19,789 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.SSLError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 260, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
2019-01-11 12:44:19,816 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x000002CF15F580F0>
2019-01-11 12:44:19,817 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.SSLError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 260, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
2019-01-11 12:44:19,820 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.8963882468378392
2019-01-11 12:44:19,820 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.8963882468378392 seconds
2019-01-11 12:44:20,717 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002CF16D57198>>
2019-01-11 12:44:20,718 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002CF14E669D8>
2019-01-11 12:44:20,799 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-11 12:44:20,800 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/###############################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190111T104420Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-11 12:44:20,801 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190111T104420Z
20190111/us-east-1/lambda/aws4_request
87579068e9d38b28b9617bb66917d6b7c3f70ffe36f0356bdb252a87d0910c5c
2019-01-11 12:44:20,802 - MainThread - botocore.auth - DEBUG - Signature:
######################################################
2019-01-11 12:44:20,802 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190111T104420Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=######################################################', 'Content-Length': '15340895'}>

...

2019-01-11 12:48:36,132 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 344, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 516, in __call__
    call_parameters, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 636, in invoke
    client, operation_name, parameters, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 648, in _make_client_call
    **parameters)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\client.py", line 610, in _make_api_call
    operation_model, request_dict)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 136, in _send_request
    success_response, exception):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 205, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 273, in send
    endpoint_url=request.url
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code".
2019-01-11 12:48:36,151 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code".

Output from aws history:

AWS CLI command entered
at time: 2019-01-11 12:41:02.670
with AWS CLI version: aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7
with arguments: ['lambda', 'update-function-code', '--function-name', '####################################################', '--zip-file', 'fileb://API.zip', '--debug']

[0] API call made
at time: 2019-01-11 12:42:29.095
to service: lambda
using operation: UpdateFunctionCode
with parameters: {
    "FunctionName": "####################################################",
    "ZipFile": "<Byte sequence>"
}

[0] HTTP request sent
at time: 2019-01-11 12:43:18.195
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104318Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:44:20.802
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104420Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:45:24.212
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104524Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:46:29.277
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104629Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:47:34.508
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104734Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

AWS CLI command exited
at time: 2019-01-11 12:48:36.154
with return code: 255

I am still experiencing the problem today. Please advise further.

Edit: Updating to the latest version of aws cli (1.16.88) yields the same results.

[moldstadt]

I'm still experiencing this problem. What could the problem be?

I have tried re-uploading an unchanged project of about the same size, and that works fine.

[justnance]

@moldstadt - Thank you for your feedback and patience. Based on the debug logs and the history output it appears there is a network issue or timeout. Is there a difference in the network between the changed and unchanged project?

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[nosajio]

@moldstadt did you ever manage to resolve this? I'm a couple of hours into debugging the same issue 😣

Thanks!

[Fuiste]

I am having this issue as well when trying to deploy new lambda code.

[MimoMoreno]

I am having a similar issue when deploying to elastic beanstalk. I am working with PowerShell on Windows with the following version: EB CLI 3.14.13 (Python 3.6.2)

The repeated error is:
(DEBUG) ebcli.lib.aws : Making api call: (s3, upload_part) to region: us-west-2 with args: {...}

[nosajio]

Still having the same issue today. Other lambda commands are usable through the cli, but update-function-code continues to return the same error:

Connection was closed before we received a valid response from endpoint URL: "https://lambda.eu-west-2.amazonaws.com/2015-03-31/functions/test-fn/code".

I can confirm this error happens in all regions I've tested uploading code to (us-east-1, us-east-2, eu-west-2).

I've also reinstalled awscli with no change. current version: aws-cli/1.16.132 Python/3.7.2 Darwin/18.0.0 botocore/1.12.122

Hopefully this issue will be reopened as it seems like more people are experiencing this.

[Fuiste]

For what it's worth, my issue seemed to resolve itself after a few hours.

The command that finally worked was adding the --debug flag to the update function code command, but I'm fairly certain this had little to do with the resolution.

For posterity sake, these are the things I tried, which _did not_ solve the issue:

1. Running with a different Access Key/Secret
2. Running as a different IAM user
3. Running on a different network
4. Running on a newer version of the AWS CLI (1.16.130)
5. Restarting my computer

It is worth noting as well that my team was able to run the command (even using my IAM credentials) while this problem was occurring, so it's very likely to be an issue with the computer, and not the AWS CLI.

[moldstadt]

I'm still experiencing this problem. Some times it works, other times it
doesn't.

On Wed, 27 Mar 2019, 15:46 William Pelrine [e-mail] wrote:

For what it's worth, my issue seemed to resolve itself after a few hours.

The command that finally worked was adding the --debug flag to the update
function code command, but I'm fairly certain this had little to do with
the resolution.

For posterity sake, these are the things I tried, which did not solve
the issue:

1. Running with a different Access Key/Secret
2. Running as a different IAM user
3. Running on a different network
4. Running on a newer version of the AWS CLI (1.16.130)
5. Restarting my computer

It is worth noting as well that my team was able to run the command (even
using my IAM credentials) while this problem was occurring, so it's very
likely to be an issue with the computer, and not the AWS CLI.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-cli/issues/3842#issuecomment-477161637, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVoOPyfE30twp3uA1Vzo0SsaHfNd-6uWks5va3YhgaJpZM4Z5fYF
.

[inci90]

Also experiencing this problem

Have tried running with --debug flag to no effect.

@moldstadt are you a mac osx user? and did you install with homebrew?

[nosajio]

@moldstadt @inci90 and anyone else experiencing the above problem, I was able to resolve it by connecting to another wifi network.

This is weird but the upload always works with/without the --debug flag when I tether to my phone network. But for some reason it fails on my home network. I'm curious about what could be causing this...

[justnance]

It appears this issue is related to a local computer or network issue and not related to the CLI. I'm closing this issue for now but can reopen if there are specific steps we can do to reproduce the issue.

[inci90]

The issue has disappeared for me now. Can confirm it appears to be a network issue.

I moved to another wifi network and the issue fixed itself.

[justnance]

@inci90 - Thanks for the update.

[deepak-kapoor]

This is still an issue. I have changed wifi, re-generated access key and secret. It is now causing major issues for me as I cannot deploy functions. Any help will be appreciated.

[musahaidari]

I am also having the same issue, with a new key generated, on WSL ubuntu 18.04

[ychennay]

Same issue :(, only 7MB zip file being uploaded

[justnance]

@deepak-kapoor, @musahaidari, and @ychennay - Thanks for your feedback on this issue. We'd like to keep this issue closed as the original issue was confirmed as a network problem. Please open a new issue and provide steps we can use to reproduce the issue or provide a the output from the debug logs. Thanks.

[jemc]

the original issue was confirmed as a network problem

Nope, @moldstadt (the OP of the original issue) never confirmed that - you only asserted it in https://github.com/aws/aws-cli/issues/3842#issuecomment-485448934.

It appears this issue is related to a local computer or network issue and not related to the CLI

These two are not mutually exclusive. From my understanding of the situation, there is a network issue present and the CLI does not properly handle the network issues that arise. Bad/insufficient error handling absolutely counts as an issue with the CLI.

[rjamesnw]

I'm having this issue now. This worked both at my work site, and at home. Now it doesn't work anymore. It's not a size issue (and if so, I would hope AWS would report the error and not just ignore the request). I think a smarter CLI is needed here.

FWIW: I can only upload online (which takes a bit of time). The CLI is currently useless. The file size has not changed more than a few bytes since last time. My guess is the CLI tool is timing out too quickly, or AWS is way too slow at the moment.

[musahaidari]

I just noted that having stopped the antivirus does not work on WSL 1, it should be exited totally. I am using kaspersky internet security.

[ceoro9]

@rjamesnw @musahaidari turns out there are some problems with AWS network or servers, whatever. Some of my colleges experience the same problem on different ISPs, but all of our functions are resided in eu-cental-1 region, may be situation is different in the others.

[rjamesnw]

Yeah, after a couple hours it got much faster and the CLI was working quickly again. It doesn't matter now since I switch to using pipeline deployments.

[wreuvenspam]

I may just have got lucky but it worked for me after adding:

--cli-connect-timeout 6000

[TheoElia]

I think --cli-connect-timeout 6000 should now be --connection-timeout 6000 and that worked for me

[wayne-li2]

Just commenting here that --cli-connect-timeout 6000 worked for me.

[sanielfishawy]

It is clear from debug that for large zip files and/or a slow connection the connection times out before the zip file can be fully uploaded.

The user should not have to guess what is the proper setting for cli-connect-timeout the cli should not timeout a request that is actively uploading. So this is a bug in my view.

FWIW I believe the proper setting is --cli-connect-timeout 0 which should not timeout.

[bleon1]

--cli-connect-timeout 6000 did the trick for me too

[direbutterfly]

Feb'20: --cli-connect-timeout 6000 worked for me too

[pulidovpe]

Mar'20: --cli-connect-timeout 6000
worked for me too

[jgerardsimcock]

23 March 2020: --cli-connect-timeout 6000 is a good solution

[yasiumaster]

Check your antivirus is not blocking connection.

[gflex]

Issue imposed by antivirus software. In my case it's kaspersky. Once AV is quit, connection is flawless.

[wswardiii]

--cli-connect-timeout 6000 worked for me as well, thanks! Amazon should be more aware of that solution.

[yasiumaster]

--cli-connect-timeout 6000 worked for me as well, thanks! Amazon should be more aware of that solution.

This is not question for Amazon. This is a question of the network and/or potentially anything using the connection (like antivirus in my case).

[PranavMonotype]

Its can be a network error.
If you are connected to a VPN , try after disconnecting it
try with better internet connection
my case, closing the VPN worked

[AWS-Bassem]

check out this issue https://github.com/aws/sagemaker-python-sdk/issues/799

[prb407]

--cli-connect-timeout 6000 worked for me as well, thanks!

[redixhumayun]

Just commenting to say that none of the proposed solutions here worked for me.
My .zip file is 16.6 mb and adding the --cli-connect-timeout 6000 gave me a signature expired error.
Switching to a different network did nothing either.

Is there an official solution provided by the cli for this?

[shwetapurushe]

July 23rd 2020
Home network and VPN both didn't work. --cli-connect-timeout 6000 worked
function.zip was 38MB.

[wswardiii]

Just commenting to say that none of the proposed solutions here worked for me.
My .zip file is 16.6 mb and adding the --cli-connect-timeout 6000 gave me a signature expired error.
Switching to a different network did nothing either.

Is there an official solution provided by the cli for this?

I have had to retry the upload a number of times to get around this. I believe that one needs a fast enough upload speed to usually avoid this message. Otherwise, it is a persistent issue, because the signature seems to expire DURING the upload.

[mowliv]

Turning off Mac bitdefender AV didn't help but --cli-connect-timeout 6000 fixed it for me. File size only 10 MB, rural DSL internet cxn (15 down, 1 up).

[life-efficient]

Looks like just a timeout due to a slow network and relatively large upload.

[dskvr]

I'm having this issue as well, and none of the answers address my issue.

1. This started happening when I enabled accelerate on s3 buckets.
2. I have properly configured accelerate to true
3. This obscure error only occurs on Windows while using git-bash
4. The logic is ran in a matrix, macos and ubuntu run the same logic with no error.
5. The download fails at the exact same place every time, suggesting this is not a timeout isssue.
6. Unless buckets with accelerated transfers use Lambda on the backend, I fail to see how Lambda would have anything to do with this. Especially considering the exact same logic is used on two other platforms.

I will switch over the cmd for windows only, as it will probably fix the issue, but it makes everything a pain.

Always fails at roughly 42 MiB, short of the 50mb limit suggested above.

Completed 41.2 MiB/136.6 MiB (1.3 MiB/s) with 1 file(s) remaining   
upload failed: .\***.tar.gz Connection was closed before we received a valid response from endpoint URL: "***.tar.gz".
fatal error: An error occurred (404) when calling the HeadObject operation: Key "***.tar.gz" does not exist
Completed 41.5 MiB/136.6 MiB (1.3 MiB/s) with 1 file(s) remaining   

[mowliv]

In case it helps someone, I tried this and it works for when I'm in a rural area. I changed the python library code to retry several times. For some reason, in my rural location it always goes on the 4th retry. But when I was at a hotel with bad wifi, it still wouldn't go.

Change file /lib/python3.8/http/client.py - here's the whole send() method, with my change marked with a comment:

def send(self, data):
    """Send `data' to the server.
    ``data`` can be a string object, a bytes object, an array object, a
    file-like object that supports a .read() method, or an iterable object.
    """

    if self.sock is None:
        if self.auto_open:
            self.connect()
        else:
            raise NotConnected()

    if self.debuglevel > 0:
        print("send:", repr(data))
    if hasattr(data, "read") :
        if self.debuglevel > 0:
            print("sendIng a read()able")
        encode = self._is_textIO(data)
        if encode and self.debuglevel > 0:
            print("encoding file using iso-8859-1")
        while 1:
            datablock = data.read(self.blocksize)
            if not datablock:
                break
            if encode:
                datablock = datablock.encode("iso-8859-1")
            self.sock.sendall(datablock)
        return
    try:
        # mowliv Edit 10/31/20
        count = 0
        while True:
            try:
                self.sock.sendall(data)
                break
            except socket.timeout:
                count += 1
                if count >= 8:
                    raise
                else:
                    print(f"mowliv: ssl send - retry {count}")

    except TypeError:
        if isinstance(data, collections.abc.Iterable):
            for d in data:
                self.sock.sendall(d)
        else:
            raise TypeError("data should be a bytes-like object "
                            "or an iterable, got %r" % type(data))

[LarryKlugerDS]

--cli-connect-timeout 6000 also solved the problem for me

Home network with corp security proxy that gives a slow connect time.
22M zip payload

Version:
aws-cli/2.1.4 Python/3.7.4 Darwin/18.7.0 exe/x86_64

Other aws lambda commands work. Just aws lambda update-function-code times out.

[LarryKlugerDS]

Please upvote / comment on the feature request to make --cli-connect-timeout 6000 be the default:

See https://github.com/aws/aws-cli/issues/5754