Botocore: Add option to enable HTTP keep-alive

[horike37]

AFAIK, to enable HTTP keep-alive, you need to inject a custom header via the botocore event system.
https://github.com/boto/botocore/issues/902

However, it would be more user friendly if there is http_keepalive option to enable that.
What do you think? I will work on this if the botocore team thinks this is a reasonable approach.

[stealthycoin]

tcp_keepalive is already exposed. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html?highlight=tcp_keepalive#configuration-file

[horike37]

Thank you for the response @stealthycoin 😄
Yup, I know that option but that is to enable TCP keep alive. My suggestion is to enable HTTP keep alive.

[marcy-terui]

Hi @stealthycoin , Please let me add more information.

botocore isn't sending Connection: keep-alive in the HTTP request header. We will get the performance benefit by the header.
https://theburningmonk.com/2019/02/lambda-optimization-tip-enable-http-keep-alive/

[milancermak]

FWIW, you don't need to enable Connection: keep-alive to get the above mentioned performance benefit. I wrote about why that is so TL;DR is that DynamoDB keeps the connection open, but NodeJS explicitly sets a Connection: close header. In Python/boto, this is not an issue.

Regardless, I find the event system to be sufficient to manipulate the headers:

def set_connection_header(request, operation_name, **kwargs):
    request.headers['Connection'] = 'keep-alive'

ddb = boto3.client('dynamodb')
ddb.meta.events.register('request-created.dynamodb', set_connection_header)

[joguSD]

As @milancermak mentions connection reuse shouldn't be a problem, as it's the default in HTTP/1.1 and we use urllib3's connection pooling. The related issue #902 was using a non-AWS product on localhost to serve an S3 compatible API which that closed the connection by default, which is why sending the header was helpful to them. Adding the header shouldn't be needed and if there is a case where it's required I think the event to inject the header is sufficient.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[vzf146]

https://github.com/aws/aws-sdk-js/blob/master/CHANGELOG.md#24630