Aws-sdk-net: SQSclient http issue under highload

[Lapeno94]

SQS client cause socket exceptions under high load in .net core.

Expected Behavior

Sqsclient should able to manage the httpclient lifetime with the applications.

Resused the httpclients and not consume all of the ports. 8124

Current Behavior

Clientconfig httpclient factory is null. Therefore no disposable httpclients. Cache is increasing to the maximum.

Possible Solution

    public class SqsHttpClientFactory : Amazon.Runtime.HttpClientFactory
    {
        private readonly IHttpClientFactory _httpClientFactory;

        public SqsHttpClientFactory(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }

        public override HttpClient CreateHttpClient(IClientConfig clientConfig)
        {
            return _httpClientFactory.CreateClient();
        }
    }

            services.AddAWSService<IAmazonSQS>();
            services.AddSingleton<IAmazonSQS>(x => new AmazonSQSClient(new AmazonSQSConfig
            {
                HttpClientFactory = new SqsHttpClientFactory(x.GetRequiredService<IHttpClientFactory>())
            }));

Steps to Reproduce (for bugs)

Create ecs fargate cluster. Run a task where the container runs a .net core app on linux.

Context

Services goes against high load. Not supports the SLAs which are defined for sqs.

Are you sure about the http a best protocol for messaging?

Your Environment

• AWSSDK.Core version used: newest
  Ecs Fargate via linux containers.

.NET Core Info

• .NET Core version used for development: 2.2
• .NET Core version installed in the environment where application runs: 2.2

[klaytaybai]

What do you have set in the client config around this topic? Is CacheHttpClient true?

[Lapeno94]

Yes I am using .net core. I think the problem could be the targeting. (Using .net standart 1.3 and not the 2.0 httpclient)
public abstract class HttpClientFactory
{
///

/// Create and configure an HttpClient.
///

///
public abstract HttpClient CreateHttpClient(IClientConfig clientConfig);

/// <summary> /// <para> /// This is a switch used for performance testing and is not intended for production applications /// to change. This switch may be removed in a future version of the SDK as the .NET Core platform matures. /// </para> /// <para> /// If true, the HttpClient is cached and reused for every request made by the service client /// and shared with other service clients. /// </para> /// <para> /// For the .NET Core platform this is default to true because the HttpClient manages the connection /// pool. /// </para> /// </summary> bool CacheHttpClient { get; }

[chillbilly79]

I am also having this issue. Is there a suggested way to use the SQSClient without consuming all the connections?

[Lapeno94]

I spoke with the aws support through my company, and they wont solve it,bc there is workaround for it.
What I mentioned above. SqsHttpClientFactory with asp.net core httpclientfactory
pros.: solve the issue
cons: asyncblock the client (if u want to consume and produce) So everything which is could be event-sourcing

https://github.com/aws/aws-sdk-net/issues/565

Implementation guide:
https://www.stevejgordon.co.uk/httpclient-connection-pooling-in-dotnet-core

same issues:
https://github.com/aws/aws-sdk-net/issues/1339
https://github.com/aws/aws-sdk-net/issues/971