Aws-sdk-net: GetObjectAsync throws System.IO.Exception (and other exceptions) under load

[bkrchristopher]

We are doing a lot of S3 object reads concurrently using GetObjectAsync and experiencing intermittent exceptions for closed connection or disposed objects. This problem does not occur when we use the synchronous version of the call. I have included a code snippit below that can be run as a console application that will reproduce the problem. All you need for setup is an S3 bucket with a single small file to read.

When run in sync mode, this code will NOT experience any errors calling GetObject. Running in async mode, the WILL experience dropped connection errors and sometimes Disposed Object exceptions when calling the GetObjectAsync methods.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Amazon;
using Amazon.S3;

namespace S3StressTest
{
    class Program
    {
        static AmazonS3Client _amazonS3Client;
        private static long _batchNumber = 0;
        const string BucketName = "dev-sample-data";

        static void Main(string[] args)
        {
            string fileKey = "txt/DeleteMe.txt";

            var doSync = args.Any(i => i == "sync");

            _amazonS3Client = new AmazonS3Client(RegionEndpoint.USWest2);

            GetBatches(fileKey, doSync).Wait();
        }

        private static async Task GetBatches(string fileKey, bool doSync)
        {
            int maxBatches = 26;
            List<Task> tasks = new List<Task>();
            while (true)
            {
                var newTasks = Enumerable
                    .Range(0, Math.Min(maxBatches - tasks.Count, maxBatches))
                    .Select(s => ReadABatch(fileKey, doSync))
                    .ToArray();
                tasks.AddRange(newTasks);
                await Task.WhenAny(tasks);
                tasks = tasks.Where(s =>
                    s.Status != TaskStatus.RanToCompletion && 
                    s.Status != TaskStatus.Faulted &&
                    s.Status != TaskStatus.Canceled
                    ).ToList();
            }
        }

        public static async Task<string> ReadABatch(string fileKey, bool sync)
        {
            var batchNumber = Interlocked.Increment(ref _batchNumber);
            Task<string>[] sources = null;

            if (sync)
            {
                sources = Enumerable.Range(0, 15).Select(s => Task.Run(() => ReadFileAsString(fileKey))).ToArray();
            }
            else
            {
                sources = Enumerable.Range(0, 15).Select(s => ReadFileAsStringAsync(fileKey)).ToArray();
            }

            await Task.WhenAll(sources);

            if (batchNumber%1000 == 0)
            {
                Console.WriteLine($"Batch {batchNumber} completed.");
            }

            return "";
        }

        public static string ReadFileAsString(string fileKey)
        {
            try
            {
               var response = _amazonS3Client.GetObject(BucketName, fileKey);
                using(response)
                {
                    using (var reader = new StreamReader(response.ResponseStream))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
            catch (Exception e)
            {
                var type = e.GetType();
                Console.WriteLine($"Exception is of type {type.FullName}");
            }
            return null;
        }

        public static async Task<string> ReadFileAsStringAsync(string fileKey)
        {
            try
            {
              var response = await _amazonS3Client.GetObjectAsync(BucketName, fileKey);
                using (response)
                {
                    using (var reader = new StreamReader(response.ResponseStream))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
            catch (Exception e)
            {
                var type = e.GetType();
                Console.WriteLine($"Exception is of type {type.FullName}");
            }
            return null;
        }
    }
}

[sstevenkang]

Thanks for reporting this. We noticed that S3 service is dropping TLS connections which surfaces as IOException. This doesn't seem to be .NET SDK specific issue. We think this is service-side issue and have a research task assigned to the team.

[rgmills]

@sstevenkang has there been any updates around this? Something about this feels similar to what I'm seeing under some load, but there isn't an IOException being thrown instead it's a SocketException.

An unhandled exception was thrown by the application.
System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Amazon.Runtime.Internal.Util.CachingWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Amazon.Runtime.Internal.Util.HashStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.<>c.<BeginReadInternal>b__39_0(Object )
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()

In this scenario, taking the resulting stream from GetObjectAsync and handing off to ASP.NET Core to stream the response to the API caller. If this feels like a separate issue, I'd be happy to open up a new one.

[rgmills]

@normj @sstevenkang any idea if there is an update around this? We're continuing to see these sort of failures under load. Any help would be appreciated.

[sstevenkang]

Hi, I apologize for the silence. I took a second look at this. Our SDK out of the box does multiple retries when it detects unstable network condition. However, because the example code you provided access the response stream directly, the SDK doesn't get a chance to handle the exception and the onus is on the client code to retry in case of socket errors.

I've ran the above code multiple times and the only exception I am seeing is IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. I also get the same exception when running sync calls.

And, regarding the An established connection was aborted by the software in your host machine. exception. There's nothing the SDK can do when some other process/thread in your environment forcefully kills the connection. Generally, this happens due to firewalls/AV software.

[sstevenkang]

Closing due to inactivity.

[VladimirTyrin]

Having the same problem now (i'm using version 3.3.17.4 with .Net 4.5.2).
There is around 0.8% error rate on 5-20 simultaneous requests (rate is basically constant). All errors are IOException's

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

There are no specific firewall settings and the problem can be reproduced with Windows Defender turned off. Problem occures on both EU and US buckets, BUT: only on recently created ones (around 1 month), but not on old ones (4 years). Every connection loss happens after HTTP headers being read (during body streaming).

Any updates on this issue?

[sachinpachori]

I am facing the same error when trying to upload 200000 files using latest S3 AWSSDK.NET. I am using async Upload method. Any resolution to this problem?

[rajeshlh]

I have been asked to provide S3 RequestID for AWS Support to troubleshoot intermittent exceptions when reading an object from S3 using the .NET SDK call GetObjectAsync. The exception is below:
System.Net.Sockets.SocketException: Connection reset by peer
How do I get Request ID consistently for such S3 failures that result in IO Exception? I understand that the Request ID is a 2 piece info. The catch block for AmazonS3Exception doesn't work since the exception thrown is of type SocketException. So I cannot use the RequestID or the ResponseHeaders property of the AmazonS3Exception class. Please note that I cannot reproduce these IO Exception on demand..its intermittent in production.