Aws-sdk-net: Intermittent SSL handshake error when calling S3

Created on 17 Mar 2020  路  4Comments  路  Source: aws/aws-sdk-net

Current Behavior

We are experiencing intermittent errors ("SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN") when interacting with the S3 client from ECS in one of our applications. The error seems random and occasional, and it may occur while loading or getting objects from S3.

This is an example of an exception we got while getting an object from S3, right after loading it: 

{
    "message": "The SSL connection could not be established, see inner exception.",
    "data": {},
    "innerException": {
        "ClassName": "System.Security.Authentication.AuthenticationException",
        "Message": "Authentication failed, see inner exception.",
        "Data": null,
        "InnerException": {
            "message": "SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.",
            "data": {},
            "stackTrace": "   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
            \n   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)",
            "source": "System.Net.Security",
            "hResult": -2146233088
        },
        "HelpURL": null,
        "StackTraceString": "   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
        \n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
        \n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)",
        "RemoteStackTraceString": null,
        "RemoteStackIndex": 0,
        "ExceptionMethod": null,
        "HResult": -2146233087,
        "Source": "System.Private.CoreLib",
        "WatsonBuckets": null
    },
    "stackTrace": "   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
    \n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
    \n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
    \n   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
    \n   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at MyNameSpace.AwsS3Context.GetContentVersionId(Guid id) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.AwsS3Context.PutEncryptedContentAsync(Guid id, Stream content) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.ManagerClass.SaveContentAsync(Guid id, Stream content) in /filePath/ManagerClass.cs:line XX
    \n   at MyNameSpace.ControllerClass.CreateAsync() in /filePath/ControllerClass.cs:line XX
    \n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
    \n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
    \n   at MyNameSpace.APIClass.Startup.<Configure>b__8_1(HttpContext context, Func`1 next) in /filePath/Startup.cs:line XX
    \n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)",
    "source": "System.Net.Http",
    "hResult": -2146233087
}

S3 client is configured as below:

// At startup
services.AddAWSService<IAmazonS3>();

// S3 client class
public class AwsS3Context : IAwsS3Context
{
    private readonly IAmazonS3 awsS3Client;

    public AwsS3Context(IAmazonS3 awsS3Client)
    {
        this.awsS3Client = awsS3Client;
    }

    public async Task<string> GetContentVersionId(Guid id)
    {
        var getS3Request = new GetObjectRequest
                  {
            BucketName = "bucket_name",
            Key = id.ToString()
        };

        using (var s3Item = await awsS3Client.GetObjectAsync(getS3Request).ConfigureAwait(false))
        {
            return s3Item.VersionId;
        }
    } 

    public async Task<string> PutEncryptedContentAsync(Guid id, Stream content)
    {
        var config = new TransferUtilityConfig();

        try
        {
            using (var awsTransferUtility = new TransferUtility(awsS3Client, config))
            {
                var request = new TransferUtilityUploadRequest
                {
                    BucketName = "bucket_name",
                    Key = id.ToString(),
                    InputStream = content,
                    ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256 
                };

                await awsTransferUtility.UploadAsync(request, CancellationToken.None).ConfigureAwait(false);
                return await GetContentVersionId(Guid);
            }
        }
        finally
        {
            content?.Dispose();
        }
    }
}

Expected Behavior

Intermittent SSL handshake errors when calling S3 are not expected.

Steps to Reproduce (for bugs)

We are not able to reproduce this error consistently. In order to test, we tried isolating the application in a dev environment provisioning a single container and sending it a regular load for some days, adjusting the request rate and size. After this we got the mentioned exception a couple of times but root cause or conditions that produce the error are not evident.

Context

This is a simple API implemented with ECS that receive different types of files to internally save them in an S3 bucket, providing a resource id to its clients.

Your Environment

  • AWSSDK.Core version used: AWSSDK.Core (3.3.104.34)
  • Service assembly and version used: AWSSDK.S3 (3.3.110.32)
  • Targeted .NET platform: .Net Core 2.1 (Microsoft.NETCore.App 2.1.0)
  • Visual Studio version: VS 2019 (16.4.1)
  • Operating System and version: Alpine 3.11 (From image mcr.microsoft.com/dotnet/core/aspnet:2.1-alpine)

.NET Core Info

  • .NET Core version used for development: 2.1
  • .NET Core version installed in the environment where application runs: 2.1
A bug moduls3 queued
Source
picture AlfredoDiaz90
👍4

Most helpful comment

Have the same type of issue when working with Amazon.S3.Transfer.TransferUtility of AWSSDK.S3, v.3.3.110.50

picture GlebObodovsky on 28 Sep 2020
👍3

All 4 comments

Seeing the same problem intermittently while reading data from S3

GKrivosheev-rms picture GKrivosheev-rms on 13 Apr 2020

Have the same type of issue when working with Amazon.S3.Transfer.TransferUtility of AWSSDK.S3, v.3.3.110.50

GlebObodovsky picture GlebObodovsky on 28 Sep 2020
👍3

Getting the same error intermittently. Running on Lambda. 

Exception Message: SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.
Stack Trace:
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
  at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
  at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
  at internal method...
ymwymw picture ymwymw on 19 Nov 2020

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.

UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

samiechan picture samiechan on 19 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Technolink picture Technolink  路  4Comments
berkeleybross picture berkeleybross  路  3Comments
ShahriatHossain picture ShahriatHossain  路  4Comments
NinoFloris picture NinoFloris  路  4Comments
jc1001 picture jc1001  路  4Comments