Aws-sdk-php: random cURL errors on HTTPS requests to SWF and DynamoDB

[ppaulis]

Hello!

we recently switched on our EC2 instances from the v2 to v3 (most recent version) of the PHP SDK. We use SWF, DynamoDB and S3. Since we switched to the v3, we are facing cURL errors that appear completely randomly it seems:

The application has thrown an exception!
Aws\Swf\Exception\SwfException
Error executing "PollForActivityTask" on "https://swf.us-east-1.amazonaws.com"; AWS HTTP error: cURL error 56: SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Sometimes the exception appears twice per hour, sometimes once in three hours... there's no scheme.

We already switched back to an older AMI and updated cURL from 7.35.0 (there is a problem with chunked upload in the 7.35) to 7.36.0 but nothing helps. We are polling SWF for activity tasks with long polling enabled, so there shouldn't be too many requests.

I googled this of course before opening an issue, but the only topics I found either date from 2012-2013 or were related to a broken load balancer on AWS. And that would probably be too much of a coincidence in our case...

Does anyone know about this problem..? I can't even say if it's a problem in the SDK or something else. So I'm grateful for every hint you can give me!

Thanks a lot!
Pascal

[jeskew]

It looks like the connection is being reset while OpenSSL is attempting to decrypt data, which can be caused by the client and server being unable to agree on a TLS protocol. Which version of OpenSSL are you using?

OpenSSL emits error 104 (ECONNRESET) when the connection is reset by a peer host, and this error is happening while cURL is attempting to read data. cURL is therefore surfacing this as a read error (cURL error 56 - CURLE_RECV_ERROR).

I'll look into why the SDK's behavior in response to this event would differ from v2 to v3.

[ppaulis]

Hey @jeskew ,

thanks for the quick reply! What I have tried so far:

• using the standard setting without specifying a SSL/TLS version
• I tried with TLS 1.2:

``` return
[
'aws' =>
[
'region' => 'us-east-1',
'curl.options' => [
'CURLOPT_SSLVERSION' => 'CURL_SSLVERSION_TLSv1_2'
]
]
]

or by directly using the value of the constant:

'CURLOPT_SSLVERSION' => 6
```

It's a ZF2 application, so this code is in a _aws.local.php_ file in my application configuration. I'll try to put it directly in the creation of the SwfClient.

I tried, so far, OpenSSL 1.0.1f and 1.0.2g. Both with the same result.

Thanks for your help!
Pascal

[ppaulis]

I also tried using the CA cert bundle from the curl website by specifying it in the php.ini with _openssl.cafile_, no luck :-/

[ppaulis]

a few additional infos:

ubuntu@ip-10-0-11-226:/tmp$ curl -V
curl 7.36.0 (x86_64-pc-linux-gnu) libcurl/7.36.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

and here's the output of a connection attempt to the SWF endpoint:

ubuntu@ip-10-0-11-226:/tmp$ openssl s_client -connect "swf.us-west-2.amazonaws.com:443"
CONNECTED(00000003)
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=swf.us-west-2.amazonaws.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5

Server certificate
-----BEGIN CERTIFICATE-----
(removed for the sake of readability)
-----END CERTIFICATE-----
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=swf.us-west-2.amazonaws.com
issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
---
No client certificate CA names sent
---
SSL handshake has read 2762 bytes and written 621 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: DDAB0212D8D9B49556328037D194D03EB7F22E50B0548E447B015F024874963C
    Session-ID-ctx: 
    Master-Key: D1968248414FFB3CB5AE009DB37E5836837F8257F1540170936EDC45BDDC3CFC60ECE93168D9D15D8B89765D917B1782
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1457086840
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

When I run instead:

openssl s_client -connect "swf.us-west-2.amazonaws.com:443" -CAfile /tmp/cacert.pem

I get a return code 0:

ubuntu@ip-10-0-11-226:/tmp$ openssl s_client -connect "swf.us-west-2.amazonaws.com:443" -CAfile /tmp/cacert.pem 
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = "Amazon.com, Inc.", CN = swf.us-west-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=swf.us-west-2.amazonaws.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=swf.us-west-2.amazonaws.com
issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
---
No client certificate CA names sent
---
SSL handshake has read 2762 bytes and written 621 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: D908CDB1F5DD27E4281543407894B92FA33A181172068DA98FCAD4CF77E2963C
    Session-ID-ctx: 
    Master-Key: 658CED64C142D37CCCAC7F2163F707357D071894874288AEF5B364DD81A533A88B111BB68059F9C8073A47F86EA212BC
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1457087123
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[ppaulis]

btw, the problem appears both on us-east-1 and us-west-2

[jeskew]

Is the CA bundle at /tmp/cacert.pem the bundle provided on the curl website?

[ppaulis]

@jeskew yes, it is.

[ppaulis]

I have currently running a few machines with an older version of our application, that uses the v2 of the SDK. To double check, that the exception doesn't appear on this version. I'll keep you updated!

[ppaulis]

@jeskew The result is as expected. The same machines with the old application version (sdk v2.8.18, guzzle 3.7.1) don't produce the exceptions.

[jeskew]

v2 of the SDK uses Guzzle 3, which includes its own certificate authority bundle. v3 uses Guzzle 5/6, which rely on an external bundle.

Can you set the openssl.cafile ini setting and then check what is output by running openssl_get_cert_locations()? (This function is only defined in PHP >= 5.6.) Since running curl on the command line fails unless you specify a CAFile option, this will need to be set in PHP.

[ppaulis]

@jeskew without specifying the the openssl.cafile I get:

array(8) {
  ["default_cert_file"]=>
  string(21) "/usr/lib/ssl/cert.pem"
  ["default_cert_file_env"]=>
  string(13) "SSL_CERT_FILE"
  ["default_cert_dir"]=>
  string(18) "/usr/lib/ssl/certs"
  ["default_cert_dir_env"]=>
  string(12) "SSL_CERT_DIR"
  ["default_private_dir"]=>
  string(20) "/usr/lib/ssl/private"
  ["default_default_cert_area"]=>
  string(12) "/usr/lib/ssl"
  ["ini_cafile"]=>
  string(0) ""
  ["ini_capath"]=>
  string(0) ""
}

and after specifying openssl.cafile in the php.ini, I get:

array(8) {
  ["default_cert_file"]=>
  string(21) "/usr/lib/ssl/cert.pem"
  ["default_cert_file_env"]=>
  string(13) "SSL_CERT_FILE"
  ["default_cert_dir"]=>
  string(18) "/usr/lib/ssl/certs"
  ["default_cert_dir_env"]=>
  string(12) "SSL_CERT_DIR"
  ["default_private_dir"]=>
  string(20) "/usr/lib/ssl/private"
  ["default_default_cert_area"]=>
  string(12) "/usr/lib/ssl"
  ["ini_cafile"]=>
  string(15) "/tmp/cacert.pem"
  ["ini_capath"]=>
  string(0) ""
}

I'm currently using PHP 5.6.18.

I already tested this configuration, with the CA bundle specified in the php.ini. The exceptions keep appearing :-/

[jeskew]

Is it possible that the processes that terminate following an OpenSSL error are either setting the SSL_CERT_FILE environment variable or unsetting the openssl.cafile ini variable? Clearly OpenSSL/cURL is loading this ini value (the equivalent of the command-line CAfile option) normally, but as you mentioned, the error you're seeing is occurring in a random minority of cases.

v2 of the SDK uses Guzzle 3, which will use a vendored CA bundle by default, whereas v3 (via Guzzle 5/6) will use PHP, OS, and OpenSSL configuration as its default. You can override this on a client by specifying the verify HTTP option. For example:

$dynamoClient = new \Aws\DynamoDb\DynamoDbClient([
    'version' => 'latest',
    'region' => 'us-west-2',
    'http' => [
        'verify' => '/tmp/cacert.pem',
    ],
]);

[ppaulis]

We never handled the SSL_CERT_FILE or openssl.cafile manually so far, because guzzle v3 handled this perfectly itself.

I'll give it a try with the http option and keep you updated!

Thank you, Jonathan, for your time!

[ppaulis]

I specified the ca bundle now manually in the php code:

$aws->createSwf([
            'version' => '2012-01-25',
            'http' => [
                'verify' => '/tmp/cacert.pem',
            ]
        ]);

the region, domain, etc. is loaded via the ZF2 config files.

[jeskew]

One last question: are you connecting to AWS through any kind of network proxy? A similar error message was reported on the Docker repository: https://github.com/docker/docker/issues/2011

They mention that this can happen to traffic from a Docker container, and one comment mentions that they saw a similar error when using a VPN.

[ppaulis]

Hmm, we are using simple EC2 instances with Ubuntu in a VPC on AWS. The outgoing traffic passes through a NAT (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html). But I'm not sure if the connection to the AWS services like SWF remain in the internal network... I'll check that with a collegue and get back to you.

Thanks!

[ppaulis]

@jeskew In confirm that connections to SWF go through the NAT

[ppaulis]

@jeskew The 'verify' => '/tmp/cacert.pem' unfortunately doesn't help either :-/

[jeskew]

Since neither specifying an openssl.cafile ini variable nor a verify http option worked, the error is not related to the CA bundle. One other change from Guzzle 3 to Guzzle 6 is that Guzzle now reuses cURL handles between requests. There's an open pull request on OpenSSL that would remedy a failure case where you connect to a pool of hosts (the fleet addressable as swf.us-west-2.amazonaws.com, for example) and negotiate an SSL session, save that session, and then use the session to connect to a different host in the same pool. If the second host only supports TLS 1.0 and the first host supports TLS 1.2, the session will not downgrade but will instead error out. There was a lot of discussion on the JavaScript SDK about how this issue was affecting DynamoDB customers.

Could you try forcing TLS 1.0? In v3, you would do so like this:

$ddb = new \Aws\DynamoDb\DynamoDbClient([
    ...
    'http' => [
        'curl' => [
            CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_0,
        ]
    ]
]);

OpenSSL is working on a fix but it has not yet been accepted and released.

[ppaulis]

it's in place:

protected function getClient()
    {
        /** @var \Aws\Sdk $aws */
        $aws = $this->getServiceLocator()->get(Sdk::class);

        return $aws->createSwf([
            'version' => '2012-01-25',
            'http' => [
                'curl' => [
                    CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_0,
                ]
            ]
        ]);
    }

I'll keep you updated! Thanks!

[ppaulis]

This sounds like a very serious lead.. It would explain the fact that the error is completely random.

[ppaulis]

Got again an exception, but I'm trying now with:

protected function getClient()
    {
        /** @var \Aws\Sdk $aws */
        $aws = $this->getServiceLocator()->get(Sdk::class);

        return $aws->createSwf([
            'version' => '2012-01-25',
            'curl.options' => [
                CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_0,
            ]
        ]);
    }

I think I've seen that somewhere in the docs...

[jeskew]

curl.options will only be used by a v2 client. Did you get the same cURL/OpenSSL error codes (cURL error 56: SSL read: error:00000000:lib(0):func(0):reason(0), errno 104)?

[ppaulis]

Yes I still get them :-/

Sent from my Phone
Am 08.03.2016 7:03 nachm. schrieb "Jonathan Eskew" <[email protected]

:

curl.options will only be used by a v2 client. Did you get the same
cURL/OpenSSL error codes (cURL error 56: SSL read:
error:00000000:lib(0):func(0):reason(0), errno 104)?

—
Reply to this email directly or view it on GitHub
https://github.com/aws/aws-sdk-php/issues/924#issuecomment-193894388.

[ppaulis]

@jeskew just scanned multiple times the SWF endpoint in us-east-1:

https://www.ssllabs.com/ssltest/analyze.html?d=swf.us-east-1.amazonaws.com

On different IPs, the result seems always the same. TLS 1.2 not supported, but TLS 1.0 is.

[ppaulis]

I just tried to force TLS 1.2, and indeed, I get the following:

GuzzleHttp\Exception\ConnectException
 cURL error 35: error:14077102:SSL routines:SSL23_GET_SERVER_HELLO:unsupported protocol (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

[jeskew]

The root cause might be something else that's supported by one server and not another. I'm still unable to reproduce the issue, so if you can capture any more context about a failing request that would be very helpful.

In the meantime, could you verify that this is related to sharing cURL handles? You can disable handle sharing by creating a custom Guzzle client like so:

use Aws\Handler\GuzzleV6\GuzzleHandler;
use Aws\Swf\SwfClient;
use GuzzleHttp\Client;
use GuzzleHttp\Handler\CurlFactory;
use GuzzleHttp\Handler\CurlHandler;
use GuzzleHttp\Handler\CurlMultiHandler;
use GuzzleHttp\Handler\Proxy;
use GuzzleHttp\HandlerStack;

// Create a Guzzle client that will not share cURL handles
$guzzleClient = new Client([
    'handler' => HandlerStack::create(Proxy::wrapSync(
        new CurlMultiHandler(['handle_factory' => new CurlFactory(0)]),
        new CurlHandler(['handle_factory' => new CurlFactory(0)])
    ))
]);

// Use the no-shared handle client to create an AWS client
$swfClient = new SwfClient([
    'region' => 'us-east-1',
    'version' => 'latest',
    'http_handler' => new GuzzleHandler($guzzleClient),
]);

[ppaulis]

I'm still getting the exceptions, but I'll check if I can get more verbose output from curl about this.

Thanks!
Pascal

[ppaulis]

I just added:

sudo ssldump -a -A -H -i eth0 > /tmp/ssldump.txt &

I'll get back to you in a few hours!

Thanks!

[ppaulis]

@jeskew I'm getting this in my ssldump output:

          a0 22 63 89 f1 ba 0f 6f 29 63 66 2d 3f ac 8c 72 
          c5 fb c7 e4 d4 0f f2 3b 4f 8c 29 c7 
      ServerHelloDone
80 3  0.0060 (0.0009)  C>SV3.1(262)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[256]=
          1d eb bc eb 0e 84 8d a2 db 4b db 39 23 e6 a2 b2 
          <...........................................>
          8a 32 21 3a 43 73 4b ef 52 cd 27 24 1f 9a 31 17 
          3c b7 8eERROR: Length mismatch
 53 eb 71 68 38 86 26 ce 43 4c 84 ba 66 
          6f 9e a4 da ff d9 b2 21 0c 25 56 0e 61 21 cd 70 
          ad f7 fa cb da 57 dd d3 79 d6 b0 e9 0a 1a 75 58 
          05 09 95 e7 85 d4 f4 ee 6c ba 36 38 1e 51 ca 47 
80 4  0.0060 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
80 5  0.0060 (0.0000)  C>SV3.1(48)  Handshake
80 6  0.0110 (0.0050)  S>CV3.1(1)  ChangeCipherSpec
80 7  0.0110 (0.0000)  S>CV3.1(48)  Handshake
80 8  0.0118 (0.0007)  C>SV3.1(32)  application_data
80 9  0.0119 (0.0000)  C>SV3.1(1520)  application_data
80 10 0.0140 (0.0021)  C>SV3.1(32)  application_data
80 11 0.0140 (0.0000)  C>SV3.1(96)  application_data

Googling this error often showed me a problem with _tcpdump_ where you have to specify the packet size with -s. But for _ssldump_, there is no such option. So I'm wondering if this could mean something.

[jeskew]

@ppaulis Just wanted to give you an update. I've engaged the SWF team on this and will keep you posted.

[ppaulis]

@jeskew Thank you very much!

[oberman]

I've been debugging a similar issue and didn't see this thread until just now. I'm looking at my logging to see if I can add anything. For me, it's SQS in us-east. It's _extremely_ transient for me. I had it happen 5 times on 3/22/16 but nothing yesterday.

Here is one of my failed requests, which was reported up the stack as "AWS HTTP error: cURL error 56: TCP connection reset by peer":

* Hostname sqs.us-east-1.amazonaws.com was found in DNS cache
*   Trying 54.239.27.139...
* Connected to sqs.us-east-1.amazonaws.com (54.239.27.139) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*       subject: CN=queue.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
*       start date: Sep 25 00:00:00 2015 GMT
*       expire date: Dec 23 23:59:59 2016 GMT
*       common name: queue.amazonaws.com
*       issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> POST /825286309336/attributeUpdateProd HTTP/1.1
Host: sqs.us-east-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20160322T064347Z
Authorization: AWS4-HMAC-SHA256 Credential=[KEY]/20160322/us-east-1/sqs/aws4_request, SignedHeaders=host;x-amz-date, Signature=[SIGNATURE]
User-Agent: aws-sdk-php/3.15.2 GuzzleHttp/6.1.1 curl/7.40.0 PHP/5.5.31
Content-Length: 894

* upload completely sent off: 894 out of 894 bytes
* SSL read: errno -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0

I haven't changed any curl reuse or certificates defaults. I only changed the connect timeout to be smaller, and enabled debug logging.

[ppaulis]

@oberman Thanks a lot for sharing! I'm happy for every bit of information you can send me :)

[oberman]

Well, one thing I can share/ask: are you tracking internal retries of the API?

To log retries you can use the Middleware hooks. I also write to a rolling buffer in the hook to be retroactively logged if things are taking too long. That's what the whole "getGuzzleSlowRequestPlugin" is all about. In fact, I was really debugging slow requests and happened upon this curl error 56 issue. For me, I see around 1 in 10,000 requests to SQS that are >= 3 seconds.

My Middleware hook looks like this:

 private static function getClient() {
        static $sqs = null;
        if ($sqs == null) {
            try {
                $sqs = Aws::sdk()->createSqs([
                    'debug' => [
                        'logfn' => function ($msg) {
                            self::getGuzzleSlowRequestPlugin()->log($msg);
                        },
                        'stream_size' => 0,
                        'http' => true,
                    ],
                    'http' => [
                        'on_stats' => function ($stats) {
                            $handlerStatsMessage =
                                "HandlerErrorData: " . print_r($stats->getHandlerErrorData(), true) . "\n"
                                . "HandlerStats: " . print_r($stats->getHandlerStats(), true);
                            self::getGuzzleSlowRequestPlugin()->setLastHandlerStats($handlerStatsMessage);
                            self::getGuzzleSlowRequestPlugin()->log($handlerStatsMessage);
                        },
                    ],
                ]);
                $sqs->getHandlerList()
                    ->after('retry', '', function (callable $handler) {
                        return function (
                            \Aws\CommandInterface $command,
                            \Psr\Http\Message\RequestInterface $request
                        ) use ($handler) {
                            return $handler($command, $request)
                                ->then(
                                    function ($result) {
                                        //onFulfilled, I can peek at the success response here
                                        return $result;
                                    },
                                    function ($value) {
                                        //onRejected, probably an amazon exception
                                        if ($value instanceof AwsException) {
                                            $requestId = $value->getAwsRequestId();
                                            $errorCode = $value->getAwsErrorCode();
                                            $errorType = $value->getAwsErrorType();
                                            $statusCode = $value->getStatusCode();
                                            $connectionError = $value->isConnectionError();
                                            $timeSinceStart = -1;
                                            if (isset($_SERVER["REQUEST_TIME_FLOAT"])) {
                                                $timeSinceStart = microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"];
                                            }
                                            Logger::getLogger(__CLASS__)->error("sqs retry requestId[$requestId]"
                                                . " errorCode[$errorCode] errorType[$errorType] statusCode[$statusCode]"
                                                . " connectionError[$connectionError] timeSinceStart[$timeSinceStart]"
                                                . " lastHandlerStats[" . print_r(self::getGuzzleSlowRequestPlugin()->getLastHandlerStats(), true) . "]");
                                        }
                                        return \GuzzleHttp\Promise\rejection_for($value);
                                    }
                                );
                        };
                    });
            } catch (\Exception $e) {
                throw new AmazonException("Cannot get the SQS client.", $e);
            }
        }
        return $sqs;
    } // getClient 

[serkanozcelik]

Randomly receive this error when issuing a create snapshots for EC2 with using sdk v3. This may be the same or related issue.

createSnapshot:Error executing "CreateSnapshot" on "https://ec2.eu-west-1.amazonaws.com"; AWS HTTP error: cURL error 35: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

AWS PHP SDK v3 ## 3.18.1 - 2016-04-21
Source : AWS eu-west region - Ireland
PHP 5.5.9-1ubuntu4.16
curl 7.35.0 (x86_64-pc-linux-gnu)
libcurl/7.35.0
OpenSSL/1.0.1f

I have tried;
'http' => [ 'curl' => [ CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_0, ] ]

but no chance, not affected.

[ParadoxNL]

I have the exact problem, except for the SQS service, where it fails on the "Receive Message" part.

[2016-05-02 13:31:52] local.ERROR: exception 'Aws\Sqs\Exception\SqsException' with message 'Error executing "ReceiveMessage" on "https://sqs.eu-central-1.amazonaws.com/id/queue"; AWS HTTP error: cURL error 35: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)'
GuzzleHttp\Exception\ConnectException: cURL error 35: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /project/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:186

Source:
PHP 7.0.5
curl 7.35.0
OpenSSL 1.0.1f

[jeskew]

@serkanozcelik and @ParadoxNL cURL error 35 is retried by the SDK. If it's being surfaced in your application, all retries must have been exhausted.

[ppaulis]

got another random error:

Part 14: Error executing "UploadPart" on "https://s3-us-west-2.amazonaws.com/.........."; AWS HTTP error: cURL error 52: Empty reply from server (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

[senorbacon]

We're seeing this problem as well, and like @ppaulis, we only started seeing when we upgraded from PHP SDK 2 to 3. It is also fairly random for us as well. @jeskew has the SWF team made any headway with this?

[hanoii]

I think I could be facing the same issue, only that I am using the sdk v2 and attempting to do a simple s3 operation. I can't get to pin down the issue. At one point I disabled curl ssl_verifypeer and curl_verifyhost and it worked but I think it was by chance, and similar to the random behaviour explained here, as it stopped working at some point even with that.

I am behind a proxy and am pretty sure this has something do, although it worked OK before and it seems to be working OK very randomly.

[ppaulis]

@hanoii Do you have a more specific error message that you can share with us?

[hanoii]

@ppaulis sorry for the poorly written comment and with lack of details. I edited the previous comment a bit and as far as a specific error message, unfortunately there's not much I have to share.

The error message is Aws\Common\Exception\TransferException: [curl] 28: Operation timed out after 0 milliseconds with 0 out of 0 bytes received [url] https://xxxxxx in Aws\Common\Client\AbstractClient->send() (line 258 de .../aws-sdk-php/Aws/Common/Client/AbstractClient.php).

This is a module within a web app, I attempted several things to obtain more output, adding a LogPlugin didn't help at all. What I ended up doing, besides adding debug => true to both the config, request.options and curl.options was also manually editing CurlHandle.php and changing php://temp of the verbose output to php://output and this is what I get:

BTW, I ended up removing the proxy as it turned out that I have direct access now, and that made it work, but to provide you with the following, I added the proxy back again, and it worked 4-5 times and then throw the following (I think I stripped most of the details):

* Rebuilt URL to: https://x.y.z/?prefix=somedirectory%2F
* Hostname was NOT found in DNS cache
*   Trying x.x.x.x...
* Connected to pro.xy (x.x.x.x) port 8080 (#0)
* Establish HTTP proxy tunnel to x.y.z:443
> CONNECT x.y.z:443 HTTP/1.1
Proxy-Connection: Keep-Alive
Host: x.y.z
User-Agent: aws-sdk-php2/2.8.30 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17 ITR
Date: Fri, 03 Jun 2016 21:18:44 +0000
Authorization: AWS ******

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: ...//sites/all/libraries/aws-sdk-php/Guzzle/Http/Resources/cacert.pem
  CApath: /etc/ssl/certs
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 0
* Rebuilt URL to: https://x.y.z/?prefix=somedirectory%2F
* Hostname was found in DNS cache
*   Trying x.x.x.x...
* Connected to pro.xy (x.x.x.x) port 8080 (#1)
* Establish HTTP proxy tunnel to x.y.z:443
> CONNECT x.y.z:443 HTTP/1.1
Proxy-Connection: Keep-Alive
Host: x.y.z
User-Agent: aws-sdk-php2/2.8.30 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17 ITR
Date: Fri, 03 Jun 2016 21:18:50 +0000
Authorization: AWS ******

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: ...//sites/all/libraries/aws-sdk-php/Guzzle/Http/Resources/cacert.pem
  CApath: /etc/ssl/certs
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 1
* Rebuilt URL to: https://x.y.z/?prefix=somedirectory%2F
* Hostname was found in DNS cache
*   Trying x.x.x.x...
* Connected to pro.xy (x.x.x.x) port 8080 (#2)
* Establish HTTP proxy tunnel to x.y.z:443
> CONNECT x.y.z:443 HTTP/1.1
Proxy-Connection: Keep-Alive
Host: x.y.z
User-Agent: aws-sdk-php2/2.8.30 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17 ITR
Date: Fri, 03 Jun 2016 21:18:56 +0000
Authorization: AWS ******

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: ...//sites/all/libraries/aws-sdk-php/Guzzle/Http/Resources/cacert.pem
  CApath: /etc/ssl/certs
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 2
* Rebuilt URL to: https://x.y.z/?prefix=somedirectory%2F
* Hostname was found in DNS cache
*   Trying x.x.x.x...
* Connected to pro.xy (x.x.x.x) port 8080 (#3)
* Establish HTTP proxy tunnel to x.y.z:443
> CONNECT x.y.z:443 HTTP/1.1
Proxy-Connection: Keep-Alive
Host: x.y.z
User-Agent: aws-sdk-php2/2.8.30 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17 ITR
Date: Fri, 03 Jun 2016 21:19:04 +0000
Authorization: AWS ******

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: ...//sites/all/libraries/aws-sdk-php/Guzzle/Http/Resources/cacert.pem
  CApath: /etc/ssl/certs
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 3

Things I tried/comments:

• Disabling opcache, same behaviour.
• Setting CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST to 0, same behaviour
• The timeout being 0 I think comes from curl/curl#619, I tried different amount of timeouts, including incredible long ones.

[srinivasudadi9000]

Hi ppaulis

I would like to send push notifications for android
so for that i wrote code using curlinit() method initialization
but aws server showing like as below connection closed

[srinivasudadi9000]

in webservice it is showing error like as below
Fatal error: Call to undefined function curl_init() in /home/ubuntu/projects/furreka/trunk/server/furreka_rest_api/api.php on line 267

[ppaulis]

@srinivasudadi9000 it seems to me that there is a package missing on your system? Did you install the php-curl package?

[srinivasudadi9000]

Hi ppaulis please guide me , what is that package

$fields = array
(
'registration_ids' => $registrationIds,
'data' => $msg
);

$headers = array
(
'Authorization: key=' . API_ACCESS_KEY,
'Content-Type: application/json'
);

$ch = curl_init();
curl_setopt( $ch,CURLOPT_URL, 'https://gcm-http.googleapis.com/gcm/send' );
curl_setopt( $ch,CURLOPT_POST, true );
curl_setopt( $ch,CURLOPT_HTTPHEADER, $headers );
curl_setopt( $ch,CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch,CURLOPT_SSL_VERIFYPEER, false );
curl_setopt( $ch,CURLOPT_POSTFIELDS, json_encode( $fields ) );
$result = curl_exec($ch );
// print_r($headers );
curl_close( $ch );

        $userdetails = array
        ( "Userdetails"=>failure,
           "status"=>$result,
           "users"=>$users

        );

       print_r(json_encode($userdetails));

[srinivasudadi9000]

I wrote code for sending push notification for android and my local server 192 ..working fine but coming to aws - ec2 server showing fatal errror ..so how to resolve it ..If possible can you send the package related to curl

[srinivasudadi9000]

how to install php-curl package in aws ppaulis can you send me any reference

[ppaulis]

@srinivasudadi9000 Well it depends on your machine... Sometimes it's php-curl, or php5-curl, etc. The command to install it depends in your operating system.

Because your questions are clearly not related to the original topic of this issue, I suggest that you ask it rather on a site like stackoverflow.

Greetings,
Pascal

[imshashank]

@ppaulis Is there any update on the issue? Are you all getting random curl errors?

[oberman]

Yes and no. Internal (to the SDK) curl errors with retries are unavoidable. But if you mess around with the sdk settings you can reduce the amount times you run out of retries causing an error to bubble up.

The settings are here:
https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/configuration.html

My goal was to maximize API call success while minimizing time spent waiting on a bad remote server (or servers) so I globally changed:
retries, 3 -> 40
connect_timeout, infinity to 1 second
I left the request timeout (timeout) as the infinity default, but I overrode it on a per service or per service method call (usually to 1 second) if I knew that service or service method was idempotent (e.g. doing the same request twice doesn't have any problematic side effects, trivial examples x++ is NOT idempotent but setting x to equal 3 is). It's important to know that on connect_timeout or timeout the SDK internally retries, thus the retries bump from 3 to 40 (to give AWS time to reroute traffic from bad to good servers, or to allow a server to recover, etc...).

I also added instrumentation to my code to peek into the SDK internal retries. See the graph of my last week.

You can see that retries happen constantly (avg of 1 every 2 minutes). But, there are spikes. In this case ganglia is dividing the truth, and "max 15.7" was really a period where I saw ~300 retries in 1 minute. It's important to know that I'm making on the order of 10k AWS API calls per minute so even 300 retries was barely noticed.

[kstich]

@oberman @ppaulis @senorbacon

We are currently unable to reproduce the issue and are opening new discussions to explore any possible leads. As part of trying to get to the bottom of this issue, we would like to reaffirm what your broken states are.

• Are you all still experiencing these cURL errors?
• Are the errors inconsistent with regards to time, capacity, and network traffic?
• Are the errors arbitrary in that they are different cURL messages, or are they the same?

Additionally, if you're still experiencing errors, can you update us with your current state on the following information:

• PHP Version
• cURL Version
• OpenSSL Version
• Guzzle Version
• What AWS service the issue is occurring with
• If you have tried contacting the service/support team with regards to the issue
• What steps in this thread, via comment #, you have tried to resolve the issue
• Exception text
• If possible, the outputs of curl_errno or (if Guzzle 6+) GuzzleHttp\Exception\RequestException::getHandlerContext for each of the arbitrary errors

We know this is a lot of information to ask for, but it's all in hopes of driving this to a resolution.

Thank you!

[oberman]

This is a long thread now and I've contributed in two ways:

1.) cURL error 56: Searching my logs, I see this exception is exceedingly rare, but still happens. By "exceedingly rare" I mean 4 times in 2017 out of what I guess are billions of calls to AWS services. My info:
php55-5.5.38-2.119.amzn1.x86_64
curl-7.47.1-9.66.amzn1.x86_64
openssl-1.0.1k-15.96.amzn1.x86_64
AWS SDK 3.24.4
Guzzle 6.2.3

The most recent failure was on 2017-06-07 10:27:04 GMT-4. The stack trace:
exception 'GuzzleHttp\Exception\RequestException' with message 'cURL error 56: TCP connection reset by peer (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)' in REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:187
Stack trace:

0 REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(150): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)

1 REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(103): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))

2 REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(180): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))

3 REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(108): GuzzleHttp\Handler\CurlMultiHandler->processMessages()

4 REDACTED/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(123): GuzzleHttp\Handler\CurlMultiHandler->tick()

5 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(true)

6 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()

7 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()

8 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()

9 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()

10 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()

11 REDACTED/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()

12 REDACTED/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(59): GuzzleHttp\Promise\Promise->wait()

13 REDACTED/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(78): Aws\AwsClient->execute(Object(Aws\Command))

This error is a bummer because the SDK doesn't try to retry it. I think it might be the only error like that?

2.) The general issue of failures at the curl level. I'm able to control the time a call takes and the success rate of the call by configuring timeouts and number of retries (see my most previous post). In my log file processing I break internal retries into the following categories:

capacity = DynamoDB specific and is generally zero.

error = {5xx, PR_CONNECT_RESET_ERROR, PR_NOT_CONNECTED_ERROR}. Generally has been decreasing over time. A year ago I could have measured this category at the hour level (e.g. at least 1 event/hour), but recently I would have to expand to day (and even then I have days without an error).

timeout = {timed out before SSL handshake, Connection timed out after, Resolving timed out after, Operation timed out}. This is steady and I see happen at the "10 minute level" (e.g. at least 1 event every 10 minutes). Again, as previously noted I have extremely agressive timeout settings and a generous number of retries.

[kstich]

@oberman Thanks for all the updated information. cURL error 56 is retried, so you must be running over the specified number of retries.

You mentioned that there are fewer of these errors now than before. What, if anything, has changed in that time?

Is there any correlation between this cURL error surfacing and your outbound traffic? It's possible there is an issue with how many TLS connections are running simultaneously from your application.

[oberman]

curl error 56:
is this a change? I was told curl errorno 56 is not retried other than AWS API calls that are guaranteed to be idempotent:
https://github.com/aws/aws-sdk-php/issues/983

fewer errors:
my error tracking doesn't distinguish between {5xx, PR_CONNECT_RESET_ERROR, PR_NOT_CONNECTED_ERROR}. I peeked at today's logging and the 2 errors I had in the last four hours were 500 errors against DynamoDB (that retried and succeeded). The last time I had consistent networking problems was before August 2016. AWS released a server-side fix (details of that hidden from me) of DyanmoDB around then that dropped that class of problem I was having.

[kstich]

Re: cURL 56 - Apologies, I was looking at outdated information for that portion of the response.

Are the cURL 56 errors you are receiving specific to an AWS service, or is it across multiple services? You've mentioned both SQS and DynamoDB in this thread, but with a decrease in DynamoDB issues as well.

[oberman]

I think we're mixing things up by talking about different issues at the same time:

curl 56 = I see no upward or downward trend. It's extremely rare (a small # per month). I see it in SQS and DDB, but that makes sense since those are my two highest volume APIs. I dug up the latest time it happened and will paste it at the bottom.

AWS API errors = I've seen two plateaus. One before August 2016 and one after. It was higher before and lower now. I believe before was networking errors + 5xx errors. After is just 5xx errors.

The last curl 56 error I saw:

* Rebuilt URL to: https://dynamodb.us-east-1.amazonaws.com/
* Hostname dynamodb.us-east-1.amazonaws.com was found in DNS cache
*   Trying 52.119.226.86...
* Connected to dynamodb.us-east-1.amazonaws.com (52.119.226.86) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=dynamodb.us-east-1.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
*       start date: Sep 30 00:00:00 2016 GMT
*       expire date: Dec 02 23:59:59 2017 GMT
*       common name: dynamodb.us-east-1.amazonaws.com
*       issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> POST / HTTP/1.1
Host: dynamodb.us-east-1.amazonaws.com
X-Amz-Target: DynamoDB_20120810.Query
Content-Type: application/x-amz-json-1.0
aws-sdk-invocation-id: 4190b86981be52048a57ea4ac7a102e4
aws-sdk-retry: 0/0
X-Amz-Date: 20170607T142703Z
Authorization: AWS4-HMAC-SHA256 Credential=[KEY]/20170607/us-east-1/dynamodb/aws4_request, SignedHeaders=aws-sdk-invocation-id;aws-sdk-retry;host;x-amz-date;x-amz-target, Signature=[SIGNATURE]
User-Agent: aws-sdk-php/3.24.4 GuzzleHttp/6.2.1 curl/7.47.1 PHP/5.5.38
Content-Length: 295

* upload completely sent off: 295 out of 295 bytes
* SSL read: errno -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0

[ppaulis]

@kstich sorry for the late reply! Here are already a few the infos you asked for:

• PHP : PHP 7.0.15-0ubuntu0.16.04.4
• cURL : curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
  Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
  Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets
• OpenSSL : 1.0.2g
• Guzzle : 6.2.1
• AWS Services : Mainly on SWF, but having seen it also on Put operations for S3 if I remember well

I will try to provide you with the detailed error output asap.

Thanks!

[kstich]

@ppaulis @oberman Thank you both for the updates on your current state, we're continuing to try to diagnose the issue. Please keep us updated if you have any new or interesting information, we will do the same.

[gregholland]

I finally upgraded from v2 to v3 of the SDK a couple of weeks ago and have been seeing lots of these errors. For me they only occur on query requests to DynamoDB using Aws\ResultPaginator.

Error executing "Query" on "https://dynamodb.us-east-1.amazonaws.com"; AWS HTTP error: cURL error 56: SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) Unable to parse error information from response - Error parsing JSON: Control character error, possibly incorrectly encoded

I've read this entire thread but still have no idea how to go about solving it. Has anyone had any luck getting to the bottom of this? If so, care to share?

PHP: 7.1.10
OpenSSL: 1.0.1t
Curl: 7.38.0
Guzzle: 6.3.0
AWS Services: DynamoDB (Specifically Query using the ResultPaginator)

Thanks!

[gregholland]

Also occurring with the following setup:
PHP: 7.2.0
OpenSSL: 1.1.0f
Curl: 7.52.1
Guzzle: 6.3.0
AWS Service: DynamoDB only using the ResultPaginator.

[kstich]

Root Cause

Guzzle v5 and Guzzle v6 do not trigger automatic retry of CURLE_RECV_ERROR like was the case in Guzzle v3. The AWS SDK for PHP v2 uses Guzzle3, where AWS SDK for PHP v3 specifies Guzzle v5 or v6.

But why?

At this point in the request/response lifecycle, there is no guarantee that retrying on that error code would be free of side-effects. ~This same thought from Guzzle applies to the SDK itself, so the error code is not retried in an SDK specific manner.~

Now what?

If you are receiving this error, and believe the operation is safe to retry in your situation (no destructive or unintended side-effects, etc.), you may wish to retry the error in your own code.

Middlewares Solution

Create an additional 'retry-curl56' middleware with a custom decider for just this case and add it to the client's handler list via appendSign. You can read more about working with the SDK's middlewares here.

[kstich]

After some further discussion based around the operations being handled, it is believed we can retry this at the SDK level since it's not safe to do at the Guzzle level. Support is being added for automatic retry of CURLE_RECV_ERROR via #1463.

[ppaulis]

Thanks guys! That was not an easy one to find :-)

[kstich]

This was included in the 3.52.0 release.

[neoacevedo]

OK, with aws.phar 3.54.3 I'm having this performance using it on Amazon S3.