Aws-sdk-php: Crash on S3Class Instatiate

[davelassanske]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for PHP?

3.109.1

Version of PHP (php -v)?

7.0.33

What issue did you see?

Page crash (504 timeout) when page gets to instantiating a S3Client

Steps to reproduce

use Aws\Common\Aws;
use Aws\S3\S3Client;
use Aws\S3\Iterator;
use Aws\S3\Exception\S3Exception;
use Aws\Common\Exception\InvalidArgumentException;
use Aws\Common\Exception\RequiredExtensionNotLoadedException;

try{
    $this->s3Client = new S3Client([
        'credentials' => array(
            'key' => XXXXXXXX,
            'secret' => XXXXXXXXX
        ),
        'region' => 'us-east-1',
        'version' => '2006-03-01'
    ]);
} catch (AwsException $e) {
    // This catches the more generic AwsException. You can grab information
    // from the exception using methods of the exception object.
    echo $e->getAwsRequestId() . "\n";
    echo $e->getAwsErrorType() . "\n";
    echo $e->getAwsErrorCode() . "\n";

    // This dumps any modeled response data, if supported by the service
    // Specific members can be accessed directly (e.g. $e['MemberName'])
    var_dump($e->toArray());
} catch (Error $e) {
    $this->log_model->log('info', 's3 ERROR Error:'.print_r($e->getMessage(), true));
} catch (Exception $e) {
    $this->log_model->log('info', 's3 ERROR Exception:'.print_r($e->getMessage(), true));
} catch (ServiceResponseException $e) {
    $this->log_model->log('info', 's3 ERROR ServiceResponseException:'.print_r($e->getMessage(), true));
} catch (S3Exception $e) {
    $this->log_model->log('info', 's3 ERROR S3Exception:'.print_r($e->getMessage(), true));
} catch (InvalidArgumentException $e) {
    $this->log_model->log('info', 's3 ERROR InvalidArgumentException:'.print_r($e->getMessage(), true));
} catch (RequiredExtensionNotLoadedException $e) {
    $this->log_model->log('info', 's3 ERROR RequiredExtensionNotLoadedException:'.print_r($e->getMessage(), true));
}

Additional context

AWS SDK loaded via composer in an AWS Elastic Beanstalk environment using their "PHP 7.0 running on 64bit Amazon Linux/2.8.12" instance type. In a duplicate environment this error does NOT happen. I can not replicate it on my local machine either.

[ibrahimlawal]

Hello team,

This is still very much an issue.

• Started midnight UTC on Friday, August 9, 2019.
• After doing a remote step-through, our team discovered that the line at which it crashes on our instances is https://github.com/aws/aws-sdk-php/blob/master/src/EndpointDiscovery/ConfigurationProvider.php#L135
• Being production environment, this has been directly responsible for service impairment.

Kindly help investigate as soon as possible please

[ibrahimlawal]

We have 2 PHP configurations running same code, but the result of instantiating a client is same. A fatal crash that doesn't get caught as an Exception or a Throwable:

Configurations:

• PHP 7.1 running on 64bit Amazon Linux/2.8.1
• PHP 7.1 running on 64bit Amazon Linux/2.8.14

Sample code:

<?php

require_once 'vendor/autoload.php';

try {
    $sqs_client = new SqsClient([
        'region' => getenv('AWS_REGION'),
        'version' => 'latest',
        'credentials' => [
            'key' => getenv('AWS_ACCESS_KEY_ID'),
            'secret' => getenv('AWS_SECRET_ACCESS_KEY'),
        ],
        'http' => [
            'connect_timeout' => intval(getenv('AWS_SQS_CONNECT_TIMEOUT')) ?: 5,
            'timeout' => intval(getenv('AWS_SQS_TIMEOUT')) ?: 5,
        ],
    ]);
} catch (\Exception $e) {
    print_r($e->getMessage());
} catch (\Throwable $th) {
    print_r($th->getMessage());
}

cc @diehlaws

[davelassanske]

@ibrahimlawal our team descended down into the rabbit trail and couldn't identify the bug. We finally realized that we were getting a segmentation fault error in one environment and not in another identical environment (except for the server type). We use ElasticBeanstalk and switched the EC2 instance type from an m4.xlarge to a t3.medium - that did the trick for us for the time being.

We're still curious about what is causing the error and why.

[ibrahimlawal]

Thanks for the update @davelassanske .

[ibrahimlawal]

We are experiencing crashes on t2.small and m5.large.

[davelassanske]

@ibrahimlawal interesting - works for us on t2.medium and t3.medium for our codebase - not sure how the instances are configured differently but they must be...

[diehlaws]

Hi @davelassanske and @ibrahimlawal, thanks for reaching out to us about this and I do apologize for the delay in response on our end. I tested Elastic Beanstalk environments that used t2.small, m4.xlarge, and m5.large instance types with code that initializes an S3 client and lists my buckets, and have been unable to reproduce the described behavior. At first I was receiving HTTP 500 errors but this turned out to be due to my vendor/ directory not getting zipped as expected before I uploaded my code to Elastic Beanstalk. You may be able to see additional information on why you're getting HTTP 504 errors by reviewing /var/log/httpd/error_log in the instance(s) returning this error.

Given that the behavior is occurring inconsistently depending on the instance type used in the Elastic Beanstalk environment this seems like it is related to your Elastic Beanstalk environment (or the Elastic Beanstalk service itself) rather than something specific to the AWS SDK for PHP. As a result I'd suggest reaching out to our Premium Support team by opening a new technical support case under the Elastic Beanstalk service for further support on the service end, noting in the case the environments under which the SDK works as expected and the environments under which it does not, along with the errors received on the problem environments and any pertinent log entries in /var/log/httpd/error_log.

[davelassanske]

Thanks for responding @diehlaws - we looked at our logs and saw segmentation fault errors on the m4.xlarge every time we instantiated the S3 class - but not at all on the t2 and t3s. Could there be a memory leak that the burstable t2 and t3 instances can handle but not the m4s?

[ibrahimlawal]

@diehlaws thanks for the suggestion. We have been in contact with same.

For your investigations, is it possible to get an exact copy of the t2.small instances we were getting in eu-west-2 around August 10, 2019 to August 12, 2019 or (m5.large in eu-west-1)? The issue stopped occurring on the instances we got since August 13 evening, and didn't ever happen in the one year since we started using the SDK on those same instances.

[diehlaws]

@davelassanske while there are cyclical references involved in a PutObject call that can lead to a memory leak (see issues #1816, #1572, #1274, and #1273 for more information on this), this shouldn't be occurring at the time the S3Client is being instantiated, and I would expect that an m4.xlarge instance with 16GiB of RAM would be more resilient to such a memory leak than a t2.medium or a t3.medium with 4GiB of RAM unless the m4.xlarge instance had other RAM-heavy tasks running that were not running on the t2/t3 instances.

@ibrahimlawal Unfortunately I do not have that level of access to the back-end of Elastic Beanstalk to identify and clone the t2.small instances you were using during the given timeframe, such a request is better suited for Premium Support and the Elastic Beanstalk service team since they will have more expertise with the service itself and will be able to better identify Elastic Beanstalk-specific behaviors and resources on your account given an instance ID or other resource ID. They should be able to attempt a reproduction of the behavior you're seeing with more knowledge on what to look for on the service end, specially if you send them the .zip file you're using on Elastic Beanstalk. That being said, I am glad to hear you are no longer facing the described issue on your current instances.