Aws-sdk-php: S3 putObject fails on Elastic Beanstalk with SSL/cURL error

[cpinto]

Hi,

My app is deployed on Elastic Beanstalk v2.1.7. When it tries to upload an object into S3, I'm seeing this error:

>>> $result = $s3->putObject(['Bucket' => 'my.bucket.com','Key'    => 'public/test.txt','Body'   => 'this is the body!','ContentType' => 'text/plain','ACL' => 'public-read']);

Aws\S3\Exception\S3Exception with message 'Error executing "PutObject" on "https://s3-eu-west-1.amazonaws.com/my.bucket.com/public/test.txt"; AWS HTTP error: cURL error 35: A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)'

It'd appear that updating a system package through yum would fix this, however I'm not very comfortable with this as the changes would be lost whenever a new EC2 instance goes up.

Is there a workaround?

[cjyclaire]

@cpinto It appears related to SSL or cURL issue that needs to be fixed from OS side instead of SDK side, so for workaround, here is verify option that you could change in client configuration.

If this still happens in regardless of upgrade solution etc, it would be helpful if you could provide a wire log information with debug option.

[cpinto]

Thanks for sharing the debug option @cjyclaire. As I turned it on, it revealed a trace of the HTTP connection.

Searching for one of the errors yielded some guidance on disabling a fork verification for NSS, as per https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_environment_variables.

The relevant command is: export NSS_STRICT_NOFORK=DISABLED which can be set up as an environment variable for EB easily.

Once this is done, no more errors occur. That said, Mozilla clearly states it's an error to do this but I couldn't find an explanation as to why, which makes me a bit uneasy. However, given that this is the best way forward (I couldn't find an updated libcurl that works with Amazon Linux AMI and compiling the source code to build a custom package feels as it could break at any time) I'll be using it.

Thanks again for sharing the instructions to turn on the HTTP debug.

[cjyclaire]

@cpinto Glad to know that you figured this out, and thank you for sharing this with others. Closing the issue right now, feel free to reopen with further comments or questions :)

[SamuelNorbury]

@cpinto I recently had a similar problem, specifically querying an S3 bucket twice in succession from a Laravel application, and your fix seems to work, though I'm not feeling overly confident about it.

[cwhittl]

I just used to, scary fix but it works... Thank you!

[matt-allan]

Hi,

I have also run into this error. I don't think there is really anything the SDK can do to prevent it but I figured I should share what I learned in case anyone else has this issue.

The issue seems to be caused by a combination of compiling the PHP curl extension against a version of curl that uses NSS instead of OpenSSL or LibreSSl and sharing a curl handle between a parent process and a child process forked using the pcntl extension.

Here is a minimal reproduction script:

<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.example.com/");
curl_exec($ch);
var_dump(curl_error($ch));

$pid = pcntl_fork();
if ($pid == -1) {
     die('could not fork');
} else if ($pid) {
     // we are the parent
     pcntl_wait($status); //Protect against Zombie children
} else {
     // we are the child
    curl_exec($ch);
    var_dump(curl_error($ch));
}

If you run this script you should see:

string(0) ""
string(63) "Bulk data encryption algorithm failed in selected cipher suite."

The error seems to be specifically caused by calling curl_exec($ch) in both the parent and the child process. If you only call curl_exec in either the parent or the child you will not encounter an error.

The workarounds that seem to work are:

1.) Set the environment variable NSS_STRICT_NOFORK to DISABLED. You probably shouldn't do this, I'm guessing they added that protection for a reason.
2.) Compile curl with a different TLS backend. This is explained here.
3.) Don't ever call curl_exec in a forked process. You probably can't always avoid this, but sometimes it's a decent fix. For example, if using php artisan tinker in Laravel you can disable forking to prevent this.

Unfortunately there doesn't seem to be a way to force the PKCS#11 crypto module to re-initialize. Even if you call curl_close on the handle in the parent process it seems to re-use the crypto module. For example, this code does not work:

<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.example.com/");
curl_exec($ch);
var_dump(curl_error($ch));
curl_close($ch);

$pid = pcntl_fork();
if ($pid == -1) {
     die('could not fork');
} else if ($pid) {
     // we are the parent
     pcntl_wait($status); //Protect against Zombie children
} else {
     // we are the child
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, "https://www.example.com/");
    curl_exec($ch);
    var_dump(curl_error($ch));
    curl_close($ch);
}

[jonmcclung]

@yuloh So, instead of compiling with --without-ssl --with-nss, we should compile with which option? Does anyone know which TLS the AWS SDK expects?

[matt-allan]

@jonmcclung any option besides nss should work. The default instructions in the link above should work fine.

I don't think the AWS SDK cares which ssl backend curl uses. It should work just fine with any of them. I've personally confirmed that it works fine with LibreSSL.

[jonmcclung]

@yuloh Thanks! I can confirm that normal OpenSSL seems to solve the problem as well!