Aws-sdk-php: Add support for AWS Elasticsearch actual Elasticsearch operations (insert, search, etc.)

[et304383]

The SDK REALLY needs to be updated. The hassle one has to go through now manually signing requests using Signature 4 and temporary credentials in a huge pain. Please add ES support.

[jeskew]

Hi @eric-tucker,

Are you using an Elasticsearch library like Elastica or Elasticsearch-php?

[et304383]

Neither. Right now, I'm using a python script to do the Signature 4 signing as outlined here:
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

With modifications for Elasticsearch of course.

Then, I have to put it into Lambda (tried deploying under WSGI - wow, what a pain) and then put API Gateway in front of it and pass in temporary credentials. Basically, I had to write an entire REST API because none of the SDKs support the application operations against Elasticsearch in AWS (just domain operations at the AWS object level).

[jeskew]

There's a lot of great Elasticsearch tooling, some of which is pretty easy to inject a signer into. Take Elasticsearch-PHP, for example. The following code will let you use the Elasticsearch-PHP API and automatically sign all requests for you:

$psr7Handler = \Aws\default_http_handler();
$signer = new \Aws\Signature\SignatureV4('es', <your-region>);
$credentialProvider = \Aws\Credentials\CredentialProvider::defaultProvider();

$handler = function (array $request) use ($psr7Handler, $signer, $credentialProvider) {
    // Amazon ES listens on standard ports (443 for HTTPS, 80 for HTTP).
    $request['headers']['host'][0] = parse_url($request['headers']['host'][0])['host'];

    // Create a PSR-7 request from the array passed to the handler
    $psr7Request = new \GuzzleHttp\Psr7\Request(
        $request['http_method'],
        (new \GuzzleHttp\Psr7\Uri($request['uri']))
            ->withScheme($request['scheme'])
            ->withHost($request['headers']['host'][0]),
        $request['headers'],
        $request['body']
    );

    // Sign the PSR-7 request with credentials from the environment
    $signedRequest = $signer->signRequest(
        $psr7Request,
        call_user_func($credentialProvider)->wait()
    );

    // Send the signed request to Amazon ES
    /** @var \Psr\Http\Message\ResponseInterface $response */
    $response = $psr7Handler($signedRequest)->wait();

    // Convert the PSR-7 response to a RingPHP response
    return new \GuzzleHttp\Ring\Future\CompletedFutureArray([
        'status' => $response->getStatusCode(),
        'headers' => $response->getHeaders(),
        'body' => $response->getBody()->detach(),
        'transfer_stats' => ['total_time' => 0],
        'effective_url' => (string) $psr7Request->getUri(),
    ]);
};

$client = \Elasticsearch\ClientBuilder::create()
    ->setHandler($handler)
    ->setHosts(['https://<your-amazon-es-domain>:443'])
    ->build();

That would give you IAM authentication and access to the entire Elasticsearch-PHP API.

Hope that helps!

[et304383]

I appreciate the work-arounds. Are you implying that support for the ES API through the AWS SDK won't be added anytime soon?

[jeskew]

Adding an ES data plane client isn't currently on our timeline. There are a lot of mature, fully-featured, open-source Elasticsearch clients already on the market, and my recommendation would be using one of them along with the SDK's signer.

I will let the ES team know about your request. Customer input is always taken into consideration.

[et304383]

Can you clarify where $request comes from in the example you've given?

[jeskew]

In the handler, the request comes from Elasticsearch-PHP. You would pass the $handler function to Elasticsearch\ClientBuilder::setHandler to tell Elasticsearch-PHP to pass all requests through it.

They document handlers here.

[NielsCorneille]

Thanks a lot for the snippet to sign requests.
Errors outputted "The promise was rejected" instead of the normal exceptions (for example: Missing404Exception).
I seems to be fixed by adding this promise:

$oResponse = $oPsr7Handler($oSignedRequest)->then(
function (\Psr\Http\Message\ResponseInterface $oResponse)
{
    return $oResponse;
},
function ($aError)
{
    return $aError['response'];
}
)->wait();

Not sure if that is the correct fix, feel free to suggest a better one!

[jeskew]

@NielsCorneille The snippet just uses a Guzzle client, not an SDK client, so you can decorate it with whatever middleware works for you. Your solution is very similar to how we handle promise rejections in the SDK.

[sergey-rud]

@jeskew 's solution started working when I changed key 'host' to 'Host':

// Amazon ES listens on standard ports (443 for HTTPS, 80 for HTTP).
$request['headers']['Host'][0] = parse_url($request['headers']['Host'][0])['host'];

// Create a PSR-7 request from the array passed to the handler
$psr7Request = new \GuzzleHttp\Psr7\Request(
    $request['http_method'],
    (new \GuzzleHttp\Psr7\Uri($request['uri']))
        ->withScheme($request['scheme'])
        ->withHost($request['headers']['Host'][0]),
    $request['headers'],
    $request['body']
);

[spellmen]

@jeskew 's solution started working when I changed key 'host' to 'Host':

// Amazon ES listens on standard ports (443 for HTTPS, 80 for HTTP).
$request['headers']['Host'][0] = parse_url($request['headers']['Host'][0])['host'];

// Create a PSR-7 request from the array passed to the handler
$psr7Request = new \GuzzleHttp\Psr7\Request(
    $request['http_method'],
    (new \GuzzleHttp\Psr7\Uri($request['uri']))
        ->withScheme($request['scheme'])
        ->withHost($request['headers']['Host'][0]),
    $request['headers'],
    $request['body']
);

Thanks a lot. Its working fine.
But i'm facing an issue with parse_url function. I'm using latest version of php and elasticsearch. So that the parse_url function should be like this
$request['headers']['Host'][0] = parse_url($request['headers']['Host'][0],PHP_URL_HOST);

[xiangweilee]

I need to add the content-type header to make it work.

// Amazon ES listens on standard ports (443 for HTTPS, 80 for HTTP).
$request['headers']['host'][0] = parse_url($request['headers']['host'][0])['host'];
$request['headers']['content-type'] = 'application/json';