Aws-sdk-java: AWS4Signer incorrectly signs POST request with query parameters and no payload

[gavlyukovskiy]

I'm using AWS4Signer with ES cluster 6.7.0 according to this guide https://docs.aws.amazon.com/en_us/elasticsearch-service/latest/developerguide/es-request-signing.html.
Currently most requests are working just fine, but when I'm trying to make index refresh it fails with following message:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

Here is sample code to reproduce the issue:

import com.amazonaws.auth.AWS4Signer;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.http.AWSRequestSigningApacheInterceptor;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequestInterceptor;
import org.elasticsearch.action.admin.indices.refresh.RefreshRequest;
import org.elasticsearch.action.admin.indices.refresh.RefreshResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;

import java.io.IOException;

public class AWSElasticsearchSignTest {

    public static void main(String[] args) throws IOException {
        AWS4Signer signer = new AWS4Signer();
        signer.setServiceName("es");
        signer.setRegionName("us-west-1");

        HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor("es", signer,
                new AWSStaticCredentialsProvider(new BasicAWSCredentials("<accessKey>", "<secretKey>")));

        RestHighLevelClient esClient = new RestHighLevelClient(RestClient.builder(HttpHost.create("https://<clusterName>.us-west-1.es.amazonaws.com"))
                .setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(interceptor)));

        RefreshRequest request = new RefreshRequest("test");
        RefreshResponse response = esClient.indices().refresh(request, RequestOptions.DEFAULT);

        System.out.println(response.toString());
    }
}

Issue is that elasticsearch client sends POST request with query parameters but with no payload:

POST https://<clusterName>.us-west-1.es.amazonaws.com/test/_refresh?ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true

which is special case for AbstractAWSSigner that have next assumption (from the code comments)

If we're using POST and we don't have any request payload content,
then any request query parameters will be sent as the payload, and
not in the actual query string.

I can't say if this is valid behavior for other cases, but it breaks existing code that is not under my control.
As a workaround I modified AWSRequestSigningApacheInterceptor to set empty body:

if (request instanceof HttpEntityEnclosingRequest) {
    HttpEntityEnclosingRequest httpEntityEnclosingRequest =
            (HttpEntityEnclosingRequest) request;
    if (httpEntityEnclosingRequest.getEntity() != null) {
        signableRequest.setContent(httpEntityEnclosingRequest.getEntity().getContent());
    }
    // setting empty body here as workaround to POST request signing without body
    else {
        signableRequest.setContent(new ByteArrayInputStream(new byte[0]));
    }
}

Also at the same time I noticed that python sdk doesn't fail in this case:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3

host = '<clusterName>.us-west-1.es.amazonaws.com'
region = 'us-west-1'

service = 'es'
awsauth = AWS4Auth('<accessKey>', '<secretKet>', region, service)

es = Elasticsearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

es.indices.refresh(index="test", params={'ignore_unavailable': 'false', 'expand_wildcards': 'open', 'allow_no_indices': 'true'})

[debora-ito]

Hi @gavlyukovskiy, can you check which version of Apache httpclient you're using?
We recently had some issues with httpclient-4.5.7 and 4.5.8, and the way the issue manifested itself was with a signature mismatch message.

I just want to discard this as the possible cause, before analyzing it further.

[gavlyukovskiy]

@debora-ito I'm using 4.5.5, but I also tried to generate headers on java side without using httpclient at all:

DefaultRequest<String> request = new DefaultRequest<>("es");
request.setEndpoint(URI.create("https://<clusterName>.us-west-1.es.amazonaws.com"));
request.setResourcePath("/test/_refresh");
request.setHttpMethod(HttpMethodName.POST);
request.setParameters(ImmutableMap.of("test", ImmutableList.of("param")));
request.setHeaders(ImmutableMap.of());

AWS4Signer signer = new AWS4Signer();
signer.setServiceName("es");
signer.setRegionName("us-west-1");
signer.sign(request, new BasicAWSCredentials("<accessKey>", "<secretKey>"));

String headers = request.getHeaders().entrySet().stream().map(e -> e.getKey() + ":" + e.getValue()).collect(Collectors.joining("\n"));
System.out.println(headers);

and use printed headers for manual request. Result is the same unless I put empty body into the request.

[debora-ito]

@gavlyukovskiy I see the issue but this is an use case we don't support unfortunately, as the AWS4Signer is intended to be used with the service APIs that are under the SDK project. We cannot guarantee the signer will work with external clients.

Thank you for reaching out. Feel free to reopen if you have any other question.

[gavlyukovskiy]

AWS4Signer is intended to be used with the service APIs that are under the SDK project

@debora-ito I understand it, but I believe it's different situation. Elastic's RestHighLevelClient is not part of AWS SDK, but the tutorial (https://docs.aws.amazon.com/en_us/elasticsearch-service/latest/developerguide/es-request-signing.html) states that I can (and for our ES cluster it's the only way) use signed requests to Elasticsearch Service:

This chapter includes examples of how to send signed HTTP requests to Amazon Elasticsearch Service using Elasticsearch clients and other common libraries. These code samples are for interacting with the Elasticsearch APIs, such as _index, _bulk, and _snapshot. If your domain access policy includes IAM users or roles, you must sign requests to the Elasticsearch APIs.

The tutorial then shows how can I send requests such as esClient.index() using RestHighLevelClient. I expect other methods that are available in RestHighLevelClient to be working as well, in my case esClient.indices().refresh(request, RequestOptions.DEFAULT), otherwise it is not possible to use Elasticsearch Service.

[nfw-legion]

Yeah, you should support the tutorial suggested method using the RestHighLevelClient .

[harshvasoya008]

I was stuck on this since 10-15 days! @gavlyukovskiy @debora-ito you made my day! Thanks a lot!