Aws-sdk-net: Modify CloudSearch Search to use POST instead of GET

Created on 15 Mar 2018 · 8Comments · Source: aws/aws-sdk-net

Any search request using AmazonCloudSearchDomain.SearchAsync(searchRequest) or AmazonCloudSearchDomain.Search(searchRequest) that has a large search string will end up with a 413 Full Head message being returned from the AWS service.

This exception is not handled gracefully by the Marshaller and will return a Null Pointer Exception that obfuscates the actual cause of the issue. I don't care much for this; it would be better that the actual issue above is addressed.

Expected Behavior

Searching for a long string should return results (create a search longer than 4000 characters for example).

The same search can be run in the console with no problems as it is using post

Current Behavior

Instead it returns
``[NullReferenceException: Object reference not set to an instance of an object.]
Amazon.Runtime.Internal.Transform.JsonErrorResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context) +874
Amazon.CloudSearchDomain.Model.Internal.MarshallTransformations.SearchResponseUnmarshaller.UnmarshallException(JsonUnmarshallerContext context, Exception innerException, HttpStatusCode statusCode) +40
Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.UnmarshallException(UnmarshallerContext input, Exception innerException, HttpStatusCode statusCode) +61
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +282

[AmazonUnmarshallingException: Error unmarshalling response back from AWS. ]
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) +1011
Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) +275``

And using Fiddler we see the response from the server:
HTTP/1.1 413 FULL head Content-Length: 0 Connection: keep-alive

Possible Solution

SearchRequestMarshaller has hard-coded the method of GET for requests.

https://github.com/aws/aws-sdk-net/blob/1a5187e0dcddadba10a6b90fd827f2e485af0f9c/sdk/src/Services/CloudSearchDomain/Generated/Model/Internal/MarshallTransformations/SearchRequestMarshaller.cs#L58

Change it to POST, and the problem will probably go away.

Steps to Reproduce (for bugs)

Create a search index with the field 'type'
Run test:

[Test] public void Test_AWSCloudSearch_GivenReallyLongSearchString() { var search = new AmazonCloudSearchDomainClient("<your-search-domain-here>"); var result = search.Search(new SearchRequest { QueryParser = QueryParser.Structured, Query = "(and type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' type: \'AAAAAA\' )", Start = 1, Size = 15, Sort = "_score desc,objectid asc", Return = "_score,_all_fields" }); Assert.AreNotEqual(result, null); }

Context

We are trying to do a search where words in any field are matched using an or statement. Because there is no way to group terms by field, or an ability to restrict fields conditionally in simple search, we have had to use structured search in this manner, when a long list of keywords are used across a range of fields then the string can become too long for the service to handle.

This is just the first search where we have hit this limit, I am concerned that even if we are able to work around the issue, it exposes a limitation in the ASP.Net library, which could impact whether we choose to use this service in production

Your Environment

AWSSDK.Core version used: 3.3.21.16
Service assembly and version used: AWSSDK.CloudSearch 3.3.0.25
Operating System and version: Windows 10
Visual Studio version: 2015
Targeted .NET platform: 4.5

B bug modulsdk-generated

Source

joel-cass

👍1

All 8 comments

Interesting... thanks for the detailed write up. It looks like CloudSearchDoomain is recommending using GET(the code snipped you included is generated code based on model provided by CSD team). Let me follow up with the service team if it's safe to default to POST.

Also, it looks like we have some gaps where we should re-throw a meaningful exception.

sstevenkang on 16 Mar 2018

I also have the same issue. I see this was adjusted to use POST in the Java SDK. Are there plans to update the .Net SDK soon?

EyeDreamt on 22 Apr 2019

This has become a production issue for me as well. Workarounds are failing at this point. Please look into t his.

tpund on 22 Apr 2019

This has become an issue for me as well. Please look into this.

franzcars on 22 Apr 2019

This has become an issue for our team. At least 50% of our queries are failing because we're requesting all of our fields to be returned and it's too much for a GET request. Have you guys made progress on this since March?

CerebralUnit on 25 Apr 2019

I gave up on AWS helping me out on this a while ago, checked out their code and created a modified version that uses POST instead of GET. It's very easy to swap in if you're using an IOC framework such as Windsor. This should help if you're in a pinch.

aws-sdk-net-issue-900.zip

joel-cass on 29 Apr 2019

@joel-cass, @EyeDreamt, @tpund, @franzcars, @CerebralUnit, and others, sorry we let this issue get lost in the backlog. It looks like this was an issue faced by almost all of our SDKs because we depend on the service models to generate a lot of our code. They specify the HTTP method, which is defaulted to GET for this operation, but the service models don't provide a mechanism for alternative HTTP methods. We should have escalated this as an issue that affects all SDKs when first discovered, but I don't think we did. I'll raise this as a discussion point and see if we can get a fix out for .NET relatively soon.

klaytaybai on 30 Apr 2019

👍1

@sstevenkang @klaytaybai

https://github.com/aws/aws-sdk-java/issues/660

This was fixed in the Java SDK in 2016

https://aws.amazon.com/releasenotes/release-aws-sdk-for-java-1-11-32/

It's really disappointing to see this isn't resolved, we've been running with our own implementation for like 4 years now and I don't want to maintain code that the AWS SDK should be doing for me.