Aws-sdk-net: AWS S3 upload object HttpErrorResponseException

Created on 3 Nov 2016 · 27Comments · Source: aws/aws-sdk-net

I'm trying to upload my assets to an S3 bucket but AWS keeps getting an HttpErrorResponseException exception.
I'm testing the S3Example in the editor with only one difference, instead of using CognitoAWSCredentials I'm using _credentials = new BasicAWSCredentials(awsKey, awsSecretKey);

Is there a way to upload a file to S3 with Unity? Or is it not supported at the time?

Using Unity 5.4.2f & aws-sdk-unity_3.3.14.1 plugin

Stack Trace error:

receieved error Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
  at Amazon.Runtime.Internal.UnityWwwRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  at Amazon.S3.AmazonS3Client.ProcessPostResponse (IAsyncResult result) [0x00000] in <filename unknown>:0 
UnityEngine.Debug:Log(Object)
AWSSDK.Examples.S3Example:<PostObject>m__7(AmazonServiceResult`2) (at Assets/Examples/S3Example.cs:175)
Amazon.S3.<>c__DisplayClass4_0:<PostObjectAsync>b__0(AmazonWebServiceRequest, AmazonWebServiceResponse, Exception, AsyncOptions)
Amazon.Runtime.Internal.UnityMainThreadDispatcher:ProcessRequests()
Amazon.Runtime.Internal.UnityMainThreadDispatcher:Update()

Unity bug closed-for-staleness modulunity response-requested

Source

CoderGamester

Most helpful comment

I managed to upload the file using PutObjectRequest instead of post. I remember one issue with form on unity which I had to use put intead of post to a http request. Maybe is the same thing internally to AWSSDK.Core

` Debug.Log("Creating request object");
var request = new PutObjectRequest() {
BucketName = S3BucketName,
Key = Path.GetFileName(path),
InputStream = stream,
CannedACL = S3CannedACL.Private
};

    Debug.Log("Making HTTP post call");
    lastRequestSucceed = false;
    try {

        Client.PutObjectAsync(request, (responseObj) => {

            if (responseObj.Exception == null) {
                Debug.Log(string.Format("object {0} posted to bucket {1}", responseObj.Request.Key, responseObj.Request.BucketName));
                lastRequestSucceed = true;
            } else {
                Debug.Log("\nException while posting the result object");
                Debug.Log(string.Format("\n receieved error {0}", responseObj.Response.HttpStatusCode.ToString()));
            }
            isLastRequestDone = true;
        });
    }
    catch (Exception ex) {
        Debug.Log("Unhandled Exception while posting the result object");
        Debug.Log(string.Format("\n received error {0}", ex.ToString()));

    }`

cecicosta on 8 May 2018

👍5

All 27 comments

What is the value of HttpErrorResponseException.Message?

sstevenkang on 3 Nov 2016

Message gives only "Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown."

CoderGamester on 3 Nov 2016

Huh, that's really odd. I would assume an error message from the service would show up. Is there any chance you could set up a sniffer like Fiddler and monitor the http traffic and see if the request makes it to the service?

Also, is there any inner exception you could take a look at?

sstevenkang on 3 Nov 2016

Hi, The Inner Exception gives null (strange, right?)

How do you set up Fiddler? Never used a Sniffer before :(

CoderGamester on 4 Nov 2016

Fiddler is available fore free here: http://www.telerik.com/fiddler
If you are working on Mac, there's Charles: https://www.charlesproxy.com/

sstevenkang on 4 Nov 2016

Ok I installed the Fiddler and I see that the messages sent to Amazon S3 are with HTTP 200.
What do I do know? How can I find the problem?

CoderGamester on 4 Nov 2016

Do you have the wrong time set on your device?

WebVRRollerCoaster on 18 Nov 2016

I'm also using S3Example.cs from Unity AWSSDK and got HttpErrorResponseException, then after i setting AWSConfigs in Start()

        void Start()
        {
            UnityInitializer.AttachToGameObject(this.gameObject);
            GetBucketListButton.onClick.AddListener(() => { GetBucketList(); });
            PostBucketButton.onClick.AddListener(() => { PostObject(); });
            GetObjectsListButton.onClick.AddListener(() => { GetObjects(); });
            DeleteObjectButton.onClick.AddListener(() => { DeleteObject(); });
            GetObjectButton.onClick.AddListener(() => { GetObject(); });

            AWSConfigs.HttpClient = AWSConfigs.HttpClientOption.UnityWebRequest;

            AWSConfigs.LoggingConfig.LogTo = LoggingOptions.UnityLogger;
            AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
            AWSConfigs.LoggingConfig.LogMetrics = true;
            AWSConfigs.CorrectForClockSkew = true;
        }

now i can see the problem

Received error response: [<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message>
<RequestId>99EB18D75BE52010</RequestId>
<HostId>gl3VFqE2t4esxMl5vXsVROEPZlM5hoXlSTdl3QdH5lJJW+5UYvMDCIg4pqAOYMa+vlux0qrqonw=</HostId></Error>]

try with Full Access Policy and it's works

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "FullAccessS3",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

si294r on 22 Jun 2017

👍4

Hit the same problem with the S3Example in Unity 2017.3.1f1 and the latest AWSSDK packages.

Bucket access is public, and I set AWSConfigs the same as @si294r

Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
  at Amazon.Runtime.Internal.UnityRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00008] in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_unity\UnityWebRequestFactory.cs:288 
  at Amazon.S3.AmazonS3Client.ProcessPostResponse (System.IAsyncResult result) [0x0001c] in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\_unity\AmazonS3Client.unity.cs:266

The inner exception is NULL, and responseObj.Response is also null.

Arc8ngel on 25 Apr 2018

I am having a similar issue on Unity 2017.0p4. In unity 5.6 the same code is working.

The Exception message is this:
"Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.\r\n at Amazon.Runtime.Internal.UnityRequest.EndGetResponse (IAsyncResult asyncResult) [0x00008] in E:\\JenkinsWorkspaces\\v3-trebuchet-release\\AWSDotNetPublic\\sdk\\src\\Core\\Amazon.Runtime\\Pipeline\\HttpHandler\\_unity\\UnityWebRequestFactory.cs:288 \r\n at Amazon.S3.AmazonS3Client.ProcessPostResponse (IAsyncResult result) [0x0001c] in E:\\JenkinsWorkspaces\\v3-trebuchet-release\\AWSDotNetPublic\\sdk\\src\\Services\\S3\\Custom\\_unity\\AmazonS3Client.unity.cs:266 "

The get method is working, so I think it is no problem with authentication. I as I said, I can do post to the same bucket running this on unity 5.6, so no problem with permisions either. Not sure yet if the unity version is the cause. I may continue testing to confirm.

This is my post code:

` isLastRequestDone = false;
if (!File.Exists(path)) {
Debug.Log("File does not exists");
lastRequestSucceed = false;
isLastRequestDone = true;
return;
}
Debug.Log("Retrieving the file");
var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read);

    Debug.Log("Creating request object");
    var request = new PostObjectRequest() {
        Region = _S3Region,
        Bucket = S3BucketName,
        Key = Path.GetFileName(path),
        InputStream = stream,
        CannedACL = S3CannedACL.Private
    };

    Debug.Log("Making HTTP post call");
    lastRequestSucceed = false;
    try {
        Client.PostObjectAsync(request, (responseObj) => {

            if (responseObj.Exception == null) {
                Debug.Log(string.Format("object {0} posted to bucket {1}", responseObj.Request.Key, responseObj.Request.Bucket));
                lastRequestSucceed = true;
            } else {
                Debug.Log("\nException while posting the result object");
                Debug.Log(string.Format("\n receieved error {0}", responseObj.Response.HttpStatusCode.ToString()));
            }
            isLastRequestDone = true;
        });
    }
    catch (Exception ex) {
        Debug.Log("Unhandled Exception while posting the result object");
        Debug.Log(string.Format("\n received error {0}", ex.ToString()));

    }`

cecicosta on 8 May 2018

` Debug.Log("Creating request object");
var request = new PutObjectRequest() {
BucketName = S3BucketName,
Key = Path.GetFileName(path),
InputStream = stream,
CannedACL = S3CannedACL.Private
};

    Debug.Log("Making HTTP post call");
    lastRequestSucceed = false;
    try {

        Client.PutObjectAsync(request, (responseObj) => {

            if (responseObj.Exception == null) {
                Debug.Log(string.Format("object {0} posted to bucket {1}", responseObj.Request.Key, responseObj.Request.BucketName));
                lastRequestSucceed = true;
            } else {
                Debug.Log("\nException while posting the result object");
                Debug.Log(string.Format("\n receieved error {0}", responseObj.Response.HttpStatusCode.ToString()));
            }
            isLastRequestDone = true;
        });
    }
    catch (Exception ex) {
        Debug.Log("Unhandled Exception while posting the result object");
        Debug.Log(string.Format("\n received error {0}", ex.ToString()));

    }`

cecicosta on 8 May 2018

👍5

I'm still stuck on this issue. I can list objects in the bucket, but neither Post or Put methods are getting me anywhere. The first error to hit the log is always HttpErrorResponseException, but others thrown seem to indicate it's an 'Access Denied' error.

The bucket has public access, so why would I get an access denied callback?

Arc8ngel on 16 May 2018

Maybe you can check the region you setup for accessing the S3, the key you're setting for post/put (path + filename) if it doesn't have invalid characteres, check if you can post/put with private access (if so, double check the permissions).
Maybe you can post your code here to check

cecicosta on 17 May 2018

It's finding the bucket okay. Pulling down the list of bucket objects works just fine, as does the Cognito authentication call before the request. I tried switching regions for the sake of testing, and only the one I was previously using works for the pull request. Also did a double-check of the path+filename -- File is being found, and has no unfriendly chars in its name or path.

The code is essentially just the code from the S3Example script provided by AWS (with extra debug logs):
````
private void PutObject(string fileName)
{
var stream = new FileStream(Application.persistentDataPath + Path.DirectorySeparatorChar + fileName, FileMode.Open, FileAccess.Read, FileShare.Read);
var putRequest = new PutObjectRequest()
{
BucketName = S3BucketName,
Key = fileName,
InputStream = stream,
CannedACL = S3CannedACL.Private
};
ResultText.text += "\nMaking HTTP put call";
PutObjectAsync(putRequest);
}

private void PutObjectAsync(PutObjectRequest objRequest)
{
Client.PutObjectAsync(objRequest, (responseObj) =>
{
if( responseObj.Exception == null ) {
ResultText.text += string.Format("\nobject {0} posted to bucket {1}", responseObj.Request.Key, responseObj.Request.BucketName);
} else {
ResultText.text += "\nException while posting the result object";
if( responseObj != null ) {
if( responseObj.Response != null ) {
ResultText.text += string.Format("\n receieved error {0}", responseObj.Response.HttpStatusCode.ToString());
} else {
if( responseObj.Exception.InnerException != null ) {
Debug.LogError("responseObj.Response NULL\n" + responseObj.Exception.InnerException);
} else {
Debug.LogError("responseObj.Response.InnerException NULL\n" + responseObj.Exception);
}
}
} else {
Debug.LogError("responseObj NULL");
}
}
});
}
````

I have also tried changing the CannedACL to PublicReadWrite as well as NoACL. No luck there, either.

Arc8ngel on 17 May 2018

😕1

Here's the permissions, fwiw [Sid & bucketName have been altered]:

{
    "Version": "2012-10-17",
    "Id": "AppAccess",
    "Statement": [
        {
            "Sid": "Stmt1524688564267",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "*",
            "Resource": "arn:aws:s3:::bucketName"
        }
    ]
}

Arc8ngel on 17 May 2018

GOT IT!
"Resource": "arn:aws:s3:::bucketName/*"

I was missing the /* at the end of the resource path in the access policy. That allowed the Put request, but broke the List Objects request, though, so I guess I have a few things to learn about getting the policies right.

Arc8ngel on 17 May 2018

😄2

Nice!
Why don't you use the visual interface for setting the policies in IAM? So there is no risk of misstyping =)

cecicosta on 17 May 2018

👍1

Hi, thank you for sharing that much details, i've found this topic after looking for a case with the same log error, I've already written as mentioned:

AWSConfigs.HttpClient = AWSConfigs.HttpClientOption.UnityWebRequest;

no improvement so far, the only difference with the script i use and the one posted here is that section:

{
"Version": "2012-10-17",
"Id": "AppAccess",
"Statement": [
{
"Sid": "Stmt1524688564267",
"Effect": "Allow",
"Principal": "",
"Action": "",
"Resource": "arn:aws:s3:::bucketName/*"
}
]
}

I dont use any arn yet, does that the reason why nothing work ? until now i only use a ARN generated in my datawarehouse for a different purpose which is currently part of a user's parameter generated in IAM casual way to create a pipe. If i've understood between unity and S3 bucket i only need: arn:aws:s3:::Mybucketname, but where should i write this authentification in the script?

JeremyBdda on 6 Jun 2018

👍1

Hi, try to add region ("Region = _S3Region") to your request like here:

       request = new PostObjectRequest()
           {
                Bucket = S3BucketName,
                Key = fileName,
                InputStream = stream,
                CannedACL = S3CannedACL.Private,
                Region = _S3Region
        };

The things that I've done:

Changed HttpClient

UnityInitializer.AttachToGameObject(this.gameObject); 
AWSConfigs.HttpClient = AWSConfigs.HttpClientOption.UnityWebRequest;

Added AmazonS3FullAccess policy for both Cognito_nameUnauth_Role and Cognito_nameAuth_Role
Added "Region = _S3Region" to my request

And it finally worked :)

Check out this thread as well: https://github.com/aws/aws-sdk-net/issues/332

A-Kojic on 8 Jun 2018

👍1

For me, I added the config settings suggested here, changed it to a Put request, and it seems to work. Here is my modified S3 example file https://gist.github.com/jackbrookes/d181a3ca08399959f27777886cfb7e88

jackbrookes on 7 Aug 2018

GOT IT!
"Resource": "arn:aws:s3:::bucketName/*"

I was missing the /* at the end of the resource path in the access policy. That allowed the Put request, but broke the List Objects request, though, so I guess I have a few things to learn about getting the policies right.

Very Good ! Thank you

ismaelash on 15 Oct 2018

So it sounds like the example code that does a post instead of a put is just wrong? Seems easy to fix, but i couldn't find that in this repo.

michaeleconomy on 18 Oct 2018

You can now, use sdk of S3 modified and better: https://github.com/IsmaelNascimento/aws-sdk-s3-for-unity

ismaelash on 20 Oct 2018

Following @Arc8ngel, I was able to upload if I using PUT instead of POST.
I'm not concerned with POST, so I'll leave it as PUT.
I am using Unity 2019.3.5f1.

POST will give the following two errors.
①

Curl error 56: Receiving data failed with unitytls error code 1048578

②

HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
Amazon.Runtime.Internal.UnityRequest.GetResponse () (at <b1d94c7517784dee82153027b75adc3f>:0)
Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) (at <b1d94c7517784dee82153027b75adc3f>:0)
Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) (at <b1d94c7517784dee82153027b75adc3f>:0)
Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) (at <b1d94c7517784dee82153027b75adc3f>:0)
Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) (at <b1d94c7517784dee82153027b75adc3f>:0)
Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) (at <b1d94c7517784dee82153027b75adc3f>:0)
UnityEngine.Debug:LogException(Exception)
Amazon.Runtime.Internal.Util.UnityDebugLogger:Error(Exception, String, Object[])
Amazon.Runtime.Internal.Util.Logger:Error(Exception, String, Object[])
Amazon.Runtime.Internal.ErrorHandler:ProcessException(IExecutionContext, Exception)
Amazon.Runtime.Internal.ErrorHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.CallbackHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.Signer:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.EndpointDiscoveryHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.CredentialsRetriever:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.RetryHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.CallbackHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.EndpointResolver:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.Marshaller:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.CallbackHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.ErrorCallbackHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.MetricsHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.PipelineHandler:InvokeSync(IExecutionContext)
Amazon.Runtime.Internal.RuntimePipeline:InvokeSync(IExecutionContext)
Amazon.Runtime.AmazonServiceClient:Invoke(AmazonWebServiceRequest, InvokeOptionsBase)
Amazon.CognitoIdentity.AmazonCognitoIdentityClient:GetCredentialsForIdentity(GetCredentialsForIdentityRequest)
Amazon.CognitoIdentity.CognitoAWSCredentials:GetCredentialsForIdentity(GetCredentialsForIdentityRequest)
Amazon.CognitoIdentity.CognitoAWSCredentials:GetPoolCredentials()
Amazon.CognitoIdentity.CognitoAWSCredentials:GenerateNewCredentials()
Amazon.Runtime.RefreshingAWSCredentials:GetCredentials()
Amazon.S3.Util.S3PostUploadSignedPolicy:GetSignedPolicyV4(String, AWSCredentials, RegionEndpoint)
Amazon.S3.AmazonS3Client:CreateSignedPolicy(PostObjectRequest)
Amazon.S3.<>c__DisplayClass7_0:<PostObjectAsync>b__1(Object)
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()

okamototomoyuki1988 on 18 Mar 2020

Hi @CoderGamester,

Please review the https://github.com/aws/aws-sdk-net/issues/484#issuecomment-395746536 and confirm if this issue could be closed.

Thanks,
Ashish

ashishdhingra on 5 Nov 2020

This issue has not recieved a response in 2 weeks. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

github-actions[bot] on 20 Nov 2020

I have received HttpErrorResponseException Error While Uploading a image. When i add this to policy generator . The Error is solved and file is uploaded.

 {
"Version": "2008-10-17",
"Statement": [
 {
 "Sid": "FullAccessS3",
 "Effect": "Allow",
 "Principal": "*",
  "Action": [
   "s3:GetObject",
   "s3:ListBucket",
    "s3:*Object"               
   ],
  "Resource": [
   "arn:aws:s3:::your s3 bucket name",
   "arn:aws:s3:::your s3 bucket name/*"
    ]
    }
   ]
   }