Aws-sdk-ruby: Aws::S3::Errors::MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

[mike-bailey]

See: https://github.com/aws/aws-sdk-ruby/issues/1460#issuecomment-322094403

For updates on this issue.
It's reappeared in a different format (identical to the docs, so either the docs are in some way incorrect or the SDK is doing something wrong) so I figured I'd reopen.

[awood45]

Can you provide a wire trace for this?

s3 = Aws::S3::Client.new(http_wire_trace: true)

[mike-bailey]

irb(main):014:0> resp = s3.put_bucket_logging({
irb(main):015:2*   bucket: "sourcebucket",
irb(main):016:2*   bucket_logging_status: {
irb(main):017:3*     logging_enabled: {
irb(main):018:4*     target_bucket: "targetbucket",
irb(main):019:4*     },
irb(main):020:3*   },
irb(main):021:2* })
opening connection to sourcebucketname.s3.amazonaws.com:443...
opened
starting SSL for sourcebucketname.s3.amazonaws.com:443...
SSL established
<- "PUT /?logging HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.10.25 ruby/2.4.1 x86_64-darwin16\r\nExpect: 100-continue\r\nContent-Md5: gTbnihPA+E+nrr7woTgE5Q==\r\nX-Amz-Date: 20170824T212836Z\r\nHost: sourcebucket.s3.amazonaws.com\r\nX-Amz-Content-Sha256: 7906216862bfdea0bc9a4dd1182b4315bbd2dc6ce8884259a1101a46dbdfd05e\r\nAuthorization: AWS4-HMAC-SHA256 Credential=redact/20170824/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=redact\r\nContent-Length: 202\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 100 Continue\r\n"
-> "\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "x-amz-request-id: 8D18B1FA427DF7C5\r\n"
-> "x-amz-id-2: redacted+redacted/6AhyISj4Q=\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Thu, 24 Aug 2017 21:28:30 GMT\r\n"
-> "Connection: close\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
-> "13f\r\n"
reading 319 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>8D18B1FA427DF7C5</RequestId><HostId>SqGmM48esm+NqvmkVTKvbFRBche3itT0uuuHfsOy8iyx5HVkDx6Qqb6nqs0NcRkkk/6AhyISj4Q=</HostId></Error>"
read 319 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn close
Aws::S3::Errors::MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/seahorse/client/plugins/response_target.rb:21:in `call'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/seahorse/client/request.rb:70:in `send_request'
    from /usr/local/lib/ruby/gems/2.4.0/gems/aws-sdk-core-2.10.25/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
    from (irb):14
    from /usr/local/bin/irb:11:in `<main>'
irb(main):022:0> 

I removed anything I thought could have been private. If I'm wrong about that please LMK.

[awood45]

Thanks, and I've been able to recreate this issue as well. Investigating whether this is a client or service issue, since I don't think we've touched the relevant code in some time.

[mike-bailey]

Thank you! Great response time. Let me know if there's any way I could help.

[awood45]

I've reached out to the service team for help debugging this as well.

[mike-bailey]

Cool, thanks. If it matters at all there's a - in one of the bucket names (which is mandatory per our use case). Let me know if I can help further.

[awood45]

Thanks for the note - I don't think that's a factor since I reproduced the error without a dash.

[mike-bailey]

Hello, any updates yet?

[mike-bailey]

Any news? 😄

[ddimmich]

having the same issue from within logstash. would be great to get an update.

[mike-bailey]

Anyone home?

[awood45]

Sorry, no updates yet. Will revisit after reInvent, need to collaborate with a few parties on this one.

[mike-bailey]

No problem, just making sure it wasn't abandoned.

[mike-bailey]

@cjyclaire Any updates regarding this? What's the expected turnaround if you don't mind my asking?

[cjyclaire]

Forward the ping again : ) Appreciate your patience
Also, feel free to cut a TT to AWS Support at the same time as well, they can help look into the request IDs.

[mike-bailey]

Unfortunately I can't open a technical ticket as we don't pay for that level of support.

[awood45]

May have a breakthrough here, as it appears S3 sometimes throws these errors for general usage mistakes, rather than a malformed XML request. Checking.

[awood45]

Potentially related Issue #1670

I've passed this information on to the S3 team and will look for a new update.

[mike-bailey]

Interesting. Thanks for the update. Neither of these are supposed to be improper calls though, no?
Lifecycle policies should be allowed to apply to buckets and logging as well?

[awood45]

To be entirely honest, I'm not sure. The documentation doesn't indicate that the call is improper, but I can't eliminate that possibility either. As far as I can tell:

• The Ruby SDK is properly formatting your request.
• This same error has appeared elsewhere in the case of user error, though of a different kind.

[admorphit]

This is happening for the nodejs aws-sdk as well in case that helps.

[awood45]

@admorphit That's actually very helpful. Do you have a related issue, or a JS code snippet to reproduce that you can provide? Thanks!

[mike-bailey]

That also helps me because it gives me something to tell our management about how it’s more than likely not the Ruby SDK’s fault (other than @awood45 having just said it it isn’t). Thanks!

[awood45]

Okay, I've identified the root cause. As suspected, it's user error, though granted, the requirements here are not well documented (which is feedback I'm pressing with the service team).

• The :target_prefix parameter is required. This is why you're getting an error.
• You'll also likely need to set grant permissions, but at least for that you'll get a much more user-friendly error and the way to do that is documented.

So, it's again much like the other issue, where MalformedXML is being returned in place of a user-friendly error message. This should unblock you - closing, but feel free to add any follow-up questions you have.

[mike-bailey]

The :target_prefix parameter is required. This is why you're getting an error.

However, in the console, a prefix isn't required? Why is this an API limitation only for these SDKs and not the console? I'd like to be able to apply it to the entire bucket...

[mike-bailey]

Also, any developments on the logging call? (I'm assuming you're referring to the lifecycle bit). Thank you!

[mike-bailey]

Looks like in the actual API:

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html

If you want the lifecycle rule to apply to all objects in the bucket, specify an empty prefix.

Maybe something to add to the docs for the actual SDKs. Thank you!

[awood45]

I agree with you entirely - documentation across the SDKs comes from the service teams, and I'm pushing for these updates across the impacted APIs currently. Once done, they'll be included in the standard release process.

[awood45]

However, in the console, a prefix isn't required? Why is this an API limitation only for these SDKs and not the console? I'd like to be able to apply it to the entire bucket...

It is required. My guess is that leaving the field blank is translating to an empty target prefix, which is valid. A null target prefix is not valid. This should be better documented.

[mike-bailey]

My guess is that leaving the field blank is translating to an empty target prefix

Thank you. That makes perfect sense in hindsight.

Still learning as a measly intern, so I appreciate the patience. Thank you!

[awood45]

No worries, this is a very unhelpful error message so your frustration is understood and appreciated. We're pushing for a better user experience here.

[ddimmich]

In the context of logstash which uses this library - how would one add the target prefix parameter? Is this something that needs to be addressed in logstash or in this library? Thank you

[awood45]

That would be added in the caller - logstash. Unless logstash plumbs through your own parameters, in which case you would need to add it there.

[ddimmich]

Ok - Thanks.  Not sure this is one of the parameters that would get
passed through - haven't seen a configuration equivalent so it sounds
like this would need to be fixed upstream :/

On 04/01/2018 11:30, Alex Wood wrote:
>

That would be added in the caller - logstash. Unless logstash plumbs
through your own parameters, in which case you would need to add it there.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-ruby/issues/1577#issuecomment-355216092,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAD87kPOqqz6SopXP8EoWTjSNDiSwlOvks5tHH4FgaJpZM4PAoW0.

[ramd123]

I was facing the same issue while setting analytics configuration using aws java sdk, this should have fixed Amazon by now. any solution to skip filter/predicate while creating analytics using java sdk?

if i add Filter with Predicate it works correct but i dont want add the predicate/prefix to the configuration.

[mike-bailey]

Did you do a blank one like @awood45 suggests in this issue?

[ramd123]

no, blank also not working.. same exception with blank as well.

[mike-bailey]

Maybe post the code if you can?

[ramd123]

private void manageAnalytics(String name, AmazonS3 s3) {

    if (name.equalsIgnoreCase("abc-files")) {
        SetBucketAnalyticsConfigurationRequest req = new SetBucketAnalyticsConfigurationRequest();
        req.setBucketName(name);
        AnalyticsS3BucketDestination s3destination = new AnalyticsS3BucketDestination()
                .withBucketAccountId("12345").withBucketArn("arn:aws:s3:::test-lifecycle")
                .withFormat(AnalyticsS3ExportFileFormat.CSV);
        AnalyticsExportDestination destination = new AnalyticsExportDestination()
                .withS3BucketDestination(s3destination);
        StorageClassAnalysisDataExport dataExport = new StorageClassAnalysisDataExport()
                .withDestination(destination).withOutputSchemaVersion("V_1");
        AnalyticsConfiguration analyticsConfiguration = new AnalyticsConfiguration()
                .withId(name + "-analytics-conf-2")
                .withStorageClassAnalysis(new StorageClassAnalysis().withDataExport(dataExport))
                .withFilter(new AnalyticsFilter().withPredicate(new AnalyticsPrefixPredicate("123456")));
        req.setAnalyticsConfiguration(analyticsConfiguration);

        s3.setBucketAnalyticsConfiguration(req);

    }
}

[awood45]

@ramd123 I'd recommend opening an issue on the AWS SDK for Java repo - hard for me to debug any edge cases you may be running in to with that tool.

[ramd123]

Thanks, I have created the Issue - https://github.com/aws/aws-sdk-java/issues/1613

[eric-swann-q2]

Thanks be to this thread! This issue also cropped up for me when setting up bucket replication. The lack of a prefix causes this exception as well, which is super unhelpful. Thanks!

[pcchannz]

@eric-swann-q2 I have the same the problem for setting bucket replication too but via c# sdk. The workaround i have is to add empty filter, priority and DeleteMarkerReplication.

[rtrscience]

@eric-swann-q2 I have the same the problem for setting bucket replication too but via c# sdk. The workaround i have is to add empty filter, priority and DeleteMarkerReplication.

Can you please elaborate the work around for the c# version of this error..

[mullermp]

@rtrscience If you are having trouble, feel free to open an issue on their SDK: https://github.com/aws/aws-sdk-net

[odykyi]

I have the same issue

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>MalformedXML</Code>
    <Message>The XML you provided was not well-formed or did not validate against our published schema</Message>
    <RequestId>9Er245g2B531</RequestId>
    <HostId>3iTCg2o9sft8G5245gewgetg23g23gECfSIjDxt7wlI8mHVvo=</HostId>
</Error>

[NeelAurea]

The recommendation mentioned here worked for me. I am attempting to assign transition policies to the buckets using Python SDK. Using an empty string for 'Prefix' in the 'Filter' section did the trick.