Boto3: Set expiration for s3 bucket - MalformedXML error

[bhanukumar22]

Hi,

I have created s3 bucket using saltstack.
Im getting error when trying to set expiration policy, please let me know if im missing something.
Im trying the below command

salt-run salt.cmd boto_s3_bucket.put_lifecycle_configuration '[{"Expiration": {"Days": 1},"Status": "enabled","Prefix": "prefixstring"}]' region=us-east-1 key= keyid= -l debug

Error I'm getting:
An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema

Quick help would be appreciated.

[jamesls]

Can you show me a python code snippet? I thought maybe this was using the bucket resource but there is no put_lifecycle_configuration on a bucket object, so I'm not exactly sure what method you're trying to call. Here's the docs for put_bucket_lifecycle_configuration that might help: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.put_bucket_lifecycle_configuration

[bhanukumar22]

Sorry, my bad. Forgot to add the resource on which I was trying to execute. Original command was like this:
salt-run salt.cmd boto_s3_bucket.put_lifecycle_configuration tests3bucket '[{"Expiration": {"Days": 1},"Status": "enabled","Prefix": "prefixstring"}]' region=us-east-1 key= keyid= -l debug

[bhanukumar22]

Any update for the above?

[edgan]

The issue I found in my case is that boto3 doesn't require a prefix, but the api does. If I set am empty Prefix with Prefix: '', it worked for me.

response = client.put_bucket_lifecycle_configuration(
    Bucket='foobar123-log',
    LifecycleConfiguration={
        'Rules': [
            {
                'Expiration': {
                    'Days': 91,
                },
                'ID': '90-days-to-glacier',
                'Prefix': '',
                'Status': 'Enabled',
                'Transitions': [
                    {
                        'Days': 90,
                        'StorageClass': 'GLACIER'
                    },
                ],
                'NoncurrentVersionTransitions': [
                    {
                        'NoncurrentDays': 90,
                        'StorageClass': 'GLACIER'
                    },
                ],
                'NoncurrentVersionExpiration': {
                    'NoncurrentDays': 91
                }
            },
        ]
    }
)

[bhanukumar22]

Same error still:
An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema

I think, there is issue with the input provided from me. Could someone please help me identifying the issue

salt-run salt.cmd boto_s3_bucket.put_lifecycle_configuration s3buckettest '[{"Expiration":{"Days": 1},"Status":"enabled","Prefix":""}]' region=us-east-1 key= keyid= -l debug

[stealthycoin]

I don't really know what you are doing since that line you provided is not valid python code. I don't know what salt-run means. Is this on a command line? How/where are you running this line?

[edgan]

I am another salt user. The salt-run command is part of Saltstack, a configuration management tool. I was running into this error message in the same context as @bhanukumar22 .

[bhanukumar22]

Yes, I'm running this in a command line. Saltstack supports boto3 modules.

[mcarlton00]

Just to chime in: I'm seeing this as well, but I'm trying to run it from a state instead. The relevant code block:

test-s3:
  boto_s3_bucket.present:
    - name: test-s3
    - Bucket: test-s3.saltstack.example
    - LifecycleConfiguration:
      - Expiration:
          Days: 7
        ID: "expire storage"
        Status: "enabled"
    - profile: {{ profile }}

and the response when running:

[ERROR   ] Failed to create bucket: An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.
local:
----------
          ID: test-s3
    Function: boto_s3_bucket.present
      Result: False
     Comment: Failed to create bucket: An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.
     Started: 15:08:06.275110
    Duration: 746.378 ms
     Changes:

Edit: Including a link to the saltstack documentation about the topic: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.boto_s3_bucket.html#salt.states.boto_s3_bucket.present

[mcarlton00]

@bhanukumar22 I have the solution.

The "Status" argument MUST be capitalized.

ie: "Status": "enabled" will fail, but "Status": "Enabled" will succeed

[bhanukumar22]

I just tried with the same state file which you used. Still getting same error

test-s3:
boto_s3_bucket.present:
- name: test-s3
- Bucket: bhanu59
- LifecycleConfiguration:
- Expiration:
Days: 7
ID: "expire storage"
Status: "Enabled"
- region: us..
- key: ....
- keyid: ....

Got below error

master:

      ID: test-s3
Function: boto_s3_bucket.present
  Result: False
 Comment: Failed to update bucket: An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.
 Started: 06:00:02.964894
Duration: 2992.747 ms
 Changes:

[stealthycoin]

I would guess that your botocore model is out of date try updating botocore pip install -U botocore. I don't really have any particular knowledge of how salt works, but I assume it is picking up whatever version of botocore you have installed on your system.

[mcarlton00]

@bhanukumar22 sorry, this thread got caught up in an email filter and I just found it again. This is the salt state that I have functioning properly. I had to include both an empty prefix and the capitalized "Enabled"

test-s3:
  boto_s3_bucket.present:
    - name: test-s3
    - Bucket: test-s3.saltstack.example
    - LifecycleConfiguration:
      - Expiration:
          Days: 7
        ID: "expire storage"
        Prefix: ""
        Status: "Enabled"
    - profile: {{ profile }}

edit: Thanks @joguSD for the correction. Fingers were working faster than the brain

[joguSD]

@mcarlton00 Did you mean this?

Status: "Enabled"

@bhanukumar22 Were you able to resolve this issue or does an empty prefix and capital Enabled status work for you as well?

[bhanukumar22]

Finally worked.
I corrected the below things:
Prefix: ""
Status: "Enabled"

Thanks a lot @mcarlton00 for your suggestion, I was able to fix my long pending issue.

[joguSD]

@bhanukumar22 Awesome! Glad to hear that.

[alecbz]

I would say that the error message here is still a bit of a bug.

An error occurred (MalformedXML) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema

The above:

1. Says nothing about what actually went wrong, which makes it _very_ difficult to debug the problem.

2. Is confusing because the provided input was JSON, not XML.

   The fact that the JSON is being converted to XML and validated as XML under-the-hood is an implementation detail that shouldn't be exposed to the user. (When I first got this error I assumed the bug was that I was providing JSON to the aws CLI when it was expecting XML).

[itsjwala]

I can confirm we need Prefix in api call as suggested by @edgan but it will still fail if you have
both Filter > Prefix also in place. Work around for me was to remove Prefix and keep Filter > Prefix

....
"Prefix": "", <---- remove
      "Filter": {
        "Prefix": "dummy",  <----- keep
        "Tag": {
          "Key": "string",
          "Value": "string"
        },
...

^ excerpt json is from aws cli doc