Aws-sdk-ruby: Filter element can only be used in Lifecycle V2, when updating lifecycle policy

[SamSaffron]

Per a previous discussion we moved away from using deprecated put_bucket_lifecycle towards using put_bucket_lifecycle_configuration

Trouble is we now start seeing this in tons of s3 buckets we have

Filter element can only be used in Lifecycle V2, when updating lifecycle policy

/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/client.rb:4442:in `put_bucket_lifecycle_configuration'
/var/www/discourse/lib/s3_helper.rb:109:in `update_lifecycle'
/var/www/discourse/lib/s3_helper.rb:118:in `update_tombstone_lifecycle'

• Is there any way to upgrade all buckets to v2?

• Should put_bucket_lifecycle_configuration operate in some compatability mode?

[cjyclaire]

Sorry for the late response, could you try contact Aws Support for a directly conversation with service team? Since this is an server side response error, they will have more knowledge than us. They can also help look into your request IDs to see if anything goes wrong.

[rboy1]

@cjyclaire what's the solution to this? Is there a way to reach Amazon support and report the issue - they don't have support for regular accounts?

[cjyclaire]

I contacted service team internally with those info, will update

[cjyclaire]

So to clarify a bit, there is no V1 and V2 bucket concept, you just need to make sure you are using the put_bucket_lifecycle_configuration operation instead of put_bucket_lifecycle for all targeting buckets. If you still see an issue, could you try setting :http_wire_trace to true to give us a full http wire trace for debugging?

[SamSaffron]

I see, well we do have an actual bug here I think in the aws gem:

get_bucket_lifecycle_configuration is returning

[#<struct Aws::S3::Types::LifecycleRule expiration=#<struct Aws::S3::Types::LifecycleExpiration date=nil, days=30, expired_object_delete_marker=nil>, id="purge-tombstone", prefix="tombstone/", filter=nil, status="Enabled", transitions=[], noncurrent_version_transitions=[], noncurrent_version_expiration=nil, abort_incomplete_multipart_upload=nil>]

Doing a to_h on this thing returns:

{:expiration=>{:days=>30}, :id=>"purge-tombstone", :prefix=>"tombstone/", :status=>"Enabled"}

When it should really be:

{:expiration=>{:days=>30}, :id=>"purge-tombstone", :filter => {:prefix=>"tombstone/" }, :status=>"Enabled"}

So this ends up triggering the v2 error.

[SamSaffron]

https://github.com/discourse/discourse/commit/4f28c71b5082d8194129f10084738775b36b8ed3

hope this helps narrow down the issue.

[cjyclaire]

@SamSaffron Hmm I cannot reproduce by:

1. using #put_bucket_lifecycle_configuration(opts)
2. using #get_bucket_lifecycle_configuration-instance_method

The response echo the correct format as expected:

=> #<struct Aws::S3::Types::GetBucketLifecycleConfigurationOutput
 rules=
  [#<struct Aws::S3::Types::LifecycleRule
    expiration=#<struct Aws::S3::Types::LifecycleExpiration date=nil, days=3650, expired_object_delete_marker=nil>,
    id="TestOnly",
    prefix=nil,
    filter=#<struct Aws::S3::Types::LifecycleRuleFilter prefix="documents/", tag=nil, and=nil>,
    status="Enabled",
    transitions=[#<struct Aws::S3::Types::Transition date=nil, days=365, storage_class="GLACIER">],
    noncurrent_version_transitions=[],
    noncurrent_version_expiration=nil,
    abort_incomplete_multipart_upload=nil>]>

Curious and guessing, are you using #put_bucket_lifecycle(the deprecated one) to put configuration originally?

[SamSaffron]

Curious and guessing, are you using #put_bucket_lifecycle(the deprecated one) to put configuration originally?

Yes, absolutely, the issue only existed on legacy systems that were previously configured with put_bucket_lifecycle.

The repro, in theory, should be to put_bucket_lifecycle then get_bucket_lifecycle_configuration and put_bucket_lifecycle_configuration.

[cjyclaire]

I believe that might be the issue from those API coupling, I'll talk to service team for confirmation and best practice for this.

From SDK side, we just parse the response we received from server side for the API you call and we have no ability to change the logic at server side.

[SamSaffron]

thanks heaps @cjyclaire, nothing urgent on our side cause we have a
workaround in place already.

On Tue, Nov 14, 2017 at 8:09 AM, cjyclaire [e-mail] wrote:

I believe that might be the issue from those API coupling, I'll talk to
service team for confirmation and best practice for this.

From SDK side, we just parse the response we received from server side for
the API you call and we have no ability to change the logic at server side.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-ruby/issues/1653#issuecomment-344058790,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAUXQDvTFd84V3v4E8s34lb31c1VLufks5s2LAigaJpZM4QL2Uw
.

[cjyclaire]

Summary: get_bucket_lifecycle is used with put_bucket_lifecycle, and likewise, get_bucket_lifecycle_configuration is used with put_bucket_lifecycle_configuration. Mixing the usage will throw an error there.

Thus if you want to use get_bucket_lifecycle_configuration for bucket that has lifecycle config with put_bucket_lifecycle, the simplest way is to delete that first with delete_bucket_lifecycle and create a new one with put_bucket_lifecycle_configuration before calling get.

Closing, yet feel free to reopen with further questions : )

[jrkarnes]

Summary: get_bucket_lifecycle is used with put_bucket_lifecycle, and likewise, get_bucket_lifecycle_configuration is used with put_bucket_lifecycle_configuration. Mixing the usage will throw an error there.

Thus if you want to use get_bucket_lifecycle_configuration for bucket that has lifecycle config with put_bucket_lifecycle, the simplest way is to delete that first with delete_bucket_lifecycle and create a new one with put_bucket_lifecycle_configuration before calling get.

Closing, yet feel free to reopen with further questions : )

Even in 2020 this is still relevant. Thank you for being so thorough! Saved me research time and kept me productive.