Aws-sdk-java: After deleteBucket, two distinct AmazonS3Clients return different results on doesBucketExistV2

[igalratiner]

On single test repetition out of 100 test repetitions I got an odd behaviour:
There 2 instances of AmazonS3Client, one client deletes the bucket and checks (doesBucketExistV2 -> false) and later runs code which contains another distinct instance of AmazonS3Client which checks (doesBucketExistV2 -> true) on same bucket name.

Is there some caching issue that my explain it? Is there a better practice to achieve this functionality with different AmazonS3Client api calls?

reproduction code:

`public class AmazonS3ClientTest {

private static Logger logger = LoggerFactory.getLogger(AmazonS3ClientTest.class);

@Test
public void testAmazonS3Client() {
    S3BucketConfiguration bucketConfiguration = new S3BucketConfiguration();

    AmazonS3 amazonS3ClientBucketCreator = createAmazonS3Client(bucketConfiguration);

    String newBucketName = "dev-" + UUID.randomUUID().toString().toLowerCase();
    CreateBucketRequest createBucketRequest = new CreateBucketRequest(newBucketName);
    createBucketRequest.setObjectLockEnabledForBucket(false);
    amazonS3ClientBucketCreator.createBucket(createBucketRequest);
    Failsafe.with(new RetryPolicy()
            .retryIf(result -> !((boolean) result))
            .withDelay(1, TimeUnit.SECONDS)
            .withMaxRetries(120)
    )
            .onFailedAttempt(e -> logger.info("bucket not created yet", e))
            .onSuccess(result -> logger.info("bucket created now"))
            .get(() -> amazonS3ClientBucketCreator.doesBucketExistV2(newBucketName));


    AmazonS3 amazonS3ClientBucketRemover = createAmazonS3Client(bucketConfiguration);
    amazonS3ClientBucketRemover.deleteBucket(newBucketName);
    Failsafe.with(new RetryPolicy()
            .retryIf(result -> !((boolean) result))
            .withDelay(1, TimeUnit.SECONDS)
            .withMaxRetries(120)
    )
            .onFailedAttempt(e -> logger.info("bucket not deleted yet", e))
            .onSuccess(result -> logger.info("bucket deleted now"))
            .get(() -> !amazonS3ClientBucketRemover.doesBucketExistV2(newBucketName));


    AmazonS3 amazonS3ClientBucketRemovedValidator = createAmazonS3Client(bucketConfiguration);
    assertThat(amazonS3ClientBucketRemovedValidator.doesBucketExistV2(newBucketName)).isFalse();
}

private static AmazonS3 createAmazonS3Client(S3BucketConfiguration bucketConfiguration) {
    BasicAWSCredentials credentials = new BasicAWSCredentials(bucketConfiguration.accessKey, bucketConfiguration.secretKey);
    return AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(credentials))
            .withRegion(bucketConfiguration.bucketRegion)
            .build();
}

static class S3BucketConfiguration {
    final String secretKey;
    final String accessKey;
    final Regions bucketRegion;

    public S3BucketConfiguration(String secretKey, String accessKey, Regions bucketRegion) {
        this.secretKey = secretKey;
        this.accessKey = accessKey;
        this.bucketRegion = bucketRegion;
    }

    public S3BucketConfiguration() {
        this("Foo","Bar", Regions.Foobar);
    }

}`

Failed about 30 times out of 900 retries on two following excpetions, the first might be explained by some asynchronous code, but the second one is the critical one which was addressed above: one client returns the bucket doesn't exist and then the other one returns it does.

1. Failed on

amazonS3ClientBucketRemover.deleteBucket(newBucketName)

:

com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: XXXXX; S3 Extended Request ID: XXXXX), S3 Extended Request ID: XXXXX
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
at com.amazonaws.services.s3.AmazonS3Client.deleteBucket(AmazonS3Client.java:1651)
at com.amazonaws.services.s3.AmazonS3Client.deleteBucket(AmazonS3Client.java:1635)

1. Failed on

assertThat(amazonS3ClientBucketRemovedValidator.doesBucketExistV2(newBucketName)).isFalse()

:

org.junit.ComparisonFailure: expected:<[fals]e> but was:<[tru]e>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

SDK_VERSION=1.11.476, JRE_VERSION=1.8.0_151.

• minor dependency for retries on failures:
  Failsafe is a lightweight, zero-dependency library for handling failures in Java 8+, with a concise API for handling everyday use cases and the flexibility to handle everything else. It works by wrapping executable logic with one or more resilience policies, which can be combined and composed as needed.

[KaibaLopez]

Hi @igalratiner ,
The first thing that comes to mind is that the delete bucket may not be completed by the time you make the call on the instance, and so the instanced is locked from accessing the bucket, and that returns as "true":

if ((ase.getStatusCode() == Constants.BUCKET_REDIRECT_STATUS_CODE) || "AccessDenied".equals(ase.getErrorCode())) {
                return true;
            }

This might not be the case though, so I'll take a closer look, but in the mean time, and to make this faster, could you provide sample code (as small as possible) to reproduce this?

[igalratiner]

Hi @igalratiner ,
The first thing that comes to mind is that the delete bucket may not be completed by the time you make the call on the instance, and so the instanced is locked from accessing the bucket, and that returns as "true":

if ((ase.getStatusCode() == Constants.BUCKET_REDIRECT_STATUS_CODE) || "AccessDenied".equals(ase.getErrorCode())) {
                return true;
            }

This might not be the case though, so I'll take a closer look, but in the mean time, and to make this faster, could you provide sample code (as small as possible) to reproduce this?

thanks, i added code to original post for reproduction (which happened in my env about 30 out of 900 repetitions).

[debora-ito]

@igalratiner what you are experiencing is expected, because Amazon S3 offers eventual consistency for overwrite PUTS and DELETES.

For more info about S3 Data Consistency Model, please check: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel

According to the docs:

Amazon S3 achieves high availability by replicating data across multiple servers within AWS data centers. If a PUT request is successful, your data is safely stored. However, information about the changes must replicate across Amazon S3, which can take some time, and so you might observe the following behaviors:

• A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.
• A process replaces an existing object and immediately tries to read it. Until the change is fully propagated, Amazon S3 might return the previous data.
• A process deletes an existing object and immediately tries to read it. Until the deletion is fully propagated, Amazon S3 might return the deleted data.
• A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is fully propagated, Amazon S3 might list the deleted object.

[igalratiner]

@igalratiner what you are experiencing is expected, because Amazon S3 offers eventual consistency for overwrite PUTS and DELETES.

For more info about S3 Data Consistency Model, please check: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel

According to the docs:

Amazon S3 achieves high availability by replicating data across multiple servers within AWS data centers. If a PUT request is successful, your data is safely stored. However, information about the changes must replicate across Amazon S3, which can take some time, and so you might observe the following behaviors:

• A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.
• A process replaces an existing object and immediately tries to read it. Until the change is fully propagated, Amazon S3 might return the previous data.
• A process deletes an existing object and immediately tries to read it. Until the deletion is fully propagated, Amazon S3 might return the deleted data.
• A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is fully propagated, Amazon S3 might list the deleted object.

Thanks for the answer, works as expected. Does the sdk allows an alternative API the offers a deterministic result for PUT/DELETE requests or only the one i specified above?

[igalratiner]

@igalratiner what you are experiencing is expected, because Amazon S3 offers eventual consistency for overwrite PUTS and DELETES.

For more info about S3 Data Consistency Model, please check: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel

According to the docs:

Amazon S3 achieves high availability by replicating data across multiple servers within AWS data centers. If a PUT request is successful, your data is safely stored. However, information about the changes must replicate across Amazon S3, which can take some time, and so you might observe the following behaviors:

• A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.
• A process replaces an existing object and immediately tries to read it. Until the change is fully propagated, Amazon S3 might return the previous data.
• A process deletes an existing object and immediately tries to read it. Until the deletion is fully propagated, Amazon S3 might return the deleted data.
• A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is fully propagated, Amazon S3 might list the deleted object.

Also, after reading your comment again and reading the quoted consistency model, I noticed it explains consistency of objects inside s3 buckets but doesn't explain the model of buckets themselves - does all the model is eventually consistent or some of it read-after-write consistent like s3 objects in bucket? I'd like to hear a clarification about this

[debora-ito]

The same consistency model applies to S3 buckets, I believe.

An option to verify whether the bucket was successfully deleted is to use BucketNotExists waiter. Waiters are helpful in situations where you need to wait for a resource to transition into a desired state, like when working with eventually consistent services. You can read more about Waiters in this blog post: https://aws.amazon.com/blogs/developer/waiters-in-the-aws-sdk-for-java/

[AvishayHirsh]

@debora-ito correct me if i'm wrong but waiters is described as more convenient way to query for resource (in our use case s3bucket) does exists , if this still use the same API calls with just a better way to call it wouldnt we expirience the problem we described again ? because we already have polling logic on our end but still its not working because as we described 1 client with polling reports the resource exists , and then when we create another client it appears it doesnt ...

[AvishayHirsh]

@debora-ito hi still waiting for your response...