Boto3: [feature request] Delete non-empty bucket

[matt-telstra]

When I try to delete a bucket, I get the following error message:

botocore.exceptions.ClientError: An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty

I know there are objects in the bucket. When I tell boto to delete a bucket, I expect the objects to be deleted too.

I understand there are circumstances where users might like this warning. So what I propose is an option for the client.delete_bucket() function, like:

client.delete_bucket(Bucket='mybucketname',DeleteIfNonEmpty=True)

Where DeleteIfNonEmpty is False by default.

Then if DeleteIfNonEmpty is True, boto seamlessly deletes the whole bucket, just like deleting a directory in a file browser.

The alternative is for the user to

• manually list every object in the bucket;
• manually delete every object in the bucket; then
• actually delete the bucket

This is a lot of work for something which takes 2 clicks in the web console. Boto should handle this work for me.

[stealthycoin]

That would be a feature request for the service team. We only wrap their API in a python interface, if you would like to request a change to the underlying API you should open a ticket.

[matt-telstra]

Yes, I intend to open a ticket with AWS. But that could take months to actually get implemented, and I strongly doubt that they'll actually fix this ever.

What I'm proposing is that boto fixes this poor design decision by doing the deletion for the user.

I'm going to be writing a wrapper in my code anyway. I figure I may as well put that same code into a pull request to boto, to save other people from wasting time rewriting the same code as me. I'm willing to do that work myself, but I can't seem to find where the code for delete_bucket is actually implemented. A quick grep shows that delete_bucket only appears in directories for docs or tests. Which file is the code for delete_bucket in?

[matt-telstra]

Here's the code I've written as a workaround. I propose that we just integrate this into boto. If someone can me point to the right location in the repo, I'll make a pull request.

def delete_bucket_completely(bucket_name):

    client = boto3.client('s3')

    response = client.list_objects_v2(
        Bucket=bucket_name,
    )

    while response['KeyCount'] > 0:
        print('Deleting %d objects from bucket %s' % (len(response['Contents']),bucket_name))
        response = client.delete_objects(
            Bucket=bucket_name,
            Delete={
                'Objects':[{'Key':obj['Key']} for obj in response['Contents']]
            }
        )
        response = client.list_objects_v2(
            Bucket=bucket_name,
        )

    print('Now deleting bucket %s' % bucket_name)
    response = client.delete_bucket(
        Bucket=bucket_name
    )

[matt-telstra]

Apparently the aws client has a --force option which does this. link

Does anyone know how the client does this? Does it pass the force option to the api somehow, or does the client do the wraparound that I proposed?

[joguSD]

The delete_bucket function doesn't exist in any file. It's dynamically generated at runtime based on the available operations from botocore.

For your --force question, it's implemented as the latter: doing multiple operations for you. If you're curious you can see the implementation of that in the aws cli here: s3/subcommands.py.

I'm not sure if it would be better implement this feature in boto or s3transfer. Do take note of the readme on s3transfer as it's not entirely stable yet and there may be breaking changes in the future.

[matt-telstra]

Ok, that explains things. I suspected this was the case after I couldn't find the code I expected in the repo. So then I looked at the Development part of the README to confirm or disprove this hunch. The README doesn't mention this at all, so I then assumed this wasn't the case.

May I suggest mentioning this in the Development part of the README? Something this huge and unexpected makes it hard for people to contribute or understand the code. A mention in the README would fix that.

[matt-telstra]

Also, the README for s3transfer says to use boto instead of s3transfer.

[joguSD]

Yes, if you want to ultimately use the functionality in s3tranfser we recommend using the interface exposed through boto as those will remain stable, where as the underlying s3transfer interfaces may not. However, higher level s3 logic could theoretically be added there.

We try not to add too much high level logic on top of API calls as that creates feature disparity between this SDK and the ones for other languages. To be blunt, a proper implementation of it would be more complex than the snippet above and realistically I don't see this feature being incorporated in at the SDK level. This truly is a request for the service team.

For reference, a quick way to delete all (non-versioned) objects in a bucket using resource models:

import boto3

bucket = boto3.resource('s3').Bucket('bucketname')
bucket.objects.all().delete()
bucket.delete()

There is also a versioned objects collector that would allow you to do something similar for versioned objects, but do be careful as that may end up making a lot of requests.