gsutil -m rm on empty bucket fails with CommandException

[msigdel]

Running the following command fails with "CommandException: 1 files/objects could not be removed."

The error isn't meaningful as in fact the issue is that there are no files, instead it says 'objects could not be removed'. Also, should the bucket not having files raise an exception?

[houglum]

Ah, interesting. Our exception handler for rm doesn't treat "No URLs matched: " exceptions any differently from other exceptions; it assumes all captured exceptions were failures from trying to remove valid files/objects.

I agree that the error message here is misleading. And as for the exception: while nothing technically went wrong in deleting all the objects/files corresponding to the command arguments (even if this evaluates to 0), we run under the assumption that if a user says, "I want to delete gs://foo/bar," then they believe that object exists - this is just a nicety to inform them that the objects they assumed were present were, in fact, not.

[LivesleyWA]

Disagree - I might not know the state of a directory that I am clearing. I have an automated build process and the target directory may or may not have something in it depending on the previous build state.
In any case, the -f flag should ensure that the "error" is not reported. It is not working and I am still getting the error which kills my build process.

[andrelocher]

I'm encountering the same issue here, what's the workaround?

[houglum]

If you'd like to know whether the bucket is empty first, you could make a call to ls. The ls command doesn't currently provide a way to set max desired results, so if there will potentially be a lot of objects in the bucket, resulting in lots of listing calls, you could instead make a raw call to the JSON API's objects.list method. This would allow you to specify a value of 1 for the maxResults parameter, resulting in only one quick listing call. If you prefer the XML API, it offers this functionality via the max-keys option.

[shanFitGit]

I have similar issue. I wish this could be handled in a better way. We have automated build process where we clear the directory without knowing if it's empty or not. The error thrown by gsutil while trying to delete empty directory breaks the build process.

[houglum]

In the spirit of tracking how much user pain this causes, I saw another report of this behavior being confusing at
https://stackoverflow.com/questions/47885197/gsutil-cp-command-error-commandexception-no-urls-matched/47895858#47895858

[nojvek]

gsutil -m cp -n -r ../assets/* gs://foo/assets
CommandException: No URLs matched: ../asset-cache/*

This is like cp -r foo bar throwing an error if foo directory is empty.

Would love to see this edge case fixed

[pol1000000]

noticied strange behavior of
gsutil -m {mv, cp, rm} gs://some-bucket/dir1 gs://some-bucket/dir2
sometimes it still returns the error: "CommandException: count_of_files files/objects could not be copied\moved\removed" when I ran processing a lot of files.
Usually, count_of_files is a small count, the max that I noticed is 7.

The first that I thought is network troubles, but when I started to ping-tests and moving files as it shows the above, I just got 0% of the packet loss and the error with 2 not moved files =(

it's critical moment cause I using buckets as part of the CDN network and have a lot of builds of static content during the day, so, every error just ruined build processes on my test environment.

is there another way to fast managing files remotely in GC buckets instead of gsutil?

OS: Deb 9
google-cloud-sdk version: 232.0.0-0

$ cat .boto | grep -v "^#" | sed '/^$/d'
[Credentials]
[Boto]
https_validate_certificates = True
[GoogleCompute]
[GSUtil]
content_language = en
default_api_version = 2
[OAuth2]

gsutil version: 4.35
checksum: 3b22dd7820bcb962909f4f401010fb17 (OK)
boto version: 2.49.0
python version: 2.7.15+ (default, Nov 28 2018, 16:27:22) [GCC 8.2.0]
OS: Linux 4.9.0-6-amd64
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): ~/.boto,
gsutil path: /usr/lib/google-cloud-sdk/bin/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

[vinaygopalkrishnan]

Hi

We are using gsutil version: 4.38, and seem to be facing the same issue:

The command: gsutil rm -f
is throwing the error CommandException: 1 files/objects could not be removed.

Is this issue fixed or is there a work around

[nandosangenetto]

Is there a plan to make it work? How can we help?

[orn688]

It would be great to at least document this behavior. There's no indication in the cp documentation that running gsutil cp -r with an empty directory as the source will fail.

[abdurrahman]

In my case, I am using Airflow and I could not realize yet when i get this CommandException, because sometimes its working, sometimes it occurring error.
This could be a simple solution; gsutil -m rm -rf gs://bucket/sub_directory/ || true

[dozer-jang]

In my case,
In the GCE instance to which the default service account is attached
When gsutil -m option was used, there was no authority for GCS(default service account only has read only permission), but an error message was simply displayed as a CommandException. ( as CommandException: n files/objects could not be removed.)

Anyone who command in GCE, it would be good to try without -m option and check if 403 Access Denied appears.

[RezaRahmati]

This thread is started at 2017 and still issue

[vbeffa]

This might be a workaround:

touch foo && gsutil cp foo gs://${BUCKET_ID} && gsutil -m rm gs://${BUCKET_ID}/** && rm -f foo