Smart_open: Specifying a GCS directory creates an empty file with no error

[polm]

Problem description

Behavior with directory paths in GCS is surprising and not documented.

Steps/code to reproduce the problem

Suppose that this path references a directory in GCS:

gs://foo/bar

That url will fail with a 404. However, this URL will succeed:

gs://foo/bar/

It will result in an empty file instead of downloading the contents of the directory.

Since smart_open is designed to work on files I understand that downloading directories doesn't make sense, but it seems like throwing an error would be more appropriate than creating an empty file.

I checked the docs but didn't see any mention of downloading directories. For comparison, with a tool like gsutil there is a cp -r that works as you would expect.

Versions

Linux-5.11.9-arch1-1-x86_64-with-arch
Python 3.7.9 (default, Mar  6 2021, 22:28:38) 
[GCC 10.2.0]
smart_open 3.0.0

Checklist

Before you create the issue, please make sure you have:

• [x] Described the problem clearly
• [x] Provided a minimal reproducible example, including any required data
• [x] Provided the version numbers of the relevant software

[mpenkov]

@petedannemann Are you able to have a look at this?

[mpenkov]

@polm Can you try reproducing the problem with the latest smart_open (5.0.0)?

[petedannemann]

@polm please provide code examples so we can try to reproduce this

[polm]

Sorry for the delayed reply, here's test code:

from smart_open import open

path = "gs://gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/"

with open("outfile", "w") as ofile, open(path) as infile:
    ofile.write(infile.read())

However, with this sample URL it works differently from the way it does on my private bucket. On my private bucket this will create an empty file, even though the path is definitely a directory and ends in a slash. On the public GCP urls I have tried (from here), this gives an error:

Traceback (most recent call last):
  File "/mnt/pool/work/spacy/fiddle/smart_test.py", line 5, in <module>
    with open("outfile", "w") as ofile, open(path) as infile:
  File "/mnt/pool/work/spacy/env/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 222, in open
    binary = _open_binary_stream(uri, binary_mode, transport_params)
  File "/mnt/pool/work/spacy/env/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 324, in _open_binary_stream
    fobj = submodule.open_uri(uri, mode, transport_params)
  File "/mnt/pool/work/spacy/env/lib/python3.9/site-packages/smart_open/gcs.py", line 105, in open_uri
    return open(parsed_uri['bucket_id'], parsed_uri['blob_id'], mode, **kwargs)
  File "/mnt/pool/work/spacy/env/lib/python3.9/site-packages/smart_open/gcs.py", line 135, in open
    fileobj = Reader(
  File "/mnt/pool/work/spacy/env/lib/python3.9/site-packages/smart_open/gcs.py", line 220, in __init__
    raise google.cloud.exceptions.NotFound('blob %s not found in %s' % (key, bucket))
google.api_core.exceptions.NotFound: 404 blob LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/ not found in gcp-public-data-landsat

Not sure why the paths would give different results.

This behavior seems to be the same with smart_open 3.0.0 and 5.0.0.

[petedannemann]

@polm, thanks for the example. I think the current behavior is as expected and some of this issue comes from a misunderstanding of "directories" with Google Cloud Storage / object storage in general. Object storage does not have directories. Taken from this great SO post:

Google Cloud Storage objects are a flat namespace, but many tools, including gsutil and the Google Cloud Storage UI, create an illusion of a hierarchical file tree.

Therefore, gs://my_bucket/my_prefix/ does not actually represent a directory, but an object. gsutil just has a hack in place that when an object ends with / and is used with the -r option it lists files from the prefix before /.

You are comparing behavior from gsutil and smart-open which are completely different tools with completely different intents.

If you want to open many files, you should list blobs using the google-cloud-storage client and then read them individually.

from google.cloud import storage

from smart_open import open

client = storage.Client()
bucket = client.get_bucket("my-bucket")
prefix = "my_prefix"
blobs = list(client.list_blobs(bucket, prefix=prefix))

for blob in blobs:
    with open(f"gs://{bucket}/{blob.name}") as f:
        do_something(f)

[mpenkov]

@petedannemann Your description of flat object storage is valid, but I think the unexpected part is that the user is opening something for reading, and an object is getting created as a side-effect. That side effect is unexpected (reading should never create new objects).

@polm Is my summary of the problem (see above) correct?

[polm]

@mpenkov I actually think @petedannemann got this right - my understanding of object storage was lacking, and I guess this behavior is consistent. I don't think the reading created a new object. When I said it created an empty file, I mean on local disk.

My private bucket in this example was created through the Google Cloud Storage UI. I created a "folder" (what the UI calls it) and put files in it. So I guess it exists as an empty object.

I am still confused about the behavior of my private bucket vs the public bucket. Should I assume the public bucket is using method 2 described in the Stack Overflow post, and therefore no empty object with the folder name exists, which is why it gives a 404? Is there some way to confirm that for buckets I don't own?

Also, thanks for the list_blobs example, that's very helpful. It might be a good idea to have that in the README.

In any case it looks like this is not a bug in smart_open, and just a consequence of lack of understanding of object storage. Thanks for clarifying it for me!

[mpenkov]

It might be a good idea to have that in the README.

Are you able to make a PR?

[petedannemann]

Thanks for the PR @polm. @mpenkov shall we close this issue then?