Rasterio: GDAL_DISABLE_READDIR_ON_OPEN affects how rasterio handles files on S3

[ungarj]

rasterio behaves differently towards remote files on S3 depening on whether GDAL_DISABLE_READDIR_ON_OPEN is activated or not. This could also affect #987.

example 1: access a non-existing file on a bucket requiring access token

expected error message without GDAL setting:

$ rio info s3://sentinel-s2-l1c/non_existing.tif
<...>
rasterio.errors.RasterioIOError: Access Denied

unexpected error message using GDAL setting:

$ export GDAL_DISABLE_READDIR_ON_OPEN=YES
$ rio info s3://sentinel-s2-l1c/non_existing.tif
<...>
rasterio.errors.RasterioIOError: '/vsis3/sentinel-s2-l1c/non_existing.tif' not recognized as a supported file format.

example 2: access a non-existing file on a public bucket requiring no access token

expected error message without GDAL setting:

$ rio info s3://test-gtiff/non_existing.tif
<...>
rasterio.errors.RasterioIOError: '/vsis3/test-gtiff/non_existing.tif' does not exist in the file system, and is not recognized as a supported dataset name.

unexpected error message using GDAL setting:

$ export GDAL_DISABLE_READDIR_ON_OPEN=YES
$ rio info s3://test-gtiff/non_existing.tif
<...>
rasterio.errors.RasterioIOError: '/vsis3/test-gtiff/non_existing.tif' not recognized as a supported file format.

I don't know whether this behavior is intentional but I would rather expect getting a consistent error message independent from GDAL_DISABLE_READDIR_ON_OPEN.

Very cool however would be if a generic FileNotFoundError or RasterioFileNotFoundError is raised so I can avoid hacks like this :)

[sgillies]

@ungarj on the other hand, the exception type doesn't vary, which would be a big problem.

Rasterio's behavior here followed Python 2.7 where opening a non-existent file results in an IOError:

$ python2.7
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> open("lolwut.tif")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'lolwut.tif'

In the latest pythons we get a FileNotFoundError, which I agree is much more clear. I'd be in favor of that in a future version.

[sgillies]

I've concluded the fix for this belongs in GDAL, which should set an access denied error in both cases instead of only warning about denial of access in the GDAL_DISABLE_READDIR_ON_OPEN=YES case.

[sgillies]

Upstream, Even agrees that this is something to fix in GDAL. I'll take the lead on that.

[ungarj]

Very cool, thanks @sgillies!