Rasterio: Optimize for "cloud optimized GeoTIFFs"

[sgillies]

Reference: https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF

By default, GDAL will try to scan for siblings of a file, which means that if your S3 bucket my-bucket contains 1000 objects rio info s3://my-bucket/wanted.tif will make 1000 HTTP requests before opening wanted.tif. To disable this the GDAL_DISABLE_READDIR_ON_OPEN config option can be set to True.

I'm proposing to make this option on by default when accessing objects on S3.

[glostis]

I am not sure if this issue is still alive, but I would like to add something to @sgillies 's comment above.

The use of GDAL_DISABLE_READDIR_ON_OPEN is not that obvious.

Consider the script below, doing rio info on a publicly available tiff file on AWS:

# This is the default behavior, it does the same thing as when GDAL_DISABLE_READDIR_ON_OPEN is not touched
CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=FALSE rio info /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
3

# The documentation (https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_DISABLE_READDIR_ON_OPEN) lets you think
# that this is a good option when dealing with servers that are slow to list files, but in fact it is not...
CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=TRUE rio info /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
10

# This is in fact the best option
CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR rio info /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
2

This issue was already raised by @vincentsarago in https://github.com/OSGeo/gdal/issues/909, where the answer seemed to be that we are misunderstanding the documentation? (IMO if everyone reads it incorrectly, it means that the documentation is not clear)

Additionally, I have seen that doing the same 3 commands as above with gdalinfo instead of rio info gives different numbers, but I guess this could come from the fact that both commands don't do exactly the same thing?

CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=FALSE gdalinfo /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
5

CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=TRUE gdalinfo /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
85

CPL_CURL_VERBOSE=YES GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR gdalinfo /vsis3/landsat-pds/L8/008/068/LC80080682015340LGN00/LC80080682015340LGN00_B3.TIF 2>&1 >/dev/null | grep "> GET" | wc -l
2

[sgillies]

@glostis there has been no action on this issue, which is a good thing, yes? If I understand, setting GDAL_DISABLE_READDIR_ON_OPEN=TRUE by default would lead to a performance regression. I'm going to resolve this as a wontfix if that's okay.

[glostis]

Yes it is a good thing not to have added GDAL_DISABLE_READDIR_ON_OPEN=TRUE as a default because it would indeed have been a regression.

The original intent of this issue was a good one though, because I have the feeling that the COG format is being used quite widely now, and new users are not necessarily aware of all the subtleties of the GDAL environment variables to get the best performance with their files.

[sgillies]

@glostis that is very true.