Rasterio: Add flush method to writer

[sgillies]

Calling close() flushes buffered data to storage, but not being able to flush when we want to flush is a missing feature or bug 🚽😳

Current workaround:

# Copy GeoTIFF block-by-block.
import rasterio


source_path = 'original.tif'
copy_path = '/tmp/copy.tif'

# Create but do not fill the copy.
with rasterio.open(source_path) as src:
    with rasterio.open(copy_path, 'w', **src.profile) as dst:
        pass

# Copy data over, flushing each block.
with rasterio.open(source_path) as src, rasterio.open(copy_path, 'r+') as dst:
    for ij, window in dst.block_windows():
        data = src.read(window=window)
        with rasterio.open(dst.name, 'r+') as writer:
            writer.write(data, window=window)

In the future:

import rasterio


source_path = 'original.tif'
copy_path = '/tmp/copy.tif'

with rasterio.open(source_path) as src:
    with rasterio.open(copy_path, 'w', **src.profile) as writer:
        for ij, window in dst.block_windows():
            data = src.read(window=window)
            writer.write(data, window=window)
            writer.flush()

[perrygeo]

Like Python's open function, would it be possible to add an optional buffer kwarg that would allow an explicit configurable buffer size in addition to manual flush?

[sgillies]

@perrygeo hmm, not sure, I don't know if we'll really know how many bytes GDAL has buffered.

[perrygeo]

Looks like the buffer size can be controlled by the GDAL_CACHEMAX config option.

Setting it to 100k (the lowest allowable value it appears) causes very frequent disk writes:

Setting to 1M defers the IO until we're part way through

Setting to 5M (greater than the size of the RGB.byte.tif test image) means IO only occurs when we close the dataset.

(see the gist for code)

The good news is that GDAL_CACHEMAX has sensible defaults; 5% of available memory should be a safe bet for most use cases to avoid unexpected memory errors. But it does mean that the IO behavior is not deterministic. Depending on the user's machine or their GDAL version, the IO could either happen within the block window loop and/or on close.

[geowurster]

It looks like @perrygeo determined it is not practical, but I'm _very_ 👎 on hiding GDAL_CACHEMAX behind a buffer option similar to open(buffer=..). It is an environment level option (#1270) that affects all GDAL I/O, including Rasterio's use of in-memory rasters to translate Numpy arrays to GDAL datasets. Additionally, GDAL_CACHEMAX is only:

consulted the first time the cache size is requested overriding the initial default (40MB up to GDAL 2.0, 5% of the usable physical RAM starting with GDAL 2.1)

Altering the value requires going through GDALGetCacheMax() and GDALSetCacheMax(), and their 64 bit counterparts.

I have occasionally found use for the flush methods in the OSGeo bindings, but there are some driver-level considerations. GDAL caches at both the GDALRasterBand and GDALDataset level. Calling either object's FlushCache() method flushes known data, but a given driver may also sync some, all, or no data in GDALClose(). A driver may also override the flush methods and defer all I/O until GDALClose(). It seems like Rasterio should flush all bands and then the dataset. I see no reason for this to limit an implementation in Rasterio, but it seems like something that should be documented and communicated to the user.

[sgillies]

@geowurster good point. Instead of touching the block cache size, the thing for a writer to do is track the number of pixels written and flush when the buffering limit has been reached.

[geowurster]

@sgillies are you suggesting src.write() would call src.flush()?

[sgillies]

Yes, sometimes, like https://docs.python.org/3/library/io.html#io.BufferedWriter.