Rasterio: DOC: Concurrent read/write docs update

[snowman2]

Would be good to update example in docs based on: https://rasterio.groups.io/g/main/topic/28593754

Also, in the concurrent processing docs it states:

Python threads can read and write concurrently.

However, it does not actually do the read/write concurrently in the example and in the linked discussion it says read/write is not thread safe. Do you have recommendations for thread safe reading/writing?

[ardieorden]

@snowman2 I can add a code block to concurrency.rst to show how to do concurrent window processing.

[mark-boer]

I have a question after reading the concurrent docs. What is the purpose of the rasterio.Env in the example? Perhaps this could use a comment?

Related the actual issue: Could this use an another look? I can open a PR.

[sgillies]

@snowman2 @ardieorden I spent some time today updating my concurrency example gist. What do you think about what's in https://gist.github.com/sgillies/b90a79917d7ec5ca0c074b5f6f4857e3 now? You can easily see the interleaving of read and writes by using a pair of print statements and you can also time the program and see an increase in throughput with number of threads. The program runs 6x as fast with 8 threads and 10x as fast with 16 threads. There's a small useful optimization as well: we limit the number of in-flight window tasks by using concurrent.futures.wait. This gist is derived from a rio-mbtiles PR at https://github.com/mapbox/rio-mbtiles/pull/54.

[sgillies]

@mark-boer without an Env(), GDAL errors don't go to the Python logger. Since that doc was written, we've changed rasterio so that there are implicit Envs for the contexts created by with rasterio.open() as x:. That particular Env can be deleted.

[snowman2]

@sgillies and @ardieorden, it looks like both of your examples allow for concurrent reading. However, I noticed that concurrent writing is not something that is done. Is concurrent writing something that should be avoided? What are some of the problems that arise when attempting to do concurrent writing?

[mark-boer]

@sgillies (thx for the answer on Env ;-)) is there a performance gain when you do the concurrent reading? The example in the link (I added it below) shared one filehandle (DatasetReader) with multiple threads, using a lock to prevent race conditions. I mean the creation of the additional filehandles (1 per window) should result in some overhead right?

Also what is the purpose of the chunking with islice? In the example below they just use executor.map.

Regarding this piece of documentation: It would also be nice if this section in the documentation could explain the usefulness of the sharing keyword argument that can be given to the rasterio.open function.

with rasterio.open(infile) as src:
    with rasterio.open(outfile, "w", **src.profile) as dst:
        windows = [window for ij, window in dst.block_windows()]

        read_lock = threading.Lock()
        write_lock = threading.Lock()

        def process(window):
            with read_lock:
                src_array = src.read(window=window)
            result = compute(src_array)
            with write_lock:
                dst.write(result, window=window)

        with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
            executor.map(process, windows)

[ardieorden]

If anyone else would like to work on this issue, please feel free to do so! cc @mark-boer @snowman2

I just copied the code block from the gist that @sgillies wrote and I put it in the documentation. I'm afraid I can't help answer the technical questions.

[sgillies]

@mark-boer the more I look at your example, the better it looks. I think we could annotate it a little more to make it clear that it is all about executing compute() concurrently. There's only one read and one write happening at any time, and depending on your hardware, perhaps only one disk operation at a time. And this example is totally fine and valuable: users need examples of concurrent raster processing that are easy and productive. Optimizing I/O performance, on the other hand, gets complicated quickly.

@snowman2 I haven't put any thought into whether concurrent writing helps a raster application. It will depend on the hardware (in the local disk case) and the size and structure of files (and probably the format driver in some cases). And on top of that (or between, I suppose) there is GDAL's raster band cache, its size, and whether we bypass it. This would be a good topic for a conference talk.

My next action: return to #2010 and put some actionable comments and suggestions in there and get it merged soon.

[mark-boer]

To be fair, the example was made entirely by Pablo Sanfilippo on the rasterio groups :wink:. I'll update your suggestions today.

@sgillies Would parallel writing / reading differ when you are using a different virtual file system, such as VSICURL or VSIS3?

[sgillies]

@mark-boer I don't have a good answer for that question. I think we should avoid putting such an example in the rasterio docs unless it comes from some actual running code with benchmarks and a thorough analysis of the problem space.

[snowman2]

Thanks 👍