Rasterio: segfault from windowed boundless read of VRT with shared="0" set

[tovogt]

Expected behavior and actual behavior.

I expected to read a windowed (important!) band with boundless=True (important!) from a VRT dataset that has the shared="0" (important!) option set and a segmentation fault occurred.

Steps to reproduce the problem.

Execute this script:

import rasterio
import rasterio.windows
from rasterio.transform import Affine
import numpy as np

out_meta = {
    'driver': 'GTiff',
    'width': 10,
    'height': 10,
    'count': 1,
    'dtype': 'int8',
    'crs': '+proj=latlong',
    'transform': Affine.scale(2,-2),
}
with rasterio.open("data.tif", "w", **out_meta) as f:
    f.write_band(1, np.zeros((10,10), dtype=np.int8))

with open("data.vrt", "w") as f:
    f.write("""<VRTDataset rasterXSize="10" rasterYSize="10">
  <SRS dataAxisToSRSAxisMapping="2,1">GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]</SRS>
  <GeoTransform>  0.0000000000000000e+00,  2.0000000000000000e+00,  0.0000000000000000e+00,  0.0000000000000000e+00,  0.0000000000000000e+00, -2.0000000000000000e+00</GeoTransform>
  <VRTRasterBand dataType="Byte" band="1">
    <ColorInterp>Gray</ColorInterp>
    <SimpleSource>
      <SourceFilename relativeToVRT="1" shared="0">data.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="10" RasterYSize="10" DataType="Byte" BlockXSize="10" BlockYSize="10" />
      <SrcRect xOff="0" yOff="0" xSize="10" ySize="10" />
      <DstRect xOff="0" yOff="0" xSize="10" ySize="10" />
    </SimpleSource>
  </VRTRasterBand>
</VRTDataset>""")

for i in range(10000):
    with rasterio.open("data.vrt", 'r') as cdem:
        win = rasterio.windows.Window(col_off=0, row_off=0, width=1, height=1)
        cdem.read(indexes=1, window=win, boundless=True)
    print(i, end="\r")

After 1000-2000 reads, execution will be interrupted due to a segmentation fault. Here is a backtrace:

$ gdb python
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.
(gdb) run test.py
Starting program: /home/tovogt/.local/share/miniconda3/envs/dem/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffea032700 (LWP 4306)]
[New Thread 0x7fffe9831700 (LWP 4307)]
[New Thread 0x7fffe5030700 (LWP 4311)]
[New Thread 0x7fffe482f700 (LWP 4316)]
[New Thread 0x7fffe002e700 (LWP 4317)]
[New Thread 0x7fffdf82d700 (LWP 4323)]
[New Thread 0x7fffdb02c700 (LWP 4324)]
54
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff58367d8 in GDALDatasetPool::_CloseDataset(char const*, GDALAccess, char const*) () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
(gdb) backtrace
#0  0x00007ffff58367d8 in GDALDatasetPool::_CloseDataset(char const*, GDALAccess, char const*) () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#1  0x00007ffff5836d85 in GDALDatasetPool::CloseDataset(char const*, GDALAccess, char const*) () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#2  0x00007ffff5836eb6 in GDALProxyPoolDataset::~GDALProxyPoolDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#3  0x00007ffff5836eea in GDALProxyPoolDataset::~GDALProxyPoolDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#4  0x00007ffff57f9ae8 in GDALDataset::ReleaseRef() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#5  0x00007ffff577a037 in VRTSimpleSource::~VRTSimpleSource() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#6  0x00007ffff577a04a in VRTSimpleSource::~VRTSimpleSource() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#7  0x00007ffff577640f in VRTSourcedRasterBand::CloseDependentDatasets() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#8  0x00007ffff5776478 in VRTSourcedRasterBand::~VRTSourcedRasterBand() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#9  0x00007ffff57764ba in VRTSourcedRasterBand::~VRTSourcedRasterBand() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#10 0x00007ffff57ff2c8 in GDALDataset::~GDALDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#11 0x00007ffff576202a in VRTDataset::~VRTDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#12 0x00007ffff58363f2 in GDALDatasetPool::~GDALDatasetPool() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#13 0x00007ffff5836aa9 in GDALDatasetPool::Unref() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#14 0x00007ffff5836e76 in GDALProxyPoolDataset::~GDALProxyPoolDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#15 0x00007ffff5836eea in GDALProxyPoolDataset::~GDALProxyPoolDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#16 0x00007ffff57f9ae8 in GDALDataset::ReleaseRef() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#17 0x00007ffff577a037 in VRTSimpleSource::~VRTSimpleSource() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#18 0x00007ffff577a04a in VRTSimpleSource::~VRTSimpleSource() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#19 0x00007ffff577640f in VRTSourcedRasterBand::CloseDependentDatasets() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#20 0x00007ffff5776478 in VRTSourcedRasterBand::~VRTSourcedRasterBand() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#21 0x00007ffff57764ba in VRTSourcedRasterBand::~VRTSourcedRasterBand() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#22 0x00007ffff57ff2c8 in GDALDataset::~GDALDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#23 0x00007ffff576202a in VRTDataset::~VRTDataset() () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/../../../libgdal.so.26
#24 0x00007ffff6227a5a in __pyx_pw_8rasterio_5_base_11DatasetBase_15stop () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/_base.cpython-37m-x86_64-linux-gnu.so
#25 0x00007ffff626422e in __pyx_pw_8rasterio_5_base_11DatasetBase_17close () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/_base.cpython-37m-x86_64-linux-gnu.so
#26 0x00007ffff622af62 in __Pyx_PyObject_CallNoArg () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/_base.cpython-37m-x86_64-linux-gnu.so
#27 0x00007ffff626081b in __pyx_pw_8rasterio_5_base_11DatasetBase_21__exit__ () from /home/tovogt/.local/share/miniconda3/envs/dem/lib/python3.7/site-packages/rasterio/_base.cpython-37m-x86_64-linux-gnu.so
#28 0x000055555568ca55 in _PyMethodDef_RawFastCallDict () at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:515
#29 0x000055555568cbd1 in _PyCFunction_FastCallDict (func=0x7fffd5e9cfa0, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:586
#30 0x0000555555716bf1 in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:2998
#31 0x000055555566e6f9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3930
#32 0x000055555566f5f4 in PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3959
#33 0x000055555566f61c in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:524
#34 0x0000555555770974 in run_mod () at /tmp/build/80754af9/python_1578510683607/work/Python/pythonrun.c:1035
#35 0x000055555577acf1 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1578510683607/work/Python/pythonrun.c:988
#36 0x000055555577aee3 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1578510683607/work/Python/pythonrun.c:429
#37 0x000055555577bf95 in pymain_run_file (p_cf=0x7fffffffd910, filename=0x5555558b28e0 L"test.py", fp=0x5555558fc110) at /tmp/build/80754af9/python_1578510683607/work/Modules/main.c:434
#38 pymain_run_filename (cf=0x7fffffffd910, pymain=0x7fffffffda20) at /tmp/build/80754af9/python_1578510683607/work/Modules/main.c:1613
#39 pymain_run_python (pymain=0x7fffffffda20) at /tmp/build/80754af9/python_1578510683607/work/Modules/main.c:2874
#40 pymain_main () at /tmp/build/80754af9/python_1578510683607/work/Modules/main.c:3414
#41 0x000055555577c0bc in _Py_UnixMain () at /tmp/build/80754af9/python_1578510683607/work/Modules/main.c:3449
#42 0x00007ffff77e6b97 in __libc_start_main (main=0x5555556500a0 <main>, argc=2, argv=0x7fffffffdb78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdb68)
    at ../csu/libc-start.c:310
#43 0x0000555555724990 in _start () at ../sysdeps/x86_64/elf/start.S:103

I am aware that this script is very artificial. In my case, I was looping over the blocks of one file. In each iteration, I was making boundless windowed reads from other VRT datasets that have the shared="0" option set. The execution stopped after some time with a segmentation fault. After a while, I was able to boil it down to the above minimal example. My guess is, that there is a race condition somewhere in the garbage collection.

Operating system

Ubuntu 18.04

Rasterio version and provenance

Version 1.1.0 (py37h41e4f33_0) from conda.

[sgillies]

@tovogt thank you for the program. I can confirm that it's a problem and not just with the conda packaged rasterio.

A rasterio boundless read uses a VRT internally. I suspect there is disagreement about the sharing mode at the two VRT levels and I will look there. I get some confirmation of this hypothesis in seeing that your program will run to completion if I set VRT_SHARED_SOURCE=0 in the environment as described at https://gdal.org/drivers/raster/vrt.html#multi-threading-issues.

[sgillies]

Fixed by adbd848 and will be fixed in 1.1.3, coming soon.