Fiona: GPKG writing crasher

[sgillies]

Expected behavior and actual behavior.

Segmentation faults often occur when writing a small GeoPackage file.

$ python gpkg.py
Warning 6: dataset foo.gpkg does not support layer creation option ENCODING
Segmentation fault: 11

I don't immediately see a connection between the warning and the segfaut because the warning always occurs and the segfaults do not always occur.

The problem was reported by conda/conda-forge user @chryss at https://github.com/conda-forge/fiona-feedstock/issues/41 and I've been able to trigger it using the wheels on PyPI.

Steps to reproduce the problem.

gpkg.py below:

import fiona
schema = {'geometry': 'Point', 'properties': [('title', 'str')]}     
crs = {'init': 'EPSG:4326'}    
with fiona.open('foo.gpkg', 'w', layer='bar_foo', driver='GPKG',     
schema=schema, crs=crs) as d:    
    d.writerecords([{    
                    'geometry': {'type': 'Point', 'coordinates': [0.0, 0.0]},    
                    'properties': {'title': 'One'}}])    
    d.writerecords([{    
                    'geometry': {'type': 'Point', 'coordinates': [2.0, 3.0]},    
                    'properties': {'title': 'Two'}}])

Operating system

• Mac OS X 10.11.6 and Python 2.7 (@chryss)
• Mac OS X 10.12.3 and Python 3.6.0 (@sgillies)

Fiona version and provenance

1.7.5 wheel from PyPI: macosx and py36.

[sgillies]

Using lldb I found:

Warning 6: dataset foo.gpkg does not support layer creation option ENCODING
Process 18770 stopped
* thread #2: tid = 0x2709f7, 0x0000000103c80bca libgdal.20.dylib`OGRSpatialReference::Release() + 58, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x0000000103c80bca libgdal.20.dylib`OGRSpatialReference::Release() + 58
libgdal.20.dylib`OGRSpatialReference::Release:
->  0x103c80bca <+58>: callq  *0x8(%rcx)
    0x103c80bcd <+61>: jmp    0x103c80bd2               ; <+66>
    0x103c80bd2 <+66>: addq   $0x10, %rsp
    0x103c80bd6 <+70>: popq   %rbp
(lldb) up
frame #1: 0x0000000103cf5ecd libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn() + 77
libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn:
    0x103cf5ecd <+77>: addq   $0x10, %rsp
    0x103cf5ed1 <+81>: popq   %rbp
    0x103cf5ed2 <+82>: retq
    0x103cf5ed3 <+83>: nopw   %cs:(%rax,%rax)
(lldb) up
frame #2: 0x0000000103cf5ef5 libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn() + 21
libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn:
    0x103cf5ef5 <+21>: addq   $0x10, %rsp
    0x103cf5ef9 <+25>: popq   %rbp
    0x103cf5efa <+26>: retq
    0x103cf5efb <+27>: nopl   (%rax,%rax)
(lldb) up
frame #3: 0x0000000103cf5f18 libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn() + 24
libgdal.20.dylib`OGRGeomFieldDefn::~OGRGeomFieldDefn:
    0x103cf5f18 <+24>: jmp    0x103cf5f1d               ; <+29>
    0x103cf5f1d <+29>: movq   -0x20(%rbp), %rax
    0x103cf5f21 <+33>: movq   %rax, %rdi
    0x103cf5f24 <+36>: callq  0x103e313d0               ; symbol stub for: operator delete(void*)
(lldb) up
frame #4: 0x0000000103c76189 libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn() + 297
libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn:
    0x103c76189 <+297>: jmp    0x103c7618e               ; <+302>
    0x103c7618e <+302>: movl   -0x20(%rbp), %eax
    0x103c76191 <+305>: addl   $0x1, %eax
    0x103c76194 <+308>: movl   %eax, -0x20(%rbp)
(lldb) up
frame #5: 0x0000000103c761d5 libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn() + 21
libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn:
    0x103c761d5 <+21>: addq   $0x10, %rsp
    0x103c761d9 <+25>: popq   %rbp
    0x103c761da <+26>: retq
    0x103c761db <+27>: nopl   (%rax,%rax)
(lldb) up
frame #6: 0x0000000103c761f8 libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn() + 24
libgdal.20.dylib`OGRFeatureDefn::~OGRFeatureDefn:
    0x103c761f8 <+24>: jmp    0x103c761fd               ; <+29>
    0x103c761fd <+29>: movq   -0x20(%rbp), %rax
    0x103c76201 <+33>: movq   %rax, %rdi
    0x103c76204 <+36>: callq  0x103e313d0               ; symbol stub for: operator delete(void*)
(lldb) up
frame #7: 0x0000000103c762ad libgdal.20.dylib`OGRFeatureDefn::Release() + 61
libgdal.20.dylib`OGRFeatureDefn::Release:
    0x103c762ad <+61>: jmp    0x103c762b2               ; <+66>
    0x103c762b2 <+66>: addq   $0x10, %rsp
    0x103c762b6 <+70>: popq   %rbp
    0x103c762b7 <+71>: retq
(lldb) up
frame #8: 0x00000001039fe352 libgdal.20.dylib`OGRGeoPackageLayer::~OGRGeoPackageLayer() + 194
libgdal.20.dylib`OGRGeoPackageLayer::~OGRGeoPackageLayer:
    0x1039fe352 <+194>: jmp    0x1039fe357               ; <+199>
    0x1039fe357 <+199>: jmp    0x1039fe35c               ; <+204>
    0x1039fe35c <+204>: movq   -0x20(%rbp), %rax
    0x1039fe360 <+208>: addq   $0x98, %rax
(lldb) up
frame #9: 0x0000000103a05360 libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer() + 704
libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer:
    0x103a05360 <+704>: addq   $0x90, %rsp
    0x103a05367 <+711>: popq   %rbp
    0x103a05368 <+712>: retq
    0x103a05369 <+713>: movl   %edx, %ecx
(lldb) up
frame #10: 0x0000000103a05c95 libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer() + 21
libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer:
    0x103a05c95 <+21>: addq   $0x10, %rsp
    0x103a05c99 <+25>: popq   %rbp
    0x103a05c9a <+26>: retq
    0x103a05c9b <+27>: nopl   (%rax,%rax)
(lldb) up
frame #11: 0x0000000103a05ce8 libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer() + 24
libgdal.20.dylib`OGRGeoPackageTableLayer::~OGRGeoPackageTableLayer:
    0x103a05ce8 <+24>: jmp    0x103a05ced               ; <+29>
    0x103a05ced <+29>: movq   -0x20(%rbp), %rax
    0x103a05cf1 <+33>: movq   %rax, %rdi
    0x103a05cf4 <+36>: callq  0x103e313d0               ; symbol stub for: operator delete(void*)
(lldb) up
frame #12: 0x00000001039edfe9 libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset() + 489
libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset:
    0x1039edfe9 <+489>: jmp    0x1039edfee               ; <+494>
    0x1039edfee <+494>: jmp    0x1039edff3               ; <+499>
    0x1039edff3 <+499>: jmp    0x1039edff8               ; <+504>
    0x1039edff8 <+504>: movl   -0x1c(%rbp), %eax
(lldb) up
frame #13: 0x00000001039eef35 libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset() + 21
libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset:
    0x1039eef35 <+21>: addq   $0x10, %rsp
    0x1039eef39 <+25>: popq   %rbp
    0x1039eef3a <+26>: retq
    0x1039eef3b <+27>: nopl   (%rax,%rax)
(lldb) up
frame #14: 0x00000001039eef88 libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset() + 24
libgdal.20.dylib`GDALGeoPackageDataset::~GDALGeoPackageDataset:
    0x1039eef88 <+24>: jmp    0x1039eef8d               ; <+29>
    0x1039eef8d <+29>: movq   -0x20(%rbp), %rax
    0x1039eef91 <+33>: movq   %rax, %rdi
    0x1039eef94 <+36>: callq  0x103e313d0               ; symbol stub for: operator delete(void*)
(lldb) up
frame #15: 0x00000001035bd6b0 libgdal.20.dylib`GDALClose + 144
libgdal.20.dylib`GDALClose:
    0x1035bd6b0 <+144>: addq   $0x20, %rsp
    0x1035bd6b4 <+148>: popq   %rbp
    0x1035bd6b5 <+149>: retq
    0x1035bd6b6 <+150>: nopw   %cs:(%rax,%rax)
(lldb) up
frame #16: 0x0000000101b99eaf ogrext.cpython-36m-darwin.so`__pyx_pw_5fiona_6ogrext_7Session_7stop + 31
ogrext.cpython-36m-darwin.so`__pyx_pw_5fiona_6ogrext_7Session_7stop:
    0x101b99eaf <+31>: movq   $0x0, 0x10(%rbx)
    0x101b99eb7 <+39>: movq   0x2d24a(%rip), %rax       ; (void *)0x000000010023b770: _Py_NoneStruct
    0x101b99ebe <+46>: incq   (%rax)
    0x101b99ec1 <+49>: addq   $0x8, %rsp

When the writing session ends, OGRGeomFieldDefn::~OGRGeomFieldDefn() is called such that there's a crash in OGRSpatialReference::Release().

[snorfalorpagus]

I can reproduce the segfault on Windows using the sample gpkg.py. See output below.

(python36) D:\tmp>python gpkg.py
Warning 6: dataset foo.gpkg does not support layer creation option ENCODING
ERROR 1: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_bar_foo_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such
 module: rtree

Python 3.6 (64-bit) with Fiona 1.7.3 both from conda-forge on Windows 7.

Also, the segfault doesn't seem to happen for me if crs=None. The sqlite3 error is still produced, so this is probably unrelated?

[sgillies]

@snorfalorpagus thanks for the extra data points. I think the sqlite3 error is unrelated.

@snorfalorpagus @chryss I discovered this morning that the script never crashes if I change to the Shapefile driver. Can you try the following for me?

import logging

import fiona

logging.basicConfig(level=logging.DEBUG)

schema = {'geometry': 'Point', 'properties': [('title', 'str')]}
crs = {'init': 'EPSG:4326'}

with fiona.open('foo_shapefiles', 'w', layer='bar_foo', driver='ESRI Shapefile', schema=schema, crs=crs) as colxn:    
    colxn.writerecords([{    
                    'geometry': {'type': 'Point', 'coordinates': [0.0, 0.0]},    
                    'properties': {'title': 'One'}}])    
    colxn.writerecords([{    
                    'geometry': {'type': 'Point', 'coordinates': [2.0, 3.0]},    
                    'properties': {'title': 'Two'}}])

It should create a foo_shapefiles directory in your working directory.

This suggests to me that the problem is in the geopackage driver or sqlite3.

[snorfalorpagus]

@sgillies That code works fine for me.

I tried creating the equivalent using the osgeo.ogr module.

from osgeo import ogr, osr
import os

driver = ogr.GetDriverByName('GPKG')

srs = osr.SpatialReference()
srs.ImportFromEPSG(4326)

ds = driver.CreateDataSource("foo2.gpkg")

layer = ds.CreateLayer("foo_bar", srs, ogr.wkbPoint)

field_title = ogr.FieldDefn("Title", ogr.OFTString)
field_title.SetWidth(24)
layer.CreateField(field_title)

feature = ogr.Feature(layer.GetLayerDefn())
feature.SetField("Title", "One")
point = ogr.CreateGeometryFromWkt("POINT (0 0)")
feature.SetGeometry(point)
layer.CreateFeature(feature)
feature = None

feature = ogr.Feature(layer.GetLayerDefn())
feature.SetField("Title", "Two")
point = ogr.CreateGeometryFromWkt("POINT (2 3)")
feature.SetGeometry(point)
layer.CreateFeature(feature)
feature = None

layer = None
ds = None

This doesn't crash either, despite it using the GPKG driver and as far as I can tell the same GDAL binaries.

[snorfalorpagus]

Running with CPL_DEBUG=ON I get this:

GDAL: GDALOpen(foo.gpkg, this=00000000032F6490) succeeds as GPKG.
GPKG: DeleteLayer(foo)
Warning 6: dataset foo.gpkg does not support layer creation option ENCODING
ERROR 1: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_foo_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such mod
ule: rtree
OSR: Dereference() called on an object with refcount 0,likely already destroyed!

This points to freeing the srs again. We do this here:

https://github.com/Toblerity/Fiona/blob/699ece9c9c3f3d9a934d2daa895c4c98df98be2b/fiona/ogrext2.pyx#L950

Does this need to be free'd or does it take ownership? I don't see why this would cause a segfault though.

[chryss]

@sgillies It does not crash with the ESRI Shapefile or GeoJSON drivers, or MapInfo File.

Like @snorfalorpagus I do not find a crash with the ogr code without fiona (I'm pretty sure my code is nearly identical).

[sgillies]

Thanks, @chryss!

@snorfalorpagus that OSRDestroySpatialReference() call does indeed seem to be the problem. I commented it out and can't produce a crash with the modified module. I'm going to ask on gdal-dev for clarification about ownership of the srs passed to GDALDatasetCreateLayer() and why the GPKG driver is more sensitive than the Shapefile driver.

[sgillies]

Here's the answer: https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046485.html. I've got a branch going to fix this and will schedule a bug fix release tout de suite.

[sgillies]

Resolved by #443.

[sgillies]

Fixed in 1.7.6 @chryss @snorfalorpagus @kannes.

[AndrewAnnex]

👍
interesting read @sgillies, I don't expect to see issues when I update but I will let you know if I do.
Thanks for digging deep!