Geopandas: Error writing out geopackage
I'm running into an error when trying to write the following test GeoDataFrame into a .gpkg file:
import geopandas
db = geopandas.read_file('test.geojson')
db.to_file('test.gpkg', driver='GPKG')
Which returns:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-16-f5a4384ca577> in <module>()
1 db = geopandas.read_file('test.geojson')
----> 2 db.to_file('test.gpkg', driver='GPKG')
~/anaconda/envs/gds/lib/python3.6/site-packages/geopandas/geodataframe.py in to_file(self, filename, driver, schema, **kwargs)
411 """
412 from geopandas.io.file import to_file
--> 413 to_file(self, filename, driver, schema, **kwargs)
414
415 def to_crs(self, crs=None, epsg=None, inplace=False):
~/anaconda/envs/gds/lib/python3.6/site-packages/geopandas/io/file.py in to_file(df, filename, driver, schema, **kwargs)
109 with fiona.open(filename, 'w', driver=driver, crs=df.crs,
110 schema=schema, **kwargs) as colxn:
--> 111 colxn.writerecords(df.iterfeatures())
112
113
~/anaconda/envs/gds/lib/python3.6/site-packages/fiona/collection.py in writerecords(self, records)
333 if self.mode not in ('a', 'w'):
334 raise IOError("collection not open for writing")
--> 335 self.session.writerecs(records, self)
336 self._len = self.session.get_length()
337 self._bounds = self.session.get_extent()
fiona/ogrext.pyx in fiona.ogrext.WritingSession.writerecs()
ValueError: Record's geometry type does not match collection schema's geometry type: 'MultiPolygon' != 'Polygon'
This is on '0.4.0' and the test file is available here. If I try the same operation in QGIS, I have no issue. Any suggestion why?
darribas
All 14 comments
What version of Fiona are you using?
jorisvandenbossche
on 26 Sep 2018
(gds) dh073186:~ dani$ conda list | grep fiona
fiona 1.7.13 py36hb00a9d7_3 conda-forge
(gds) dh073186:~ dani$
In case it helps, this is the whole setup:
(gds) dh073186:~ dani$ conda list
# packages in environment at /Users/dani/anaconda/envs/gds:
#
# Name Version Build Channel
affine 2.2.1 py_0 conda-forge
altair 2.2.2 py36_1 conda-forge
appdirs 1.4.3 py_1 conda-forge
appnope 0.1.0 py36_0 conda-forge
arrow-cpp 0.10.0 py36h70250a7_0 conda-forge
asn1crypto 0.24.0 py36_3 conda-forge
atomicwrites 1.2.1 py_0 conda-forge
attrs 18.2.0 py_0 conda-forge
automat 0.7.0 py_1 conda-forge
backcall 0.1.0 py_0 conda-forge
blas 1.1 openblas conda-forge
bleach 2.1.4 py_1 conda-forge
bokeh 0.13.0 py36_0 conda-forge
boost-cpp 1.67.0 h3a22d5f_0 conda-forge
boto3 1.9.8 py_0 conda-forge
botocore 1.12.9 py_0 conda-forge
bottleneck 1.2.1 py36h7eb728f_1 conda-forge
branca 0.3.0 py_0 conda-forge
bzip2 1.0.6 1 conda-forge
ca-certificates 2018.8.24 ha4d7672_0 conda-forge
cairo 1.14.12 he6fea26_5 conda-forge
cartopy 0.16.0 py36h81b52dc_2 conda-forge
cenpy 0.9.8 <pip>
certifi 2018.8.24 py36_1001 conda-forge
cffi 1.11.5 py36h5e8e0c9_1 conda-forge
cftime 1.0.1 py36h7eb728f_0 conda-forge
chardet 3.0.4 py36_3 conda-forge
click 6.7 py_1 conda-forge
click-plugins 1.0.4 py_0 conda-forge
cligj 0.4.0 py_1 conda-forge
cloudpickle 0.5.6 py_0 conda-forge
colorcet 1.0.0 py_0 conda-forge
constantly 15.1.0 py_0 conda-forge
contextily 1.0rc1 py_0 conda-forge
cryptography 2.3.1 py36hdffb7b8_0 conda-forge
cryptography-vectors 2.3.1 py36_0 conda-forge
curl 7.61.0 h93b3f91_2 conda-forge
cycler 0.10.0 py_1 conda-forge
cython 0.28.5 py36hfc679d8_0 conda-forge
cytoolz 0.9.0.1 py36h470a237_0 conda-forge
dask 0.19.2 py_0 conda-forge
dask-core 0.19.2 py_0 conda-forge
datashader 0.6.6 0 conda-forge
datashape 0.5.4 py36_0 conda-forge
decorator 4.3.0 py_0 conda-forge
descartes 1.1.0 py_2 conda-forge
distributed 1.23.2 py36_0 conda-forge
docutils 0.14 py36_1 conda-forge
entrypoints 0.2.3 py36_2 conda-forge
expat 2.2.5 hfc679d8_2 conda-forge
feather-format 0.4.0 py36_2 conda-forge
fiona 1.7.13 py36hb00a9d7_3 conda-forge
folium 0.6.0 py_0 conda-forge
fontconfig 2.13.1 hce039c3_0 conda-forge
freetype 2.9.1 h6debe1e_4 conda-forge
freexl 1.0.5 h470a237_2 conda-forge
gdal 2.2.4 py36hb00a9d7_1 conda-forge
geographiclib 1.49 py_0 conda-forge
geopandas 0.4.0 py_1 conda-forge
geopy 1.17.0 py_0 conda-forge
geos 3.6.2 hfc679d8_3 conda-forge
geotiff 1.4.2 h700e5ad_4 conda-forge
gettext 0.19.8.1 h1f1d5ed_1 conda-forge
giflib 5.1.4 h470a237_1 conda-forge
glib 2.55.0 h464dc38_2 conda-forge
h5netcdf 0.6.2 py_0 conda-forge
h5py 2.8.0 py36hb794570_1 conda-forge
hdbscan 0.8.18 py36h7eb728f_0 conda-forge
hdf4 4.2.13 h951d187_2 conda-forge
hdf5 1.10.2 hc401514_2 conda-forge
heapdict 1.0.0 py36_0 conda-forge
html5lib 1.0.1 py_0 conda-forge
hyperlink 17.3.1 py_0 conda-forge
icu 58.2 hfc679d8_0 conda-forge
idna 2.7 py36_2 conda-forge
imageio 2.3.0 py_1 conda-forge
incremental 17.5.0 py_0 conda-forge
intel-openmp 2019.0 118
ipykernel 4.9.0 py36_0 conda-forge
ipyleaflet 0.9.0 py36_1 conda-forge
ipython 6.5.0 py36_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.4.2 py_0 conda-forge
jedi 0.12.1 py36_0 conda-forge
jinja2 2.10 py_1 conda-forge
jmespath 0.9.3 py_1 conda-forge
jpeg 9c h470a237_1 conda-forge
json-c 0.12.1 h470a237_1 conda-forge
jsonschema 2.6.0 py36_2 conda-forge
jupyter_client 5.2.3 py_1 conda-forge
jupyter_core 4.4.0 py_0 conda-forge
jupyterlab 0.34.12 py36_0 conda-forge
jupyterlab_launcher 0.13.1 py_2 conda-forge
kealib 1.4.9 h0bee7d0_2 conda-forge
kiwisolver 1.0.1 py36h2d50403_2 conda-forge
krb5 1.14.6 0 conda-forge
libdap4 3.19.1 h18059cb_1 conda-forge
libffi 3.2.1 hfc679d8_5 conda-forge
libgdal 2.2.4 he036fc0_8 conda-forge
libgfortran 3.0.0 1 conda-forge
libiconv 1.15 h470a237_3 conda-forge
libkml 1.3.0 hccc92b1_8 conda-forge
libnetcdf 4.6.1 h039f2a5_8 conda-forge
libpng 1.6.35 ha92aebf_2 conda-forge
libpq 9.6.3 0 conda-forge
libsodium 1.0.16 h470a237_1 conda-forge
libspatialindex 1.8.5 hfc679d8_3 conda-forge
libspatialite 4.3.0a h3b29d86_23 conda-forge
libssh2 1.8.0 h5b517e9_2 conda-forge
libtiff 4.0.9 he6b73bb_2 conda-forge
libxml2 2.9.8 h422b904_5 conda-forge
libxslt 1.1.32 h88dbc4e_2 conda-forge
llvmlite 0.23.0 py36_1 conda-forge
locket 0.2.0 py_2 conda-forge
lxml 4.2.5 py36hc9114bc_0 conda-forge
markupsafe 1.0 py36h470a237_1 conda-forge
matplotlib 3.0.0 py36h45c993b_1 conda-forge
mercantile 1.0.4 py_0 conda-forge
mistune 0.8.3 py36h470a237_2 conda-forge
mkl 2019.0 118
mkl-service 1.1.2 py36h6b9c3cc_4
more-itertools 4.3.0 py36_0 conda-forge
mplleaflet 0.0.5 py_2 conda-forge
msgpack-python 0.5.6 py36h2d50403_2 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 2.3.2 py_0 conda-forge
nbconvert 5.3.1 py_1 conda-forge
nbformat 4.4.0 py_1 conda-forge
ncurses 6.1 hfc679d8_1 conda-forge
netcdf4 1.4.1 py36h62672b6_0 conda-forge
networkx 2.2 py_1 conda-forge
nodejs 10.4.1 0 conda-forge
notebook 5.7.0 py36_0 conda-forge
numba 0.38.1 py36_0 conda-forge
numpy 1.15.1 py36_blas_openblashd3ea46f_1 [blas_openblas] conda-forge
olefile 0.46 py_0 conda-forge
openblas 0.2.20 8 conda-forge
openjpeg 2.3.0 h316dc23_3 conda-forge
openssl 1.0.2p h470a237_0 conda-forge
osmnx 0.8.2 py_0 conda-forge
owslib 0.17.0 py_0 conda-forge
packaging 17.1 py_0 conda-forge
palettable 3.1.1 py_0 conda-forge
pandas 0.23.4 py36hf8a1672_0 conda-forge
pandas-datareader 0.6.0 py36_0 conda-forge
pandoc 2.2.2 hde52d81_1 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
param 1.7.0 py_0 conda-forge
parquet-cpp 1.5.0.pre h83d4a3d_0 conda-forge
parso 0.3.1 py_0 conda-forge
partd 0.3.8 py_1 conda-forge
patsy 0.5.0 py_1 conda-forge
pcre 8.41 hfc679d8_3 conda-forge
pexpect 4.6.0 py36_0 conda-forge
pickleshare 0.7.4 py36_0 conda-forge
pillow 5.2.0 py36hc736899_1 conda-forge
pip 18.0 py36_1 conda-forge
pixman 0.34.0 h470a237_3 conda-forge
pluggy 0.7.1 py_0 conda-forge
polyline 1.3.2 <pip>
poppler 0.67.0 h4d7e492_3 conda-forge
poppler-data 0.4.9 0 conda-forge
proj4 4.9.3 h470a237_8 conda-forge
prometheus_client 0.3.1 py_1 conda-forge
prompt_toolkit 1.0.15 py_1 conda-forge
psutil 5.4.7 py36h470a237_1 conda-forge
psycopg2 2.7.5 py36hdffb7b8_1 conda-forge
ptyprocess 0.6.0 py36_0 conda-forge
py 1.6.0 py_0 conda-forge
pyarrow 0.10.0 py36hfc679d8_0 conda-forge
pyasn1 0.4.4 py_0 conda-forge
pyasn1-modules 0.2.1 py_0 conda-forge
pycparser 2.19 py_0 conda-forge
pyct 0.4.3 py_0 conda-forge
pyepsg 0.3.2 py_1 conda-forge
pygments 2.2.0 py_1 conda-forge
pyhamcrest 1.9.0 py_2 conda-forge
pyopenssl 18.0.0 py36_0 conda-forge
pyparsing 2.2.1 py_0 conda-forge
pyproj 1.9.5.1 py36h508ed2a_5 conda-forge
pysal 1.14.4.post2 py36_0 conda-forge
pyshp 1.2.12 py_0 conda-forge
pysocks 1.6.8 py36_2 conda-forge
pytest 3.8.1 py36_0 conda-forge
python 3.6.6 h5001a0f_0 conda-forge
python-dateutil 2.7.3 py_0 conda-forge
pytz 2018.5 py_0 conda-forge
pywavelets 1.0.0 py36h7eb728f_0 conda-forge
pyyaml 3.13 py36h470a237_1 conda-forge
pyzmq 17.1.2 py36hae99301_0 conda-forge
qgrid 1.1.1 py36_1 conda-forge
rasterio 1.0.5 py36h1b5fcde_0 conda-forge
readline 7.0 haf1bffa_1 conda-forge
requests 2.19.1 py36_1 conda-forge
requests-file 1.4.3 py36_0
requests-ftp 0.3.1 py36_0 conda-forge
rtree 0.8.3 py36_0 conda-forge
s3transfer 0.1.13 py36_1 conda-forge
scikit-image 0.14.0 py36hfc679d8_1 conda-forge
scikit-learn 0.19.2 py36_blas_openblasha84fab4_201 [blas_openblas] conda-forge
scipy 1.1.0 py36_blas_openblash7943236_201 [blas_openblas] conda-forge
seaborn 0.9.0 py_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
service_identity 17.0.0 py_0 conda-forge
setuptools 40.4.0 py36_1000 conda-forge
shapely 1.6.4 py36h164cb2d_1 conda-forge
simplegeneric 0.8.1 py_1 conda-forge
six 1.11.0 py36_1 conda-forge
snuggs 1.4.1 py_1 conda-forge
sortedcontainers 2.0.5 py_0 conda-forge
sqlalchemy 1.2.12 py36h470a237_0 conda-forge
sqlite 3.25.1 hb1c47c0_0 conda-forge
statsmodels 0.9.0 py36_0 conda-forge
tblib 1.3.2 py_1 conda-forge
terminado 0.8.1 py36_1 conda-forge
testpath 0.3.1 py36_1 conda-forge
tk 8.6.8 ha92aebf_0 conda-forge
toolz 0.9.0 py_0 conda-forge
tornado 5.1.1 py36h470a237_0 conda-forge
traitlets 4.3.2 py36_0 conda-forge
traittypes 0.2.1 py_1 conda-forge
twisted 18.7.0 py36h470a237_1 conda-forge
urllib3 1.23 py36_1 conda-forge
vincent 0.4.4 py_1 conda-forge
wcwidth 0.1.7 py_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.31.1 py36_1001 conda-forge
widgetsnbextension 3.4.2 py36_0 conda-forge
xarray 0.10.9 py36_0 conda-forge
xerces-c 3.2.0 h5d6a6da_2 conda-forge
xlrd 1.1.0 py_2 conda-forge
xlsxwriter 1.1.1 py_0 conda-forge
xz 5.2.4 h470a237_1 conda-forge
yaml 0.1.7 h470a237_1 conda-forge
zeromq 4.2.5 hfc679d8_5 conda-forge
zict 0.1.3 py_0 conda-forge
zlib 1.2.11 h470a237_3 conda-forge
zope.interface 4.5.0 py36h470a237_1 conda-forge
(gds) dh073186:~ dani$
Ah, sorry, I forgot to add driver='GPKG' (so it was actually writing a shapefile), so therefore I incorrectly thought it was working for me.
I can confirm the error.
In Fiona, they added the ability to specify multiple geometry types in the schema (but only coming in 1.8): https://github.com/Toblerity/Fiona/pull/539
But then it still depends on whether the driver supports it I think. And not sure about that for GeoPackage, but from a quick look at the specs, it might be that it does not support it (since they store a single geometry_type_name per geometry column: https://www.geopackage.org/spec120/index.html#_geometry_columns)
cc @sgillies
jorisvandenbossche
on 27 Sep 2018
Right, GIS data formats almost always require a single geometry type. Shapefile doesn't really distinguish between single and multi lines and polygons (which is smart!). GeoJSON is the outlier in permitting any geometry type.
sgillies
on 27 Sep 2018
I've been handling this constraint personally by upcasting everything into the multi type if I want to write to GPKG. It would be nice if geopandas did this under the hood for formats that only support a single geometry type...
ljwolf
on 27 Sep 2018
Yes, I can see how in some cases you want to make sure you don't write different types. It'd be nice to have an option at least that'll let you have combined types when writing. Maybe along the lines of @ljwolf suggestion, although I wouldn't make the conversion silently. QGIS must do something under the hood because the example above writes fine into GPKG.
darribas
on 27 Sep 2018
QGIS must do something under the hood because the example above writes fine into GPKG.
@darribas and what are the geometry types if you read that file? (eg all MultiPolygons?)
It would indeed be a nice option to do such a conversion automatically (opt-in). A good start would maybe to have a helper function to do such a conversion to single combined type.
jorisvandenbossche
on 31 Oct 2018
Sorry, should've posted this when I initially commented. My code reads:
from shapely import geometry
upcast_dispatch = {geometry.Point: geometry.MultiPoint,
geometry.LineString: geometry.MultiLineString,
geometry.Polygon: geometry.MultiPolygon}
def maybe_cast_to_multigeometry(geom):
caster = upcast_dispatch.get(type(geom), lambda x: x[0])
return caster([geom])
I would do this any time we're writing in a schema we know supports only one geometry type & we see a df.geometry.unique() with both instances of multi and single types.
I can probably manage a PR with this by end of week.
ljwolf
on 31 Oct 2018
oh, I should add, I then df.geometry.apply(maybe_cast_to_multigeometry).
ljwolf
on 31 Oct 2018
Right, GIS data formats almost always require a single geometry type. Shapefile doesn't really distinguish between single and multi lines and polygons (which is smart!). GeoJSON is the outlier in permitting any geometry type.
I have to disagree... I am trying to write a geodataframe with the following contents:
0 POLYGON ((417342.0511559457 2411704.757801995,...
1 POLYGON ((381292.0511559457 2409712.602504985,...
2 POLYGON ((382892.0511559457 2397112.602504985,...
3 (POLYGON ((397042.0511559457 2393612.602504985...
6 (POLYGON ((382342.0511559457 2381604.757801995...
The last two records are MultiPolygons. I get an error when I use the GeoJSON driver:
>>> valid_data_gdf.to_file('test_intermediate\\valid_data_unbuffered.geojson', driver='GeoJSON')
fiona.errors.GeometryTypeValidationError: Record's geometry type does not match collection schema's geometry type: 'MultiPolygon' != 'Polygon'
awa5114
on 19 May 2019
@awa5114 that's a limitation of GDAL's GeoJSON driver. The format specification allows multiple geometry types.
sgillies
on 19 May 2019
I've been handling this constraint personally by upcasting everything into the
multitype if I want to write to GPKG. It would be nice ifgeopandasdid this under the hood for formats that only support a single geometry type...
@ljwolf
What about using such a feature as a method on the GeoDataFrame object? That way users could convert as they wish, but also, maybe gdf.to_file() could execute the method when the specified driver is in a list (under-the-hood). I've worked up a draft method and test for it. Thoughts?
def enforce_single_multitype(self, inplace=False):
"""
Cast the geometry of the ``GeoDataFrame`` to a single Multi-type.
Returns
-------
GeoDataFrame or None
"""
df = self
if not inplace:
df = self.copy()
geom_types = geopandas.io.file.infer_schema(self)["geometry"]
# Do nothing for already single-geometry typed data
if isinstance(geom_types, (str, unicode)):
return
# Raise error on different base geometry types
# e.g. [Point, (Multi)Polygon]
if not len(set([i.split("Multi")[-1] for i in geom_types])) == 1:
raise AttributeError(
"Geometry contains different base types: {}".format(geom_types))
to_gtype = max(geom_types, key=len) # Gets the type with longest len
# Choose the casting function by geometry type
_caster_func = {
"MultiPoint": geometry.MultiPoint,
"MultiLineString": geometry.MultiLineString,
"MultiPolygon": geometry.MultiPolygon
}[to_gtype]
df.geometry = df.geometry.apply(
# Convert non-Multi types to Multi and leave the rest alone
lambda x: _caster_func([x]) if x.geom_type != to_gtype else x)
if inplace:
return
return df
Test:
gdf = geopandas.read_file("https://pastebin.com/raw/W7JyEa6J")
assert geopandas.io.file.infer_schema(gdf)["geometry"] == ["Polygon", "MultiPolygon"]
gdf.enforce_single_multitype(inplace=True)
assert geopandas.io.file.infer_schema(gdf)["geometry"] == "MultiPolygon"
The PasteBin data is a two-feature GeoJSON (one Polygon, one MultiPolygon) exported from US Census block data (Missoula County): gpd_geojson_two_feature_2geomtypes
WindfallLabs
on 19 Dec 2019
@WindfallLabs We already have similar function: https://github.com/geopandas/geopandas/blob/50c59d395bc539eb411f9a5c6cdf04d37510060a/geopandas/tools/util.py#L13-L25
Not very smart one though.
martinfleis
on 20 Dec 2019
Related issues
cheng-chi
路
4Comments
galak75
路
6Comments
StevenLi-DS
路
4Comments
raiphilibert
路
6Comments
perrygeo
路
3Comments
Most helpful comment
Sorry, should've posted this when I initially commented. My code reads:
I would do this any time we're writing in a schema we know supports only one geometry type & we see a
df.geometry.unique()with both instances of multi and single types.I can probably manage a PR with this by end of week.