Fiona: GeoJSONSeq append to file

[kylebarron]

Expected behavior and actual behavior.

Appending to a GeoJSON newline-delimited file is not currently supported.

https://github.com/Toblerity/Fiona/blob/2ec38d087fea72c8bd0e7696d8ac1a6203df8851/fiona/drvsupport.py#L54

I can't find any other issues mentioning appending using GeoJSONSeq, and given the nature of the format, it seems like that should be possible. Where would a PR to enable that take place?

Steps to reproduce the problem.

import fiona
import geopandas as gpd
from geopandas.io.file import infer_schema
from shapely.geometry import Point

gdf = gpd.GeoDataFrame(geometry=[Point([0, 1])])
schema = infer_schema(gdf)
with fiona.open('test.geojson', "w", driver='GeoJSONSeq', schema=schema) as colxn:
    colxn.writerecords(gdf.iterfeatures())

gdf = gpd.GeoDataFrame(geometry=[Point([2, 2])])
schema = infer_schema(gdf)
with fiona.open('test.geojson', "a", driver='GeoJSONSeq', schema=schema) as colxn:
    colxn.writerecords(gdf.iterfeatures())

DriverError                               Traceback (most recent call last)
<ipython-input-26-01e338412202> in <module>
----> 1 with fiona.open('test.geojson', "a", driver='GeoJSONSeq', schema=schema) as colxn:
      2 
      3     colxn.writerecords(gdf.iterfeatures())
      4 

~/local/anaconda3/envs/create-database/lib/python3.7/site-packages/fiona/env.py in wrapper(*args, **kwargs)
    405             with Env.from_defaults(session=session):
    406                 log.debug("Credentialized: {!r}".format(getenv()))
--> 407                 return f(*args, **kwargs)
    408     return wrapper
    409 

~/local/anaconda3/envs/create-database/lib/python3.7/site-packages/fiona/__init__.py in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, **kwargs)
    252         if mode in ('a', 'r'):
    253             c = Collection(path, mode, driver=driver, encoding=encoding,
--> 254                            layer=layer, enabled_drivers=enabled_drivers, **kwargs)
    255         elif mode == 'w':
    256             if schema:

~/local/anaconda3/envs/create-database/lib/python3.7/site-packages/fiona/collection.py in __init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, **kwargs)
    161 
    162         if self.session is not None:
--> 163             self.guard_driver_mode()
    164 
    165         if self.mode in ("a", "w"):

~/local/anaconda3/envs/create-database/lib/python3.7/site-packages/fiona/collection.py in guard_driver_mode(self)
    180             raise DriverError("unsupported driver: %r" % driver)
    181         if self.mode not in supported_drivers[driver]:
--> 182             raise DriverError("unsupported mode: %r" % self.mode)
    183 
    184     @property

DriverError: unsupported mode: 'a'

Operating system

For example: Mac OS X 10.15.1

Fiona and GDAL version and provenance

Fiona 1.8.11, from Conda
Gdal 3.0.2, from Conda

[kannes]

Where would a PR to enable that take place?

https://github.com/Toblerity/Fiona/blob/master/fiona/drvsupport.py

But with tests etc I would assume. No idea what else might be involved.

[sgillies]

Right, I think we only forgot the append case for the GeoJSONSeq driver at https://github.com/Toblerity/Fiona/blob/master/fiona/drvsupport.py#L54. Should be "raw".

@kylebarron I would accept a PR without a test in this case.

[kylebarron]

Well the reason why I asked where to put a PR was that I wasn't sure if OGR itself supported appending.

Consider this test:

in.geojson:

{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.0, 1.0 ] } }
]
}

Create GeoJSON lines

ogr2ogr -f GeoJSONSeq out.geojson in.geojson

Inspect out.geojson:

> cat out.geojson
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.0, 1.0 ] } 

Failure in appending:

> ogr2ogr -f GeoJSONSeq -append out.geojson in.geojson
ERROR 1: Layer 'in' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Terminating translation prematurely after failed
translation of layer in (use -skipfailures to skip errors)

Is there something I'm doing wrong with ogr2ogr here, or does ogr itself not support appending?

[sgillies]

@kylebarron I haven't used ogr2ogr like this but I see hints in https://gdal.org/programs/ogr2ogr.html that you might have to add an -update option.

[kylebarron]

> ogr2ogr -f GeoJSONSeq out.geojson in.geojson
> ogr2ogr -f GeoJSONSeq -update out.geojson in.geojson
ERROR 1: Layer 'in' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Terminating translation prematurely after failed
translation of layer in (use -skipfailures to skip errors)
> ogr2ogr -f GeoJSONSeq -append -update out.geojson in.geojson
ERROR 1: Layer 'in' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Terminating translation prematurely after failed
translation of layer in (use -skipfailures to skip errors)

I've tried to google around a bit, but I can't find any sources of using ogr to append to a GeoJSON lines file, so maybe it's not supported there? Should I open an issue on https://github.com/OSGeo/gdal?

[kannes]

-update AND -append!

But you are out of luck:

$ ogr2ogr -f GeoJSONSeq out.json foo.shp
$ ogr2ogr -f GeoJSONSeq -update -append out.json bar.shp
ERROR 6: GeoJSONSeq driver does not support update
ERROR 1: Unable to open existing output datasource `out.json'.

[sgillies]

@kannes thank you. This not good: the raison d'être for GeoJSON sequences is to allow incremental parsing and appending. It's not a write once format.

[kylebarron]

Yeah, agreed. I'm a little surprised it wasn't supported by default when the functionality was added. I don't know C/C++ or else I'd try to make a PR to GDAL. In any case, I'll see what responses I get on the GDAL issue, linked above.

[jratike80]

@kannes your GDAL shows a different error message "GeoJSONSeq driver does not support update", did you test with some older GDAL version? The error message from GDAL 3.1dev is "Layer 'in' does not already exist in the output dataset" and it is telling it all. You must append to an existing layer and tell the name of the layer with -nln parameter. Notice also that the name of the layer in GeoJSON is not necessarily the same as the name of the GeoJSON file https://gdal.org/drivers/vector/geojson.html so it is better to check it with ogrinfo first.

ogrinfo out.geojson
INFO: Open of `out.geojson'
      using driver `GeoJSON' successful.
1: out (Point)
ogr2ogr -f GeoJSONSeq -update -append out.geojson in.geojson -nln out
-> Success

[kylebarron]

Maybe I could've searched better, but I still get "GeoJSONSeq driver does not support update". I posted a couple examples on https://github.com/OSGeo/gdal/issues/2080

[jratike80]

Thank you for the examples. There seems to be three issues: 1) Wrong ogr2ogr syntax 2) Bug that seems to allow appending to GeoJSONSeq that has only one row, but that silently changes file into regular GeoJSON 3) Missing "update" feature in GDAL GeoJSONSeq driver.

Now 1) is OK and 3) has #OSGeo/gdal#2080 open. I am not sure if you should open a separate GDAL issue about case 2), but perhaps it is better to wait for a reaction from some GDAL developer.

"Update" feature in GDAL means that also real updates of existing data are supported so it is a bigger job than just to allow to add new lines into the file.

[sgillies]

@jratike80 that's a really good point: OGR lacks a pure "append" mode. I'm not sure what the project should do about that. A format-specific config option, perhaps.

[kylebarron]

I guess a decent ogr2ogr workaround for now is just to use stdout

ogr2ogr -f GeoJSONSeq /vsistdout/ in.geojson >> out.geojson

But presumably, fiona won't add support until ogr2ogr officially supports it.

[sgillies]

@kylebarron right, the part of the fiona API that geopandas uses requires GDAL support.

Note: fiona had one of the first implementations of GeoJSON sequences. You can see it in the fio cat CLI command.