Fiona: Driver not found (GPKG)

[jachym]

Expected behavior and actual behavior.

GPKG driver is avaliable

Steps to reproduce the problem.

import fiona
from fiona._drivers import GDALEnv
env = GDALEnv()

"GPKG" in env.start().drivers().keys()
True

Operating system

Linux, Ubuntu 17.10

Fiona version and provenance

>>> fiona.__version__
'1.7.10.post1'
>>> 

from pip

[sgillies]

Hi @jachym . I'm building GDAL for the manylinux1 wheels using this Docker image: https://github.com/sgillies/frs-wheel-builds/blob/master/Dockerfile.wheels#L105-L144. I haven't tested it against a geopackage file yet and will do so. It's quite possible that GPKG support fails in the configure step due to sqlite3 version issues, and as you know compilation of GDAL would continue without GPKG. I'll dig into this soon.

The work-around for now is to skip these manylinux1 wheels and build Fiona from a source distribution like this:

GDAL_CONFIG=/path/to/gdal-config pip install --ignore-installed --no-binary fiona fiona>=1.7.10

[sgillies]

@jachym the 1.8a1 wheels I have uploaded to PyPI this morning should have GPKG support thanks to help from Even Rouault. Can you try pip install fiona==1.8a1 and let me know if it works for you?

[jachym]

looks good, thank you

[sid-kap]

I'm trying to install Fiona into a virtual env using the wheel (using pipenv), but it doesn't seem to work for me. "GPKG" in env.start().drivers().keys() is returning false. I verified that I have installed version 1.8a1.
How can I troubleshoot this? Should it print something while it's installing the wheel that indicates whether it's including the GDAL drivers?

[kannes]

That sounds like you do not have the Python bindings of GDAL (and GDAL) installed.

edit: This was wrong, see below.

[sid-kap]

@kannes Hmm, I tried running pipenv install GDAL before pipenv install fiona and it still didn't work.
This feels kinda unconventional, since GDAL needs to be installed before fiona, even though it's not technically a dependency. Is there an accepted way of doing this, to make sure pipenv installs stuff in the right order? (Is there a way to make multiple versions of a package, where one has a dependency on GDAL?)

[sgillies]

@kannes @sid-kap Fiona requires the GDAL C library but does not depend on the GDAL Python bindings at all – except if you get Fiona from https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona, but this is Gohlke's choice for his particular distribution, and not a choice of the Fiona project.

[sid-kap]

@sgillies Thanks! So I do have the GDAL C library installed, and it still doesn't seem to be working. Do you have any advice on how to troubleshoot this? Is there any output during the installation I should look for?

[snorfalorpagus]

@sid-kap Which operating system are you using? Do you know if you're using a wheel from pypi, or from source?

You can check if your GDAL install was compiled with GPKG support with: ogr2ogr --formats

[sid-kap]

ogr2ogr --formats includes the line:

  GPKG -raster,vector- (rw+vs): GeoPackage

So I think that's not the problem? I'm using Linux/nixos. I would guess that the wheel is from pypi—where else would it come from? What do you mean by "from source"?

[sgillies]

@sid-kap I'm sorry for the confusion! Fiona 1.7.11.post1 (https://pypi.org/project/Fiona/1.7.11.post1/) is the only wheel release that currently includes support for geopackages.

[sid-kap]

Hmm. So it turns out I wasn't using the wheel (because the _manylinux.manylinux1_compatible was not set to True). But now I've reinstalled with the wheel, and made sure I'm on 1.7.11.post1, and I'm still having the same problem.
Is it possible that because it's caching the whl, it's not rebuilding with GPKG support or something? Or should rebuilding not be necessary?

[kannes]

At this point I would recommend "strace -e file python" and then a "import fiona" to not have to guess. Might be the wrong tool but it might help you circle in on the problem.

[sid-kap]

I mean, I know which fiona it's using (it's the one inside the virtualenv, in lib/python3.6/site-packages/fiona). Not sure what else to look for

[sid-kap]

Can anyone explain to me how the GPKG/GDAL support works? Does something happen at install-time, or is the decision of whether to include the feature completely at runtime? Does it throw an error or print any message at install-time to indicate whether it found GDAL? When does it call gdal-config? I just want to debug it but I don't know where to start.

[rbuffat]

Fiona is a wrapper library around gdal. It can only use the drivers that are installed with gdal.

There are many possible ways to install fiona. If you use the prepackaged binary wheels (typically when installed using pip install fiona on unix systems), then it is already a version of gdal in binary format included. There was a bug where in those wheels GPKG support was not enabled.

You can also install Fiona from source with pip install --no-binary fiona . In this case you need to ensure that you have a local installation of gdal. If you are on a unix system, fiona should then use gdal-config to automatically build fiona for your specific version of gdal.

[sid-kap]

@rbuffat Thanks for the clarification. That seems to contradict what other people have said (they suggested that even with the wheel, you still need gdal-config). I guess I'll take your word?

By "a version of gdal in binary format included" does that mean a copy of the GDAL C library (a .so file) should be in the wheel? If so, where does it copy it to? Should it be in the lib/python3.6/site-packages/fiona directory in my virtual environment?

[sid-kap]

I see some .so files in the unzipped wheel, but I don't see one with gdal in the name.

[sid-kap]

cc @sgillies do you know about this? I'm confused about how the wheel works.

[sgillies]

The GDAL library is indeed included in the wheel.

$ tar tzvf ~/Downloads/Fiona-1.7.11.post1-cp36-cp36m-manylinux1_x86_64.whl | grep libs
-rwxrwxrwx  0 0      0     3923064 Feb  9 20:30 fiona/.libs/libhdf5-f60e12ad.so.101.0.0
-rwxrwxrwx  0 0      0      174248 Feb  9 20:30 fiona/.libs/libhdf5_hl-f4337ea0.so.100.0.1
-rwxrwxrwx  0 0      0      505552 Feb  9 20:30 fiona/.libs/libproj-1a5e54ac.so.12.0.0
-rwxrwxrwx  0 0      0      296720 Feb  9 20:30 fiona/.libs/libgeos_c-bd8d3f16.so.1.10.2
-rwxrwxrwx  0 0      0     1717104 Feb  9 20:30 fiona/.libs/libnetcdf-659163d0.so.11.0.4
-rwxrwxrwx  0 0      0       43712 Feb  9 20:30 fiona/.libs/libjson-c-ca0558d5.so.2.0.1
-rwxrwxrwx  0 0      0      399568 Feb  9 20:30 fiona/.libs/libopenjp2-00a09f32.so.2.3.0
-rwxrwxrwx  0 0      0    28095144 Feb  9 20:30 fiona/.libs/libgdal-b402c5c6.so.20.3.1
-rwxrwxrwx  0 0      0      584256 Feb  9 20:30 fiona/.libs/libexpat-0962c39c.so.1.6.7
-rwxrwxrwx  0 0      0     2556976 Feb  9 20:30 fiona/.libs/libgeos-3-6a255356.6.2.so
-rwxrwxrwx  0 0      0     3750144 Feb  9 20:30 fiona/.libs/libcurl-744c4e24.so.4.4.0
-rwxrwxrwx  0 0      0       87528 Feb  9 20:30 fiona/.libs/libz-a147dcb0.so.1.2.3

When you install the wheel via pip install fiona these files will be unpacked under lib/python3.6/site-packages/fiona. If you've installed Fiona before in some other way, I suggest deleting your virtual environment and starting over.

[sid-kap]

@sgillies Thanks! Can confirm that all of these files, including libgdal-b402c5c6.so.20.3.1, are in ~/.virtualenvs/<project name>/lib/python3.6/site-packages/fiona/.libs. However, I'm still getting the "Unsupported driver: 'GPKG' " message somehow.

[sgillies]

@sid-kap I tried installing the Fiona 1.7.11.post1 wheel on a Ubuntu 16.04 server today and it looks like I've messed up the GPKG support in the GDAL library somehow, probably by failing to configure sqlite3 properly. Sorry about this. I hope to have a fix by mid-week.

[sid-kap]

Great to hear that someone can replicate my issue. Thanks so much!

[sgillies]

@sid-kap Fiona 1.7.11.post2 is on PyPI now and includes support for GPKG.

[sid-kap]

It works for me now, thanks!

[sid-kap]

@sgillies Thanks again for all your help! Just curious, you said

Fiona 1.7.11.post1 is the only wheel release that currently includes support for geopackages.

Does this mean the 1.8a1 wheel doesn't have GDAL support working? (Asking because I'm looking forward to using the transaction support for GPKG...)