Fiona: Unable to open EPSG support file gcs.csv

[drnextgis]

This issue has the same symptoms as https://github.com/mapbox/rasterio/issues/1539:

In [1]: import fiona                                                                                                                                                                                                

In [2]: ds = fiona.open('aerialway-line.shp')                                                                                                                                                                       

In [3]: ds.crs                                                                                                                                                                                                      
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Out[3]: {}

In [4]: fiona.__version__                                                                                                                                                                                           
Out[4]: '1.8.0'

The open function is decorated by ensure_env_with_credentials but it seems doesn't work.

[sgillies]

@drnextgis are you getting this error with one of the linux wheels? Can you check one thing for me: whether or not you've set the GDAL_DATA and PROJ_LIB environment variables in your process? If you have, then Fiona's environment won't check in the wheel data for GDAL and PROJ support files. See https://github.com/Toblerity/Fiona/blob/master/fiona/_env.pyx#L211.

[drnextgis]

Yes, I'm getting this error with linux wheel Fiona-1.8.0-cp36-cp36m-manylinux1_x86_64.whl:

>>> import os
>>> 'GDAL_DATA' in os.environ
False
>>> import fiona
>>> 'GDAL_DATA' in os.environ
False
>>> ds = fiona.open('aerialway-line.shp')
>>> ds.crs
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
{}

but:

>>> import fiona
>>> with fiona.Env():
...     ds = fiona.open('aerialway-line.shp')
...     ds.crs
... 
{'init': 'epsg:4326'}

[drnextgis]

@sgillies could you reproduce this issue?

[snorfalorpagus]

I can reproduce this, using docker image python:3.6-stretch and Fiona 1.8 from pip. If the new Env object is used then GDAL can find the support file, but not otherwise.

[sgillies]

@snorfalorpagus thanks for checking! I'm stuck on another project and no time to look at this today.

[sgillies]

Confirmed.

$ fio insp ~/projects/Fiona/tests/data/coutwildrnp.shp 
Fiona 1.8.0 Interactive Inspector (Python 3.6.6)
Type "src.schema", "next(src)", or "help(src)" for more information.
>>> src.crs
{'init': 'epsg:4326'}

But

$ python
Python 3.6.6 (default, Sep 12 2018, 18:26:19) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fiona
>>> with fiona.open("/home/sean/projects/Fiona/tests/data/coutwildrnp.shp") as collection:
...     print(collection.crs)
... 
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
{}

This is a different situation from the one in Rasterio.

[sgillies]

I had a moment of clarity this morning. The ensure_env_with_credentials decorator, when applied to fiona.open, is intended to make the following code

with fiona.open(path) as collection:
    print(collection.crs)

work like this

with fiona.Env.from_path(path):
    with fiona.open(path) as collection:
        print(collection.crs)

But it does not! Of course it does not. What actually happens is this:

with fiona.Env.from_path(path):
    collection = fiona.open(path)
with collection as collection:
    print(collection.crs)

There's no longer any GDAL environment when collection.crs is evaluated. It's already exited.

I'm certain we can get the behavior in the second block, though it may harder on Python 2.7.

We will not be able to support this case:

collection = fiona.open(path)
print(collection.crs)

except by setting GDAL_DATA in an environment variable or by decorating the crs getter.

[drnextgis]

Ahhh good catch @sgillies! Do you like an idea of decorating CRS getter?

[sgillies]

Yes, I've almost got that working @drnextgis . We can't easily decorate class methods using functools.wraps in Python 2.7, so I brought the context manager inside the methods.

I want to amend my comment above. The behavior in the 2nd block above will be rather difficult and in my judgement not worth doing.

[sgillies]

Done.

[drnextgis]

I've got the same issue with the latest version of Fiona (1.8.1):

>>> import fiona
>>> fiona.__version__
'1.8.1'
>>> shp = fiona.open('/home/denis/git/Fiona/tests/data/coutwildrnp.shp')
>>> next(iter(shp))
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.

[sgillies]

@drnextgis this sort of usage is not what I had in mind for Fiona or Rasterio and I don't want to change the code a lot to support it. I think the best option for you would be to set GDAL_DATA and PROJ_LIB environment variables. The problem, of course, is that Fiona and Rasterio don't have an API for getting these paths, you must know internal details, and that's not good. What would you think about the following proposal?

When Env.__enter__() is called and we discover that we are using a wheel (in other words, find GDAL data in the package) we could patch the environment os.environ['GDAL_DATA'] so that all subsequent GDAL calls would use the data in that path, whether or not the calls are made in the context of an Env. I'd prefer not to patch the environment, but it seems like the most reliable approach.

@snorfalorpagus I'd be grateful for a sanity check on this proposal from you, too.

[drnextgis]

Sounds good to me and I see you already implemented that suggestion. @sgillies could you put wheels with this fix to https://test.pypi.org before making the official release? Anyway good work, as always!

[kannes]

As I was struggling with this stuff today ->

Can you check one thing for me: whether or not you've set the GDAL_DATA and PROJ_LIB environment variables in your process? If you have, then Fiona's environment won't check in the wheel data for GDAL and PROJ support files. See https://github.com/Toblerity/Fiona/blob/master/fiona/_env.pyx#L211.

Permalink to the code as linenumbers change over time: https://github.com/Toblerity/Fiona/blob/de6a122d0ebf421fda34a0df335ef76bd904631d/fiona/_env.pyx#L321-L323

whoa, github inlines that :O

[xhoffmann]

Hi, I just upgraded fiona to 1.8.4. I tried to load a shape file created last week (with release 1.7.13) and got these warnings:
ERROR 4: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.

The loaded object doesn't have a crs, which it definitely had when I stored it. If I create a new file with 1.8.4 and then load it, the problem disappears. FYI, I'm using fiona through geopandas.

I'm not using that many files, so I could recreate all of them with 1.8.4. But others might have a larger number of files, so I thought it'd be nice to report this issue.

[sgillies]

@xhoffmann you won't need to recreate files. It's only the crs property accessor that isn't working for you.

Here's the deal: Fiona 1.7.13 patches os.environ to set GDAL_DATA and PROJ_LIB when we called fiona.drivers() whether directly or indirectly. This is smelly and I've changed it. Fiona 1.8 no longer patches os.environ. Now you need to run code that will use CRS objects within a with fiona.Env(): block like

with fiona.Env():
    with fiona.open('example.shp') as collection:
        print(collection.crs)

or you can set a GDAL_DATA environment variable to point to the directory of support files. If you're using a Fiona wheel from PyPI, you can run fio env --gdal-data (with Fiona 1.8.3 or newer) to print the path to this directory.

[astrojuanlu]

We are observing some weird caching:

$ python
>>> import fiona
>>> with fiona.open("test_id.shp", 'r') as source:
...  print(source.crs)
... 
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
{}
>>> with fiona.Env():
...   with fiona.open("test_id.shp", 'r') as source:
...     print(source.crs)
... 
{}
>>> 
$ python
>>> import fiona
>>> with fiona.Env():
...   with fiona.open("test_id.shp", 'r') as source:
...     print(source.crs)
... 
{'init': 'epsg:4326'}
>>> with fiona.open("test_id.shp", 'r') as source:
...   print(source.crs)
... 
{'init': 'epsg:4326'}

This was a bit difficult to discover because it looked like non deterministic at first, but I just reproduced with a clean installation of the latest fiona wheel.

[sgillies]

@Juanlu001 can you share that shapefile with me? I can't reproduce the issue with files I have.

[drnextgis]

@sgillies file that you asked about: test_id.tar.gz

[drnextgis]

@sgillies was you able to reproduce the described behaviour using this file?

[sgillies]

@drnextgis @Juanlu001 I'm unable to reproduce the problem with fiona 1.8.4 on my macbook:

$ python
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> 'GDAL_DATA' in os.environ
False
>>> import fiona
>>> with fiona.open('/Users/seang/Downloads/test_id/test_id.shp') as collection:
...     print(collection.crs)
...
{'init': 'epsg:4326'}

If GDAL_DATA is set in your environment and is not correct, you might see the symptoms you reported. Can you check that?

@Juanlu001 there is indeed some SRS caching in GDAL. You can see signs of it in https://github.com/OSGeo/gdal/blob/master/gdal/ogr/ogr_fromepsg.cpp. I don't fully understand how it works, but I'm glad it's there so we don't have to read the support files over and over again. I assume it's going to change quite a lot when the new PROJ features land in GDAL 2.5.

[drnextgis]

@sgillies I also cannot reproduce this issue with Fiona 1.8.3. It seems like something broken in 1.8.4. I suspect this commit https://github.com/Toblerity/Fiona/commit/fdca6dd320b9c3906b72f75c51bd9d384ed3c73a.

[sgillies]

Sorry @drnextgis, I meant 1.8.4 and have edited my comment above.

[astrojuanlu]

@sgillies I prepared some scripts and a dockerfile to reproduce the issue:

https://github.com/Juanlu001/gdal-data-issues

In any case, if the final word about this and https://github.com/mapbox/rasterio/issues/1539 is "always use with rasterio.Env() or with fiona.Env() when appropriate", then there's not much left to do.

[sgillies]

@Juanlu001 I think I may have explained the situation poorly. Setting GDAL_DATA and PROJ_LIB to point to the directory inside the installed wheel will always work. You can find these directories by using fio env --gdal-data or fio env --proj-lib. This is what I would do if I wanted to use the PyPI wheels in production. I don't, though, I compile Fiona from source for production and take advantage of GDAL and PROJ data paths being compiled into the binaries.

But if you don't want to set GDAL_DATA and PROJ_LIB in your production environment, you can execute code within an Env block and the directories will be found and GDAL will be configured within the block.

[astrojuanlu]

I think I understand it in full now. Thanks for the explanation!

[dazza-codes]

Bumped into this issue after getting an ERROR 4 warning for rio but not fio, i.e. both libraries seem to set the env when imported without the OS-env set, but the CLI tools have different behavior. The packages are installed using pip wheels and they are expected to be using the bundled shared libs and data. (In console details below a conda env is used only to create a py3.7 env, and then poetry (pip) is used to install all the packages into that env).

$ fio --version
fio, version 1.8.18
$ rio --version
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
1.2.1

More details:

$ env | grep 'GDAL_DATA'

$ python
Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> 'GDAL_DATA' in os.environ
False
>>> import rasterio
>>> 'GDAL_DATA' in os.environ
True

$ python
Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> 'GDAL_DATA' in os.environ
False
>>> import fiona
>>> 'GDAL_DATA' in os.environ
True

$ fio --gdal-version
GDAL, version 2.4.4

$ rio --gdal-version
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
3.2.1

#
# Note how they are finding different GDAL versions somehow
#

$ fio env --proj-data
/opt/conda/envs/gis/lib/python3.7/site-packages/fiona/proj_data
$ fio env --gdal-data
/opt/conda/envs/gis/lib/python3.7/site-packages/fiona/gdal_data

$ rio env --proj-data
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
/opt/conda/envs/gis/lib/python3.7/site-packages/fiona/proj_data
$ rio env --gdal-data
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
/opt/conda/envs/gis/lib/python3.7/site-packages/rasterio/gdal_data

Whatever fixed it in fio is not applied in rio in this release? Given the context of this issue above, I guess it might be useful to add commentary here rather than open another issue on rasterio (google seach landed here).

In comments above, there is a --proj-lib option, but the [fr]io env --proj-lib option seems to be gone in recent releases:

$ rio env --help
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
Usage: rio env [OPTIONS]

  Print information about the Rasterio environment.

Options:
  --formats      Enumerate the available formats.
  --credentials  Print credentials.
  --gdal-data    Print GDAL data path.
  --proj-data    Print PROJ data path.
  --help         Show this message and exit.

$ fio env --help
Usage: fio env [OPTIONS]

  Print information about the Fiona environment: available formats, etc.

Options:
  --formats      Enumerate the available formats.
  --credentials  Print credentials.
  --gdal-data    Print GDAL data path.
  --proj-data    Print PROJ data path.
  --help         Show this message and exit.

$ fio env --proj-lib
Usage: fio env [OPTIONS]
Try 'fio env --help' for help.

Error: no such option: --proj-lib

It would really help if [fr]io provided a catch-all command line option, like --versions or something, to print all the versions of GDAL and PROJ included in the binary libs and the associated data paths. The [fr]io env command is a full listing, but I did not find anything about PROJ in it; it just seems to print the formats supported and not the library-env setup (i.e. it default to [fr]io --formats). I resorted to tracking it down with e.g.

$ strings /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio.libs/libproj-rasterio-59bf5ceb.so.19.2.1  | grep PROJ
PROJ 7.2.1

$ strings /opt/conda/envs/gis/lib/python3.7/site-packages/Fiona.libs/libproj-fiona-cd06b982.so.12.0.0  | grep PROJ
PROJ_LIB
PROJ_DEBUG

• no version string is in that one

This venv also installs pyproj-3.0.1 that has another copy of binary libproj, so I wanted to check that version too:

$ strings /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj.libs/libproj-27e7e26d.so.19.2.1  | grep PROJ
PROJ 7.2.1

There is a geopandas command that is related to this, e.g.

>>> geopandas.show_versions()

SYSTEM INFO
-----------
python     : 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37)  [GCC 9.3.0]
executable : /opt/conda/envs/gis/bin/python
machine    : Linux-4.15.0-136-generic-x86_64-with-debian-buster-sid

GEOS, GDAL, PROJ INFO
---------------------
GEOS       : 3.7.1
GEOS lib   : /usr/lib/x86_64-linux-gnu/libgeos_c.so
GDAL       : 2.4.4
GDAL data dir: /opt/conda/envs/gis/lib/python3.7/site-packages/fiona/gdal_data
PROJ       : 7.2.1
PROJ data dir: /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj/proj_dir/share/proj

PYTHON DEPENDENCIES
-------------------
geopandas  : 0.9.0
pandas     : 1.2.3
fiona      : 1.8.18
numpy      : 1.20.1
shapely    : 1.7.1
rtree      : None
pyproj     : 3.0.1
matplotlib : 3.3.4
mapclassify: None
geopy      : 2.1.0
psycopg2   : None
geoalchemy2: None
pyarrow    : 3.0.0
pygeos     : None

package integrations and dependencies

This issue is only about getting consistent information about the bundled libs and data (GDAL and PROJ), but it also begs questions about package integrations. Ignore the following if it represents too much scope creep. It seems like the short answer to the notes below is to build fiona and rasterio without binary wheels against OS packages for shared libs. This gets into how packaging tools expose options to manage all that (linux distros, conda, poetry, pip, etc etc).

It seems like some kind of dummy package with integration tests could help to catch inconsistencies or some kind of abstraction for a shared package is required to manage the common dependencies and CLI tools for fio and rio libs or env. Certainly, there is a use-case for projects to use only one or the other package (fiona or rasterio) and not both of them, but it seems like the pip binary wheels could be breaking that expectation anyway - e.g. rasterio and fiona both bundle gdal-libs and data and they both have no explicit dependency on pyproj but they both bundle a libproj and the proj data into the binary wheels. Is it not possible to delegate that responsibility to pyproj and simply depend on it explicitly in fiona and rasterio? With an explicit dependency on a version of pyproj, it could help with unit tests and CI tests on CRS compatibility and a shared management of compatible libproj versions for consistent lib/env details.

All the bundled shared libs:

$ pip show fiona
Name: Fiona
Version: 1.8.18
Summary: Fiona reads and writes spatial data files
Home-page: http://github.com/Toblerity/Fiona
Author: Sean Gillies
Author-email: [email protected]
License: BSD
Location: /opt/conda/envs/gis/lib/python3.7/site-packages
Requires: munch, click-plugins, certifi, six, cligj, click, attrs
Required-by: rasterstats, geopandas

$ pip show rasterio
Name: rasterio
Version: 1.2.1
Summary: Fast and direct raster I/O for use with Numpy and SciPy
Home-page: https://github.com/mapbox/rasterio
Author: Sean Gillies
Author-email: [email protected]
License: BSD
Location: /opt/conda/envs/gis/lib/python3.7/site-packages
Requires: certifi, attrs, snuggs, affine, click-plugins, numpy, cligj, click
Required-by: rasterstats, contextily

$ pip show pyproj
Name: pyproj
Version: 3.0.1
Summary: Python interface to PROJ (cartographic projections and coordinate transformations library)
Home-page: https://github.com/pyproj4/pyproj
Author: Jeff Whitaker
Author-email: [email protected]
License: MIT
Location: /opt/conda/envs/gis/lib/python3.7/site-packages
Requires: certifi
Required-by: geopandas

$ ls -1 /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj.libs/
libcurl-9a092e29.so.4.6.0
libnghttp2-223fd912.so.14.17.1
libproj-27e7e26d.so.19.2.1
libsqlite3-69e6c45e.so.0.8.6
libtiff-4e10f209.so.5.5.0
libz-eb09ad1d.so.1.2.3

$ ls -1 /opt/conda/envs/gis/lib/python3.7/site-packages/Fiona.libs/
libcurl-fiona-ea538880.so.4.4.0
libexpat-fiona-c4a93fc7.so.1.6.8
libgdal-fiona-9fe15c06.so.20.5.4
libgeos_c-fiona-a68605fd.so.1.13.1
libgeos--no-undefined-fiona-b94097bf.so
libjpeg-fiona-3fe7dfc0.so.9.3.0
libjson-c-fiona-5f02f62c.so.2.0.2
libnghttp2-fiona-11cb20b8.so.14.17.1
libpng16-fiona-898afbbd.so.16.35.0
libproj-fiona-cd06b982.so.12.0.0
libsqlite3-fiona-25a4bc97.so.0.8.6
libz-fiona-a147dcb0.so.1.2.3

$ ls -1 /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio.libs/
libaec-rasterio-f0d4887b.so.0.0.10
libcurl-rasterio-ea538880.so.4.4.0
libexpat-rasterio-09c47d4c.so.1.6.8
libgdal-rasterio-db347237.so.28.0.1
libgeos_c-rasterio-a68605fd.so.1.13.1
libgeos--no-undefined-rasterio-b94097bf.so
libhdf5_hl-rasterio-92c1cdd8.so.100.1.2
libhdf5-rasterio-707c161f.so.103.1.0
libjpeg-rasterio-3fe7dfc0.so.9.3.0
libjson-c-rasterio-5f02f62c.so.2.0.2
liblzma-rasterio-90de1f11.so.5.2.2
libnetcdf-rasterio-cae4e7d9.so.13.1.1
libnghttp2-rasterio-11cb20b8.so.14.17.1
libopenjp2-rasterio-8f6da918.so.2.3.0
libpng16-rasterio-898afbbd.so.16.35.0
libproj-rasterio-59bf5ceb.so.19.2.1
libsqlite3-rasterio-bc0a2dd7.so.0.8.6
libsz-rasterio-53d02de5.so.2.0.1
libtiff-rasterio-bd1961ca.so.5.5.0
libwebp-rasterio-fbd93615.so.7.0.5
libz-rasterio-a147dcb0.so.1.2.3
libzstd-rasterio-0338e592.so.1.4.7

$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj.libs/
19M /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj.libs/

$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/Fiona.libs/
30M /opt/conda/envs/gis/lib/python3.7/site-packages/Fiona.libs/

$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio.libs/
49M /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio.libs/


$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj/proj_dir/
8.7M    /opt/conda/envs/gis/lib/python3.7/site-packages/pyproj/proj_dir/

$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/fiona/proj_data/
5.1M    /opt/conda/envs/gis/lib/python3.7/site-packages/fiona/proj_data/

$ du -sh /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio/proj_data/
8.7M    /opt/conda/envs/gis/lib/python3.7/site-packages/rasterio/proj_data/

[drnextgis]

https://github.com/mapbox/rasterio/issues/2126#issuecomment-786704154