Rasterio: Value of datasource CRS depends on the previous commands

[drnextgis]

After upgrading rasterio to the latest version some tests of our library are not succeed. I've made the synthetic examples that reflects the problem. As you can see in both cases we get different CRSs:

Example 1 (good):

>>> import rasterio
>>> rasterio.__version__
'1.0.15'
>>> with rasterio.open('https://github.com/satellogic/telluric/raw/master/tests/data/raster/rgb.tif') as src:
...     print(src.crs)
... 
EPSG:3857

Example 2 (bad):

>>> import rasterio
>>> rasterio.__version__
'1.0.15'
>>> from copy import copy
>>> from rasterio.crs import CRS
>>> crs = CRS({'init': 'epsg:3857'})
>>> dummy = dict(crs=CRS(copy(crs)))
>>> with rasterio.open('https://github.com/satellogic/telluric/raw/master/tests/data/raster/rgb.tif') as src:
...     print(src.crs)
... 
LOCAL_CS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["unknown",SPHEROID["unretrievable - using WGS84",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],AUTHORITY["EPSG","3857"],UNIT["metre",1,AUTHORITY["EPSG","9001"]]]

[astrojuanlu]

Confirmed in a barebones Docker container, and also confirmed that it doesn't happen if GDAL_DATA is properly set:

root@9efdec70cbab:/src# CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt python3
Python 3.7.2 (default, Jan 23 2019, 02:35:07) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> rasterio.__version__
'1.0.15'
>>> from rasterio.crs import CRS
>>> crs = CRS({'init': 'epsg:3857'})
>>> from copy import copy
>>> dummy = dict(crs=CRS(copy(crs)))
>>> with rasterio.open('https://github.com/satellogic/telluric/raw/master/tests/data/raster/rgb.tif') as src:
...   print(src.crs)
... 
LOCAL_CS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["unknown",SPHEROID["unretrievable - using WGS84",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],AUTHORITY["EPSG","3857"],UNIT["metre",1,AUTHORITY["EPSG","9001"]]]
>>> rasterio.__file__
'/usr/local/lib/python3.7/site-packages/rasterio/__init__.py'
>>> 
root@9efdec70cbab:/src# rio env --gdal-data
/usr/local/lib/python3.7/site-packages/rasterio/gdal_data
root@9efdec70cbab:/src# CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt GDAL_DATA=$(rio env --gdal-data) python3
Python 3.7.2 (default, Jan 23 2019, 02:35:07) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rasterio
>>> rasterio.__file__
'/usr/local/lib/python3.7/site-packages/rasterio/__init__.py'
>>> rasterio.__version__
'1.0.15'
>>> from rasterio.crs import CRS
>>> crs = CRS({'init': 'epsg:3857'})
>>> from copy import copy
>>> dummy = dict(crs=CRS(copy(crs)))
>>> with rasterio.open('https://github.com/satellogic/telluric/raw/master/tests/data/raster/rgb.tif') as src:
...   print(src.crs)
... 
EPSG:3857

[astrojuanlu]

Even simpler: str(CRS({...})) before opening the file already triggers this behavior.

[drnextgis]

Good catch, confirmed:

>>> import rasterio
>>> rasterio.__version__
'1.0.15'
>>> from rasterio.crs import CRS
>>> crs = str(CRS({'init': 'epsg:3857'}))
>>> with rasterio.open('https://github.com/satellogic/telluric/raw/master/tests/data/raster/rgb.tif') as src:
...     print(src.crs)
... 
LOCAL_CS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["unknown",SPHEROID["unretrievable - using WGS84",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],AUTHORITY["EPSG","3857"],UNIT["metre",1,AUTHORITY["EPSG","9001"]]]

[guydou]

why is it happening?

[sgillies]

Wow, this is an interesting bug. The 1.0.15 wheels include GDAL 2.4.0, so we may need to look for changes in the CRS code.

[sgillies]

I wasn't able to reproduce the bug with 1.0.13. It's possible that this is a bug in Rasterio but I think it is more likely to be a change in GDAL 2.4.0.

Remember: with GDAL versions 2.3 and earlier, the happy path is the one where you rely on the built-in GDAL data path or have GDAL_DATA set in your environment before you load GDAL and call any of its functions. It may have been only an accident and not intended that we could configure GDAL_DATA during execution of our code. This seems to have changed.

The place where it could be a bug in Rasterio: use of Env (which can set GDAL_DATA for wheels) in CRS constructors has been removed, so maybe look up of epsg codes fails and then GDAL refuses to look again when we open the dataset. I'll check into this later today.

[sgillies]

This is definitely fixed in 1.0.16, which is going onto PyPI now.

[astrojuanlu]

Thanks a lot @sgillies! I can't hide how happy I am that rasterio is patching GDAL_DATA again :)

[sgillies]

@Juanlu001 the purist in me hates doing this, but your forgiveness helps and I will soon get over it :laughing: