Rasterio: Read remote netcdf from ftp

[valpesendorfer]

Expected behavior and actual behavior.

I'm trying to read a netCDF file from an ftp server. Works in gdal (both cli and py api).

When trying with rasterio, it takes quite a long time and finally I'm receiving the following exceptions:

Traceback (most recent call last):
File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_IllegalArgError: Missing url parameter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 445, in wrapper
return f(args, *kwds)
File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 219, in open
s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: Missing url parameter

Also tried without netcdf:/vsicurl/ as it works with gdal, but this raises an rasterio.errors.RasterioIOError which makes sense.

Steps to reproduce the problem.

import rasterio as rio

ftp_uri = "ftp://ftp.cdc.noaa.gov/Projects/RefET/global/Gen-0/fine_resolution/data_v2/2019/ETrs_fine_20191201.nc"

ds = rio.open("netcdf:/vsicurl/" + ftp_uri)

For comparison, the gdal approach works just fine:

import gdal

ftp_uri = "ftp://ftp.cdc.noaa.gov/Projects/RefET/global/Gen-0/fine_resolution/data_v2/2019/ETrs_fine_20191201.nc"

ds = gdal.Open(ftp_uri)

Operating system

Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic

gdalinfo --version

GDAL 2.4.2, released 2019/06/28

gdalinfo --formats

VRT -raster- (rw+v): Virtual Raster
DERIVED -raster- (ro): Derived datasets using VRT pixel functions
GTiff -raster- (rw+vs): GeoTIFF
NITF -raster- (rw+vs): National Imagery Transmission Format
RPFTOC -raster- (rovs): Raster Product Format TOC format
ECRGTOC -raster- (rovs): ECRG TOC format
HFA -raster- (rw+v): Erdas Imagine Images (.img)
SAR_CEOS -raster- (rov): CEOS SAR Image
CEOS -raster- (rov): CEOS Image
JAXAPALSAR -raster- (rov): JAXA PALSAR Product Reader (Level 1.1/1.5)
GFF -raster- (rov): Ground-based SAR Applications Testbed File Format (.gff)
ELAS -raster- (rw+v): ELAS
AIG -raster- (rov): Arc/Info Binary Grid
AAIGrid -raster- (rwv): Arc/Info ASCII Grid
GRASSASCIIGrid -raster- (rov): GRASS ASCII Grid
SDTS -raster- (rov): SDTS Raster
DTED -raster- (rwv): DTED Elevation Raster
PNG -raster- (rwv): Portable Network Graphics
JPEG -raster- (rwv): JPEG JFIF
MEM -raster- (rw+): In Memory Raster
JDEM -raster- (rov): Japanese DEM (.mem)
GIF -raster- (rwv): Graphics Interchange Format (.gif)
BIGGIF -raster- (rov): Graphics Interchange Format (.gif)
ESAT -raster- (rov): Envisat Image Format
BSB -raster- (rov): Maptech BSB Nautical Charts
XPM -raster- (rwv): X11 PixMap Format
BMP -raster- (rw+v): MS Windows Device Independent Bitmap
DIMAP -raster- (rov): SPOT DIMAP
AirSAR -raster- (rov): AirSAR Polarimetric Image
RS2 -raster- (rovs): RadarSat 2 XML Product
SAFE -raster- (rov): Sentinel-1 SAR SAFE Product
PCIDSK -raster,vector- (rw+v): PCIDSK Database File
PCRaster -raster- (rw+): PCRaster Raster File
ILWIS -raster- (rw+v): ILWIS Raster Map
SGI -raster- (rw+v): SGI Image File Format 1.0
SRTMHGT -raster- (rwv): SRTMHGT File Format
Leveller -raster- (rw+v): Leveller heightfield
Terragen -raster- (rw+v): Terragen heightfield
GMT -raster- (rw): GMT NetCDF Grid Format
netCDF -raster,vector- (rw+vs): Network Common Data Format
HDF4 -raster- (ros): Hierarchical Data Format Release 4
HDF4Image -raster- (rw+): HDF4 Dataset
ISIS3 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 3)
ISIS2 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 2)
PDS -raster- (rov): NASA Planetary Data System
PDS4 -raster- (rw+vs): NASA Planetary Data System 4
VICAR -raster- (rov): MIPL VICAR file
TIL -raster- (rov): EarthWatch .TIL
ERS -raster- (rw+v): ERMapper .ers Labelled
JP2OpenJPEG -raster,vector- (rwv): JPEG-2000 driver based on OpenJPEG library
L1B -raster- (rovs): NOAA Polar Orbiter Level 1b Data Set
FIT -raster- (rwv): FIT Image
GRIB -raster- (rwv): GRIdded Binary (.grb, .grb2)
RMF -raster- (rw+v): Raster Matrix Format
WCS -raster- (rovs): OGC Web Coverage Service
WMS -raster- (rwvs): OGC Web Map Service
MSGN -raster- (rov): EUMETSAT Archive native (.nat)
RST -raster- (rw+v): Idrisi Raster A.1
INGR -raster- (rw+v): Intergraph Raster
GSAG -raster- (rwv): Golden Software ASCII Grid (.grd)
GSBG -raster- (rw+v): Golden Software Binary Grid (.grd)
GS7BG -raster- (rw+v): Golden Software 7 Binary Grid (.grd)
COSAR -raster- (rov): COSAR Annotated Binary Matrix (TerraSAR-X)
TSX -raster- (rov): TerraSAR-X Product
COASP -raster- (ro): DRDC COASP SAR Processor Raster
R -raster- (rwv): R Object Data Store
MAP -raster- (rov): OziExplorer .MAP
KMLSUPEROVERLAY -raster- (rwv): Kml Super Overlay
WEBP -raster- (rwv): WEBP
PDF -raster,vector- (rw+vs): Geospatial PDF
Rasterlite -raster- (rwvs): Rasterlite
MBTiles -raster,vector- (rw+v): MBTiles
PLMOSAIC -raster- (ro): Planet Labs Mosaics API
CALS -raster- (rwv): CALS (Type 1)
WMTS -raster- (rwv): OGC Web Map Tile Service
SENTINEL2 -raster- (rovs): Sentinel 2
MRF -raster- (rw+v): Meta Raster Format
PNM -raster- (rw+v): Portable Pixmap Format (netpbm)
DOQ1 -raster- (rov): USGS DOQ (Old Style)
DOQ2 -raster- (rov): USGS DOQ (New Style)
PAux -raster- (rw+v): PCI .aux Labelled
MFF -raster- (rw+v): Vexcel MFF Raster
MFF2 -raster- (rw+): Vexcel MFF2 (HKV) Raster
FujiBAS -raster- (rov): Fuji BAS Scanner Image
GSC -raster- (rov): GSC Geogrid
FAST -raster- (rov): EOSAT FAST Format
BT -raster- (rw+v): VTP .bt (Binary Terrain) 1.3 Format
LAN -raster- (rw+v): Erdas .LAN/.GIS
CPG -raster- (rov): Convair PolGASP
IDA -raster- (rw+v): Image Data and Analysis
NDF -raster- (rov): NLAPS Data Format
EIR -raster- (rov): Erdas Imagine Raw
DIPEx -raster- (rov): DIPEx
LCP -raster- (rwv): FARSITE v.4 Landscape File (.lcp)
GTX -raster- (rw+v): NOAA Vertical Datum .GTX
LOSLAS -raster- (rov): NADCON .los/.las Datum Grid Shift
NTv1 -raster- (rov): NTv1 Datum Grid Shift
NTv2 -raster- (rw+vs): NTv2 Datum Grid Shift
CTable2 -raster- (rw+v): CTable2 Datum Grid Shift
ACE2 -raster- (rov): ACE2
SNODAS -raster- (rov): Snow Data Assimilation System
KRO -raster- (rw+v): KOLOR Raw
ROI_PAC -raster- (rw+v): ROI_PAC raster
RRASTER -raster- (rw+v): R Raster
BYN -raster- (rw+v): Natural Resources Canada's Geoid
ARG -raster- (rwv): Azavea Raster Grid format
RIK -raster- (rov): Swedish Grid RIK (.rik)
USGSDEM -raster- (rwv): USGS Optional ASCII DEM (and CDED)
GXF -raster- (rov): GeoSoft Grid Exchange Format
DODS -raster- (ro): DAP 3.x servers
BAG -raster- (rwv): Bathymetry Attributed Grid
HDF5 -raster- (rovs): Hierarchical Data Format Release 5
HDF5Image -raster- (rov): HDF5 Dataset
NWT_GRD -raster- (rw+v): Northwood Numeric Grid Format .grd/.tab
NWT_GRC -raster- (rov): Northwood Classified Grid Format .grc/.tab
ADRG -raster- (rw+vs): ARC Digitized Raster Graphics
SRP -raster- (rovs): Standard Raster Product (ASRP/USRP)
BLX -raster- (rwv): Magellan topo (.blx)
EPSILON -raster- (rwv): Epsilon wavelets
PostGISRaster -raster- (rws): PostGIS Raster driver
SAGA -raster- (rw+v): SAGA GIS Binary Grid (.sdat, .sg-grd-z)
IGNFHeightASCIIGrid -raster- (rov): IGN France height correction ASCII Grid
XYZ -raster- (rwv): ASCII Gridded XYZ
HF2 -raster- (rwv): HF2/HFZ heightfield raster
JPEGLS -raster- (rwv): JPEGLS
OZI -raster- (rov): OziExplorer Image File
CTG -raster- (rov): USGS LULC Composite Theme Grid
E00GRID -raster- (rov): Arc/Info Export E00 GRID
ZMap -raster- (rwv): ZMap Plus Grid
NGSGEOID -raster- (rov): NOAA NGS Geoid Height Grids
IRIS -raster- (rov): IRIS data (.PPI, .CAPPi etc)
PRF -raster- (rov): Racurs PHOTOMOD PRF
RDA -raster- (ro): DigitalGlobe Raster Data Access driver
EEDAI -raster- (ros): Earth Engine Data API Image
SIGDEM -raster- (rwv): Scaled Integer Gridded DEM .sigdem
GPKG -raster,vector- (rw+vs): GeoPackage
CAD -raster,vector- (rovs): AutoCAD Driver
PLSCENES -raster,vector- (ro): Planet Labs Scenes API
NGW -raster,vector- (rw+s): NextGIS Web
GenBin -raster- (rov): Generic Binary (.hdr Labelled)
ENVI -raster- (rw+v): ENVI .hdr Labelled
EHdr -raster- (rw+v): ESRI .hdr Labelled
ISCE -raster- (rw+v): ISCE raster
HTTP -raster,vector- (ro): HTTP Fetching Wrapper

python3 --version

Python 3.6.9

pip3 --version

pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

rasterio from pypi:

rasterio-1.1.2-cp36-cp36m-manylinux1_x86_64.whl

pip3 freeze

affine==2.3.0
asn1crypto==0.24.0
attrs==19.3.0
Click==7.0
click-plugins==1.1.1
cligj==0.5.0
colorama==0.3.7
cryptography==2.1.4
GDAL==2.4.2
idna==2.6
keyring==10.6.0
keyrings.alt==3.0
numpy==1.18.1
pycrypto==2.6.1
pygobject==3.26.1
pyparsing==2.4.6
python-apt==1.6.4
pyxdg==0.25
rasterio==1.1.2
SecretStorage==2.3.1
six==1.11.0
snuggs==1.4.7
unattended-upgrades==0.1

[sgillies]

Confirmed. I'm not quite sure how to fix this one yet. When I do

$ rio info netcdf:/vsicurl/ftp://ftp.cdc.noaa.gov/Projects/RefET/global/Gen-0/fine_resolution/data_v2/2019/ETrs_fine_20191201.nc

I see

...
rasterio.errors.RasterioIOError: Missing url parameter

I've found where that error comes from in GDAL but am not sure how we get there.

[d-diaz]

I'm able to read a zipped GeoTiff on a publicly-accessible FTP server like this:

URL = 'ftp://islay.ceoas.oregonstate.edu/cms/v2/biomassfiaald/median/biomassfiaald_2018_median.zip
with rasterio.open('/vsizip/vsicurl/' + URL) as src:
    profile = src.profile

profile
>> {'driver': 'GTiff', 'dtype': 'int16', 'nodata': -32768.0, 'width': 153809, 'height': 96684, 'count': 1, 'crs': CRS.from_epsg(5070), 'transform': Affine(30.0, 0.0, -2356065.0,
       0.0, -30.0, 3172575.0), 'blockxsize': 256, 'blockysize': 256, 'tiled': True, 'interleave': 'band'}

[sgillies]

@d-diaz yes, the reported issue is specific to netcdf and still persists with the manylinux1 wheels for rasterio 1.1.5. Either netcdf and FTP don't work together with GDAL 2.4.4, or I am build netcdf and/or GDAL improperly.

GDAL and rasterio have a different approach to opening remote datasets.

What happens with ds = gdal.Open(ftp_uri) is that first GDAL downloads the entire "ftp://ftp.cdc.noaa.gov/Projects/RefET/global/Gen-0/fine_resolution/data_v2/2019/ETrs_fine_20191201.nc" to a temporary location on your computer and then opens it locally. Rasterio instead always dispatches these URLs to GDAL's viscurl mechanism, essentially rewriting ftp_uri to /vsicurl/{ftp_uri}.

To get the same experience as with GDAL Python, you can do something like this

import tempfile
from urllib.request import urlopen

import rasterio

url = "ftp://ftp.cdc.noaa.gov/Projects/RefET/global/Gen-0/fine_resolution/data_v2/2019/ETrs_fine_20191201.nc"

with tempfile.NamedTemporaryFile() as temp:
    with urlopen(url) as resp:
        chunk = resp.read(2 ** 16)
        while chunk:
            temp.write(chunk)
            chunk = resp.read(2 ** 16)
    with rasterio.open(temp.name) as dataset:
        print(dataset.profile)