Rasterio: Rasterio coerces netCDF uint8 data to float32

[felipk101]

Expected Behavior:

When rasterio opens netCDF variable of type uint8 it returns an array of type uint8.

Actual Behavior

When rasterio opens netCDF variable of type uint8 it returns an array of type float32.

Steps to reproduce the problem.
Please see attached notebook example and sample netcdf file with variables of uint8

Operating system

Both windows 10 and ubuntu

Rasterio version and provenance

pip install of rasterio version 0.36.0 and 1.0b1
rasterioIssue.zip

[sgillies]

Thanks for providing the data! I cannot confirm the issue with Rasterio 1.0b2 and so I will close this issue.

$ rio insp netcdf:/Users/sean/Downloads/rasterioIssue/ALOS2_PALSAR2_4326_50m_-4_79_20170430140716518085.nc:incidence_angle
Rasterio 1.0b2 Interactive Inspector (Python 3.6.4)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.profile
/Users/sean/code/rasterio/rasterio/rio/insp.py:1: RasterioDeprecationWarning: Creation keywords stored on datasets by Rasterio versions < 1.0b1 will always be ignored in version 1.0. You may opt in to ignoring them now by setting RIO_IGNORE_CREATION_KWDS=TRUE in your environment.
  """Fetch and edit raster dataset metadata from the command line."""
{'driver': 'netCDF', 'dtype': 'uint8', 'nodata': 1.0, 'width': 2129, 'height': 2213, 'count': 1, 'crs': CRS({'init': 'epsg:4326'}), 'transform': Affine(0.0004695459999999999, 0.0, -4.0,
       0.0, -0.0004517969999999989, 80.0), 'blockxsize': 250, 'blockysize': 250, 'tiled': True}
>>> data = src.read()
>>> data
array([[[1, 1, 1, ..., 1, 1, 1],
        [1, 1, 1, ..., 1, 1, 1],
        [1, 1, 1, ..., 1, 1, 1],
        ...,
        [1, 1, 1, ..., 1, 1, 1],
        [1, 1, 1, ..., 1, 1, 1],
        [1, 1, 1, ..., 1, 1, 1]]], dtype=uint8)

[alexgleith]

Hey @sgillies, I've tested just now with RasterIO 1.0b2 and still can reproduce the issue.

[sgillies]

@alexgleith interesting. Can you tell me more about the source of your Rasterio package? The raw string (r'NetCDF:...') suggests to me that you're on Windows, yes? What versions of HDF and netCDF libraries are you using? I have HDF5 1.10.1 and netCDF 4.4.1.1 in the macosx wheels I'm publishing to PyPI.

[alexgleith]

Hey @sgillies

This is actually running on a Ubuntu based Docker image...

Here's my library versions:

libhdf5-100:
  Installed: 1.10.0-patch1+docs-4
  Candidate: 1.10.0-patch1+docs-4
  Version table:
 *** 1.10.0-patch1+docs-4 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status
libnetcdf13:
  Installed: 1:4.6.0-2build1
  Candidate: 1:4.6.0-2build1
  Version table:
 *** 1:4.6.0-2build1 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

Edit: not sure what packages you're after... these are what I have installed for hdf5 and netcdf.

[sgillies]

@alexgleith thanks for the info. What version of GDAL do you have – what does print(rasterio.__gdal_version__) say?

[alexgleith]

Hey @sgillies,

print(rasterio.__gdal_version__)
2.2.4 

[sgillies]

And @alexgleith did you build Rasterio from source in your container or did you install the wheels from PyPI by running pip install rasterio==1.0b2?

[alexgleith]

Installed using PIP

For the test above, I installed with pip on the Jupyter server I've got.

[sgillies]

Confirmed @alexgleith

# rio insp netcdf:/data/rasterioIssue/ALOS2_PALSAR2_4326_50m_-4_79_20170430140716518085.nc:incidence_angle
WARNING:rasterio._gdal:CPLE_AppDefined in NetCDF driver detected file type=5, but libnetcdf detected type=3
WARNING:rasterio._gdal:CPLE_AppDefined in Unsupported netCDF datatype (7), treat as Float32.
Rasterio 1.0b2 Interactive Inspector (Python 3.5.2)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.profile
/root/venv/lib/python3.5/site-packages/rasterio/rio/insp.py:1: RasterioDeprecationWarning: Creation keywords stored on datasets by Rasterio versions < 1.0b1 will always be ignored in version 1.0. You may opt in to ignoring them now by setting RIO_IGNORE_CREATION_KWDS=TRUE in your environment.
  """Fetch and edit raster dataset metadata from the command line."""
{'tiled': False, 'nodata': 0.0, 'transform': Affine(0.0004695459999999999, 0.0, -4.0,
       0.0, -0.0004517969999999989, 80.0), 'count': 1, 'height': 2213, 'width': 2129, 'crs': CRS({'init': 'epsg:4326'}), 'driver': 'netCDF', 'dtype': 'float32'}

Evidently I've messed up something in my Linux wheel builds. We'll get this sorted out by the final 1.0.

[alexgleith]

Ok, thanks @sgillies. 👍

[sgillies]

I found a discrepancy in my manylinux1 and macosx wheel builds: in the Linux case, I've configured GDAL --without-hdf4 --without-hdf5. I'm going to remove these and test.

[sgillies]

Leaving HDF4 and HDF5 support in doesn't help, but I did notice some GDAL warnings related to the data type:

# rio insp !$
rio insp netcdf:/rasterioIssue/ALOS2_PALSAR2_4326_50m_-4_79_20170430140716518085.nc:incidence_angle
WARNING:rasterio._gdal:CPLE_AppDefined in NetCDF driver detected file type=5, but libnetcdf detected type=3
WARNING:rasterio._gdal:CPLE_AppDefined in Unsupported netCDF datatype (7), treat as Float32.
Rasterio 1.0b3 Interactive Inspector (Python 3.5.2)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> 

I checked the dataset variable and see warnings, but the uint8 type we expect

# rio insp netcdf:/rasterioIssue/ALOS2_PALSAR2_4326_50m_-4_79_20170430140716518085.nc:dataset
WARNING:rasterio._gdal:CPLE_AppDefined in NetCDF driver detected file type=5, but libnetcdf detected type=3
WARNING:rasterio._gdal:CPLE_AppDefined in dimension #1 (dataset_nchar) is not a Longitude/X dimension.
WARNING:rasterio._gdal:CPLE_AppDefined in dimension #0 (time) is not a Latitude/Y dimension.
WARNING:rasterio._gdal:CPLE_AppDefined in 1-pixel width/height files not supported, xdim: 2672 ydim: 1
/root/venv/lib/python3.5/site-packages/rasterio/__init__.py:221: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
Rasterio 1.0b3 Interactive Inspector (Python 3.5.2)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.profile
/root/venv/lib/python3.5/site-packages/rasterio/rio/insp.py:1: RasterioDeprecationWarning: Creation keywords stored on datasets by Rasterio versions < 1.0b1 will always be ignored in version 1.0. You may opt in to ignoring them now by setting RIO_IGNORE_CREATION_KWDS=TRUE in your environment.
  """Fetch and edit raster dataset metadata from the command line."""
{'transform': Affine(1.0, 0.0, 0.0,
       0.0, 1.0, 0.0), 'height': 1, 'crs': None, 'driver': 'netCDF', 'tiled': False, 'count': 1, 'dtype': 'uint8', 'nodata': 0.0, 'width': 2672}

So, focussing on "WARNING:rasterio._gdal:CPLE_AppDefined in Unsupported netCDF datatype (7), treat as Float32." seen when opening the incidence angle subdataset, it looks to me like the netcdf variable is a 64-bit unsigned int (https://www.unidata.ucar.edu/software/netcdf/docs_rc/data_type.html). This data type is not yet supported in any version of GDAL (it's a TODO in the source) and there's nothing Rasterio can do about it except raise a more helpful exception. I'm sorry about that!

I don't understand why there's no such problem on my macbook. There may be some subtle platform discrepancy in the netcdf library or netcdf driver.

[Kirill888]

@sgillies I pretty sure datatype (7) is UBYTE, not 64 bit:

https://www.unidata.ucar.edu/software/netcdf/docs_rc/netcdf_8h.html#a5bbc6bb240e4a5a9ab13a29e6f43f866

This matches what ncdump reports, so I guess GDAL isn't recognising this type in some configurations but does in others?

[sgillies]

@Kirill888 yes, thanks for straightening me out. I'm stumped by the behavior of the GDAL driver here. I'd be grateful for any pointers to working configurations!

[Kirill888]

Thanks for looking into this @sgillies,

it looks like for our use-case we need to ensure that GDAL used by rasterio has netcdf4 enabled

https://github.com/OSGeo/gdal/blob/7d078e0357d2998edfa713422e607cbadf77f9ff/gdal/frmts/netcdf/netcdflayer.cpp#L866-L882

[sgillies]

@Kirill888 thanks again. I've got this sorted out for the Rasterio 1.0rc1 release.

[alexgleith]

Hi all

Just wanted to confirm that the 1.0.0 build resolves this issue.