Rasterio: Rasterio read from Minio - S3 compatible Object Storage Server

[Giangblackk]

I'm working with rasterio to access my own local Object Storage Server using Minio instead of AWS S3.
This is what I tried in ipython notebook:

%env AWS_ACCESS_KEY_ID=<minio-access-key>
%env AWS_SECRET_ACCESS_KEY=<minio-secret-access-key>
%env ENDPOINT_URL=http://localhost:9000
!sudo apt-get install ca-certificates
import rasterio
dataset = rasterio.open('s3://fdi-data/image_01.png')

Here is what I got.

CPLE_OpenFailedError                      Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

rasterio/_shim.pyx in rasterio._shim.open_dataset()

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsis3/fdi-data/image_01.png' does not exist in the file system, and is not recognized as a supported dataset name.

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
<ipython-input-15-d1b17d97ee6c> in <module>()
----> 1 rasterio.open('s3://fdi-data/image_01.png')

~/virtualenv/lib/python3.5/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, **kwargs)
    238             # None.
    239             if mode == 'r':
--> 240                 s = DatasetReader(fp, driver=driver, **kwargs)
    241             elif mode == 'r-':
    242                 warnings.warn("'r-' mode is deprecated, use 'r'",

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

RasterioIOError: '/vsis3/fdi-data/image_01.png' does not exist in the file system, and is not recognized as a supported dataset name.

I don't now whether rasterio currently doesn't work with Minio or I'm doing something wrong?

Is there any solution?
Thank you very much.

Operating system

For example: Ubuntu Xenial 16.04.3.

Rasterio version and provenance

the 1.0a12 ubuntu wheel installed from PyPI using pip 9.0.1.

[sgillies]

Rasterio doesn't specially support Minio. But if Minio makes your PNG publically accessible via HTTP, you should be able to do something like rasterio.open('http://localhost:9000/fdi-data/image_01.png'). I say "publically accessible" because authentication options are limited in GDAL for http: resources until GDAL version 2.3 is released.

[glostis]

@Giangblackk I saw by chance that you have recently submitted an issue on https://github.com/OSGeo/gdal/issues/1609 in which you are trying to do in python gdal more or less what you wanted to do with rasterio in this issue.

Since you have succeeded to read the image with python gdal, there is no reason why it shouldn't work with rasterio (and rasterio's python API is much more comfortable to work with IMO).

I have successfully managed to access with rasterio files located on an S3-like file storage system similar to yours.
Since Minio seems to be an S3-compatible file-storage, accessing files with rasterio's S3 protocol handling should work as well.

You can leverage the rasterio.Env context manager to pass in GDAL env vars as keyword arguments. If you already have the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set in your environment, based on the solution you posted in your issue on GDAL, the following code snippet should work:

import rasterio

path = '/vsis3/bucket/image.tif'

with rasterio.Env(AWS_HTTPS='NO', GDAL_DISABLE_READDIR_ON_OPEN='YES', AWS_VIRTUAL_HOSTING=False, AWS_S3_ENDPOINT='localhost:9000'):
    with rasterio.open(path) as f:
        img = f.read()

[sgillies]

@glostis this is excellent advice. Thank you for the comment.