Rasterio: Access to Google Cloud data (like S3)

[sgillies]

See http://www.gdal.org/cpl__vsi_8h.html#af80a70fd45cd7a2119f1a2f7da071870. I'm not an expert on Google Cloud, but it looks like datasets would be identified using a 'gs' scheme instead of 's3'.

[geowurster]

@sgillies Yes, that's the standard Google Cloud Storage URL format. I know $ gsutil uses boto, so we may be able to manage credentials similarly to S3. If not, the google-api-python-client may be of use.

I have used Google Cloud Platform in the past, so I may be able to help with this one, although I'd like to get some fixes around https://github.com/mapbox/rasterio/issues/986 finished first.

$ gsutil ls gs://earthengine-public/
No handlers could be found for logger "oauth2client.contrib.multistore_file"
gs://earthengine-public/ned_$folder$
gs://earthengine-public/srtm30_$folder$
gs://earthengine-public/datalab/
gs://earthengine-public/example-data/
gs://earthengine-public/export-map-sample/
gs://earthengine-public/landsat/
gs://earthengine-public/ned/
gs://earthengine-public/srtm30/
gs://earthengine-public/srtm90v4/
gs://earthengine-public/timelapse/

[perrygeo]

@geowurster if you haven't already started work on this feature, could we push this to post-1.0? The idea being we could implement this without a backwards-incompatible API change (hopefully)

[geowurster]

@perrygeo 👍 I haven't worked on it at all and don't anticipate a breaking API change.

[rabernat]

This feature is crucial for pangeo-data/pangeo#66. Any chance someone will start working on it any time soon? We would be very grateful downstream.

[sgillies]

@rabernat I can't speak for @geowurster, but I won't have time to work on this until after 1.0. I would, however, be very happy to review PRs, answer design questions, and provide tips.

[rabernat]

@sgillies thanks for your encouragement!

My understanding is that gdal already supports google storage
http://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsigs

So implementing this could be just a few lines of code here:
https://github.com/mapbox/rasterio/blob/master/rasterio/vfs.py#L66

Do you think that interpretation is correct? Or am I missing something?

[sgillies]

@rabernat close!

'gs': 'gs' must be added to the SCHEMES mapping (https://github.com/mapbox/rasterio/blob/master/rasterio/vfs.py#L17) and then, yes, the code at https://github.com/mapbox/rasterio/blob/master/rasterio/vfs.py#L69-L70 must be duplicated, substituting 'gs' for 's3' so that identifiers like gs://earthengine-public/landsat/foo.tif are dispatched correctly to GDAL.

This will, I think, give you a very minimal implementation. You won't be able to authenticate in Python code like you can for AWS right now, and must set GS_SECRET_ACCESS_KEY and GS_ACCESS_KEY_ID as environment variables.

[geowurster]

@rabernat I have no plans to work on this in the near future, but it _may_ already work with something like the code snippet below. These secret keys will probably be logged if Rasterio's logging is enabled so be careful. If you work up a PR I'm happy to review.

with rasterio.Env(GS_SECRET_ACCESS_KEY='****', GS_ACCESS_KEY_ID='****'):
    with rasterio.open('/vsigs/bucket/path/to/image.ext') as src:
        ...

[sgillies]

@geowurster good point, we're not sanitizing Google Cloud keys like we do for AWS.

Yes, that snippet should work. However, I'd like to make it clear that Rasterio's moral compass points 180 degrees away from /vsigs/ filenames 😄

[rabernat]

Thanks for this encouragement.

I will probably not personally be the one to implement this. My own work does not involve geotiffs, so it’s hard to justify spending time on it. My goal here is to keep the issue alive and show that it is relatively easy to fix.

As soon as someone comes along who really needs to use rasterio to read from GCS, hopefully this issue will help that person know what work has to be done.

On Jan 16, 2018, at 10:19 PM, Sean Gillies [e-mail] wrote:

@geowurster good point, we're not sanitizing Google Cloud keys like we do for AWS.

Yes, that snippet should work. However, I'd like to make it clear that Rasterio's moral compass points 180 degrees away from /vsigs/ filenames 😄

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[pbranson]

Wondering about the status of this? Was considering using rasterio with some jp2000 images in a google cloud bucket

[ArMoraer]

Hi, same issue here. This feature would be really helpful.

[sgillies]

I do not use any cloud other than Amazon's and cannot spend my own work or personal time implementing support for other clouds. What I will do is generalize the existing AWS code so that it's easier to add support for other clouds, and will consider support for other clouds to be within the scope of 1.0.x work.

[sgillies]

I've written to https://rasterio.groups.io/g/dev/message/11 about my work to generalize access to secured data in the cloud.

[rabernat]

It looks like this issue has been closed by #1577, no?