rasterio.open fails on moto3 mocked s3 bucket

[dazza-codes]

Expected behavior and actual behavior.

Use moto3 to mock an s3 URI, upload a GeoTIFF to that mocked s3 object and use rasterio to read from it, using the s3 URI as input to rasterio.open. But it cannot open the s3 URI.

Steps to reproduce the problem.

import boto3
from moto import mock_s3

S3_MOCK_BUCKET_NAME = 'rasterio-mock-bucket'
S3_KEY = 'geotiff.tif'

def test_raster_to_strip_on_mock_s3():
    with mock_s3():
        conn = boto3.resource("s3", region_name="us-east-1")
        mock_s3_bucket = conn.create_bucket(Bucket=S3_MOCK_BUCKET_NAME)
        mock_s3_bucket.wait_until_exists()
        geotiff_file = '/tmp/geotiff.tif'  # just assume this exists already
        with open(geotiff_file, 'rb') as fp:
            mock_s3_bucket.upload_fileobj(fp, S3_KEY)

        # test the upload worked
        objects = [obj for obj in mock_s3_bucket.objects.all()]
        assert len(objects) == 1
        assert objects[0].key == S3_KEY
        assert objects[0].size > 0

        s3_uri = f's3://{S3_MOCK_BUCKET_NAME}/{S3_KEY}'
        src = rasterio.open(s3_uri)

Exception

>   ???
E   rasterio.errors.RasterioIOError: The specified bucket does not exist

rasterio/_base.pyx:218: RasterioIOError

Operating system

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

Rasterio version and provenance

From pypi wheels

$ pip freeze | grep oto
boto==2.49.0
boto3==1.9.232
botocore==1.12.232
moto==1.3.13

$ rio --version
1.0.28

$ poetry show --tree
rasterio 1.0.28 Fast and direct raster I/O for use with Numpy and SciPy
├── affine *
├── attrs *
├── click >=4.0,<8
├── click-plugins *
│   └── click >=4.0 
├── cligj >=0.5
│   └── click >=4.0,<8 
├── enum34 *
├── numpy *
└── snuggs >=1.4.1
    ├── numpy * 
    └── pyparsing >=2.1.6 

$ ls -1 ~/.cache/pypoetry/virtualenvs/ex-app-py3.6/lib/python3.6/site-packages/rasterio/.libs/
libaec-2147abcd.so.0.0.4
libcurl-835381e7.so.4.4.0
libexpat-09c47d4c.so.1.6.8
libgdal-ca0c78d5.so.20.5.2
libgeos-3-9b41901c.6.2.so
libgeos_c-1aedf783.so.1.10.2
libhdf5-8a18fd9e.so.101.1.0
libhdf5_hl-d68fbc5b.so.100.1.0
libjpeg-3fe7dfc0.so.9.3.0
libjson-c-5f02f62c.so.2.0.2
libnetcdf-6ec63985.so.11.0.4
libnghttp2-11cb20b8.so.14.17.1
libopenjp2-8f6da918.so.2.3.0
libpng16-898afbbd.so.16.35.0
libproj-cd06b982.so.12.0.0
libsqlite3-fdd57a2d.so.0.8.6
libsz-1c7dd0cf.so.2.0.1
libwebp-8ccd29fd.so.7.0.2
libz-a147dcb0.so.1.2.3

[sgillies]

Rasterio delegates the network I/O to GDAL, which runs in extension code, doesn't know anything about moto, and constructs its own HTTPS requests. You can see those requests by setting CPL_CURL_VERBOSE=1 in your environment. In a nutshell, you can't use moto to test rasterio like this. You would need a server that implements the AWS API.

It would be great if we could test rasterio in this way, but that would require moving a lot of code (all the virtual file system implementation) out of GDAL and into rasterio.

[dazza-codes]

Yes, of course. FYI

• http://docs.getmoto.org/en/latest/docs/server_mode.html
• but it's not obvious how gdal-vsis3 could be redirected to use it (and how to do that in pytest)

[sgillies]

@darrenleeweber in that case, I suggest reading about configuration of the endpoint GDAL makes requests to. Search https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files-random-reading for AWS_S3_ENDPOINT. I don't have a ton of experience with this but I've read that it is all that's required to use non-AWS S3-compatible storage systems. For some, that's Minio, but I think it's likely to work for server mode moto.

[dazza-codes]

OK, so tried using a moto[server] option, something like a synchronous version of:

• https://gitlab.com/dazza-codes/aio-aws/-/blob/master/tests/aiomoto_services.py

The test code is something like:

def test_raster_io_on_mock_s3(aws_moto_credentials, aws_region, geotiff_file):
    # To help with debugging this, use a command line pytest like
    # CPL_CURL_VERBOSE=1  pytest -s --pdb tests/test_raster_io.py

    with MotoService("s3") as svc:
        # svc.endpoint_url

        boto3.DEFAULT_SESSION = None
        s3_client = boto3.resource(
            "s3",
            region_name=aws_region,
            endpoint_url=svc.endpoint_url
        )
        bucket = s3_client.create_bucket(
            Bucket="mock-bucket",
            CreateBucketConfiguration={"LocationConstraint": aws_region}
        )
        bucket.wait_until_exists()

        with open(geotiff_file, "rb") as fp:
            bucket.upload_fileobj(fp, S3_KEY)

        # test the upload worked
        objects = [obj for obj in bucket.objects.all()]
        assert len(objects) == 1
        geotiff_object = objects[0]
        assert geotiff_object.key == S3_KEY
        assert geotiff_object.size > 0

        # The AWS_S3_ENDPOINT configuration option defaults to s3.amazonaws.com
        s3_url = urlparse(svc.endpoint_url)

        with rasterio.Env(
            AWS_HTTPS=False,
            AWS_NO_SIGN_REQUEST=True,
            AWS_S3_ENDPOINT=s3_url.netloc
        ) as env:

            src = rasterio.open(S3_MOCK_GEOTIFF_URI)

The moto[server] is running OK and the werkzeug logging indicates the bucket and object are created OK

$ CPL_CURL_VERBOSE=1  pytest -s --pdb tests/test_raster_io.py

...

2020-10-08 15:50:53 - werkzeug     - INFO     - 127.0.0.1 - - [08/Oct/2020 15:50:53] "GET /static HTTP/1.1" 404 -
2020-10-08 15:50:53 - werkzeug     - INFO     - 127.0.0.1 - - [08/Oct/2020 15:50:53] "PUT /mock-bucket HTTP/1.1" 200 -
2020-10-08 15:50:53 - werkzeug     - INFO     - 127.0.0.1 - - [08/Oct/2020 15:50:53] "HEAD /mock-bucket HTTP/1.1" 200 -
2020-10-08 15:50:53 - werkzeug     - INFO     - 127.0.0.1 - - [08/Oct/2020 15:50:53] "PUT /mock-bucket/test/geo.tif HTTP/1.1" 200 -
2020-10-08 15:50:53 - werkzeug     - INFO     - 127.0.0.1 - - [08/Oct/2020 15:50:53] "GET /mock-bucket?encoding-type=url HTTP/1.1" 200 -

However, the error from rasterio/gdal/curl is like:

* Couldn't find host mock-bucket.127.0.0.1 in the .netrc file; using defaults
* getaddrinfo(3) failed for mock-bucket.127.0.0.1:39867
* Couldn't resolve host 'mock-bucket.127.0.0.1'
* Closing connection 1
* Couldn't find host mock-bucket.127.0.0.1 in the .netrc file; using defaults
* getaddrinfo(3) failed for mock-bucket.127.0.0.1:39867
* Couldn't resolve host 'mock-bucket.127.0.0.1'
* Closing connection 2
* Couldn't find host mock-bucket.127.0.0.1 in the .netrc file; using defaults
* getaddrinfo(3) failed for mock-bucket.127.0.0.1:39867
* Couldn't resolve host 'mock-bucket.127.0.0.1'
* Closing connection 3

It seems to be impossible to get rasterio/gdal/curl to use the moto server endpoint. The rasterio/gdal/curl request internals insist on inserting the bucket name into the host name of the endpoint, which seems to be absurd.

[sgillies]

@dazza-codes that sounds frustrating. I haven't tried alternate endpoints before. Could you check the comment at https://github.com/mapbox/rasterio/issues/1293#issuecomment-502651617 and see if the AWS_VIRTUAL_HOSTING config option (which I confess to not understanding) needs to be set to false?

[glostis]

@sgillies from what I understood:

• with AWS_VIRTUAL_HOSTING=True, GDAL will build a URL that looks something like mybucket.cname.domain.com to access the bucket
• with AWS_VIRTUAL_HOSTING=False, the URL will be built like cname.domain.com/mybucket.

Some non-AWS S3 servers implement only one or the other route, so both need to be tested in order to know which one works.

@dazza-codes something that comes to mind when reading your code is that you do not pass your AWS region to rasterio. I don't know if it will make a difference since you are doing unsigned requests, but you could try changing your code to:

with rasterio.Env(
    AWS_HTTPS=False,
    AWS_NO_SIGN_REQUEST=True,
    AWS_S3_ENDPOINT=s3_url.netloc,
    AWS_DEFAULT_REGION=aws_region
) as env:

(or AWS_REGION if you have a GDAL version < 2.3)

FWIW, I was able to use rasterio with the S3 service of localstack without too much trouble, but I don't know how that compares to moto[server].