Rasterio: When there are AWS credentials in the environment, the session is ignored.

[wmaiouiru]

Expected behavior and actual behavior.

Expected: After setting boto3 credentials, rasterio environment should use the boto3 credentials.
Actual: Instead, when there are AWS credentials in the environment, the session credentials is ignored.

Steps to reproduce the problem.

Code in Rasterio: 1.0.3
with rasterio.Env(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key):

Code in Rasterio: 1.0.8
from rasterio.session import AWSSession

boto3_session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key)

with rasterio.Env(AWSSession(boto3_session)):

Local Dev Message when invoking rasterio.open:
Found credentials in shared credentials file: ~/.aws/credentials

Python AWS Lambda Message when invoking rasterio.open:
Found credentials in environment variables.

Checking rasterio.env.getenv(), the correct credentials are there, but when opening the image, I receive the following error:
[ERROR] - '/vsis3/<s3-bucket>/<path-name>' not recognized as a supported file format.

In Python AWS Lambda deployed using serverless.yml, the credentials are changed even after setting the enviroment variable in Makefile.
--env AWS_ACCESS_KEY_ID=<id>\ --env AWS_SECRET_ACCESS_KEY=<key>\

Setting the image to be publically accessible in s3 without using the session credentials, both Python AWS Lambda and local dev works.

Operating system

Dev: Ubuntu 16.04
Production: AWS Lambda Python 3.6

Rasterio version and provenance

rasterio[s3]==1.0.3
rasterio[s3]==1.0.8

[sgillies]

@wmaiouiru I am not able to reproduce this locally on my laptop. If I pass an empty boto3 session to Env, the logs show that boto3 finds credentials in my shared credentials file (expected) and hands them to Rasterio.

>>> import rasterio
>>> from rasterio.session import AWSSession
>>> import logging
>>> import boto3
>>> logging.basicConfig(level=logging.DEBUG)
>>> with rasterio.Env(AWSSession(boto3.Session())) as env:
...     print(env.options)
...
DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway
DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable data_path from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_timeout from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_num_attempts from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.credentials:Looking for credentials via: env
DEBUG:botocore.credentials:Looking for credentials via: assume-role
DEBUG:botocore.credentials:Looking for credentials via: shared-credentials-file
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x1188c27f0>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x1188c2ba8> created
DEBUG:rasterio._env:Logging error handler pushed.
DEBUG:rasterio._env:All drivers registered.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x1188c2ba8>.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable region from config file with value 'us-west-2'.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable region from config file with value 'us-west-2'.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x1188c27f0>
{'AWS_ACCESS_KEY_ID': '***', 'AWS_SECRET_ACCESS_KEY': '***', 'AWS_REGION': 'us-west-2'}
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x1188c27f0>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x1188c2ba8> options
DEBUG:rasterio._env:Stopping GDALEnv <rasterio._env.GDALEnv object at 0x1188c2ba8>.
DEBUG:rasterio._env:Error handler popped.
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x1188c2ba8>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x1188c27f0>

If I pass an access key and a secret key to boto3.Session() then these values are passed along to Rasterio, which is also what we expect.

>>> with rasterio.Env(AWSSession(boto3.Session(aws_access_key_id='foo', aws_secret_access_key='bar'))) as env:
...     print(env.options)
...
DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway
DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable data_path from defaults.
DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x1188da518>
DEBUG:rasterio.env:Starting outermost env
DEBUG:rasterio.env:No GDAL environment exists
DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x1188da8d0> created
DEBUG:rasterio._env:Logging error handler pushed.
DEBUG:rasterio._env:All drivers registered.
DEBUG:rasterio._env:Started GDALEnv <rasterio._env.GDALEnv object at 0x1188da8d0>.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable region from config file with value 'us-west-2'.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable region from config file with value 'us-west-2'.
DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x1188da518>
{'AWS_ACCESS_KEY_ID': 'foo', 'AWS_SECRET_ACCESS_KEY': 'bar', 'AWS_REGION': 'us-west-2'}
DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x1188da518>
DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x1188da8d0> options
DEBUG:rasterio._env:Stopping GDALEnv <rasterio._env.GDALEnv object at 0x1188da8d0>.
DEBUG:rasterio._env:Error handler popped.
DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x1188da8d0>.
DEBUG:rasterio.env:Exiting outermost env
DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x1188da518>

I'm using boto3==1.9.6 and botocore==1.12.6.

[wmaiouiru]

@sgillies Thank you for the follow-up.
I am wondering if I am using the environment incorretly.

import boto3
import rasterio
from rasterio.session import AWSSession
with rasterio.Env(AWSSession(boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key))) as env:
    print(env.options)
    src = rasterio.open(s3_url)

The following is my logs.
It looks like rasterio is exisiting the env context before opening the file.

[DEBUG] 2018-10-15T16:54:08.267Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Entering env context: <rasterio.env.Env object at 0x7f13f7b8b0f0>
[DEBUG] 2018-10-15T16:54:08.267Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Got a copy of environment <rasterio._env.GDALEnv object at 0x7f13fb022da0> options
[DEBUG] 2018-10-15T16:54:08.267Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Entered env context: <rasterio.env.Env object at 0x7f13f7b8b0f0>
{'AWS_ACCESS_KEY_ID': '***', 'AWS_SECRET_ACCESS_KEY': '***', 'AWS_REGION': 'us-west-2'}
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Exiting env context: <rasterio.env.Env object at 0x7f13f7b8b0f0>
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Cleared existing <rasterio._env.GDALEnv object at 0x7f13fb022da0> options
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Stopping GDALEnv <rasterio._env.GDALEnv object at 0x7f13fb022da0>.
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Error handler popped.
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f13fb022da0>.
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    No GDAL environment exists
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    New GDAL environment <rasterio._env.GDALEnv object at 0x7f13fb022da0> created
[DEBUG] 2018-10-15T16:54:08.455Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Logging error handler pushed.
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    All drivers registered.
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Started GDALEnv <rasterio._env.GDALEnv object at 0x7f13fb022da0>.
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Exited env context: <rasterio.env.Env object at 0x7f13f7b8b0f0>
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Exiting env context: <rasterio.env.Env object at 0x7f13fb022d68>
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Cleared existing <rasterio._env.GDALEnv object at 0x7f13fb022da0> options
[DEBUG] 2018-10-15T16:54:08.456Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Stopping GDALEnv <rasterio._env.GDALEnv object at 0x7f13fb022da0>.
[DEBUG] 2018-10-15T16:54:08.457Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Error handler popped.
[DEBUG] 2018-10-15T16:54:08.457Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f13fb022da0>.
[DEBUG] 2018-10-15T16:54:08.457Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Exiting outermost env
[DEBUG] 2018-10-15T16:54:08.457Z    edd1c9f5-d09a-11e8-a2e1-514fa391bbb6    Exited env context: <rasterio.env.Env object at 0x7f13fb022d68>
[ERROR] - '/vsis3/<url>.tif' not recognized as a supported file format.

[sgillies]

@wmaiouiru thanks for reporting! This is looking like a bug to me. I'll follow up here with news about the fix.

[fungjj92]

I'm seeing similar behavior, however instead of the specified AWS profile, rasterio picks up the [default] AWS profile credentials instead.

s = boto3.Session(profile_name='apple')
with rasterio.Env(s):
    rasterio.open('s3://apple-data/fall_for_18_3k.tif', mode='r')

RasterioIOError: '/vsis3/apple-data/fall_for_18_3k.tif' does not exist in the file system, and is not recognized as a supported dataset name.

I'm on MacOSX Sierra 10.12.6
rasterio[s3]==1.0.8
Python 2.7 and 3.7

[Kirill888]

I think this is because rasterio.env.ensure_env_credentialled creates new AWSSession with default settings even when running inside already credentialed environment see issue #1513. Quick work around is to not call open but instead create DatasetReader directly:

src = rasterio.DatasetReader(rasterio.path.parse_path(url))

[sgillies]

@wmaiouiru @fungjj92 I've got the fix in d5f2cd6. Sorry about the bug!