Moto: InvalidAccessKeyId - mocking no longer works

[thomashenson]

Versions

• boto==2.49.0
• boto3==1.9.225
• botocore==1.12.225
• moto==1.3.13

This issue has been raised many times before it appears, however after reading through all the reports and changing versions up and down to no avail, I've decided to raise another issue. When I try to mock out AWS using moto, I am getting the following error:

(InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

I've tried setting fake credentials using os.environ, I've tried deleting my credentials from the cli configuration, I've tried downgrading botocore/boto3 versions, I've tried re-arranging imports. None of this has helped unfortunately. Hopefully I'm missing something, but if not, there is something wrong.

The code

@mock_sqs
@mock_s3
@mock_sns
def test_bad_data(self):
    ...
    response = function.lambda_handler(
        None,
        None
    )
    ...

Calling the lambda_handler from the test is what lead us to the error:

def lambda_handler(event, context):
    ...
    try:
        ...
        s3 = boto3.resource('s3', region_name="eu-west-2")
        sqs = boto3.client('sqs', region_name="eu-west-2")
        lambda_client = boto3.client('lambda', region_name="eu-west-2")
        ...
        input_file = read_from_s3(s3, config['bucket_name'], config['file_name'])
        ...

It is where the read_from_s3 function is called that it tells me I have invalid credentials, which is the first instance of connecting to AWS. For reference, here is the read_from_s3 function:

def read_from_s3(s3, bucket_name, file_name):
    # Read from S3
    object = s3.Object(bucket_name, file_name)
    input_file = object.get()['Body'].read()

    return input_file

What was expected to happen
Successfully mock AWS in tests.

What actually happens
InvalidAccessKeyId error, suggesting that moto mocking isn't working.

If I've missed any important details please let me know so I can update.

[yabirgb]

Same issue here

[jkolaric]

I also have the same issue

[xiang-zhu]

I'm seeing this same issue, had figured out what's happening. It's not caused by library versions, I just updated to use latest version for the libraries

• boto: 2.49.0
• boto3: 1.9.232
• botocore: 1.12.232
• moto: 1.3.13

moto is injecting the botocore_stubber to botocore.handlers.BUILTIN_HANDLERS, but for my case, there's another import from code which includes a line of s3 = boto3.client('s3') call outside of functions, That import is before the import moto in my test, so it's causing boto creating the DEFAULT_SESSION without the extra botocore_stubber handler, and it was used when the AWS call in test supposed to be mocked.

The workaround is to set boto3.DEFAULT_SESSION to None before the usage of moto mock, and this will cause a creation of new DEFAULT_SESSION with the botocore_stubber handler.

I did it with a pytest fixture

````python
@pytest.fixture(scope="session")
def clear_default_boto3_session():
boto3.DEFAULT_SESSION = None

def test_with_moto(clear_default_boto3_session):
with mock_s3():
s3 = boto3.resource('s3', region_name='us-east-1')
assert [b for b in s3.buckets.all()] == []
````

Update: I also have to move the line of boto3.client in my code into a method to get the mock work with my code.

Update again: Instead of resetting the boto3.DEFAULT_SESSION and move my boto3.client call in code, I added import boto3 into my tests/__init__.py so it gets imported before anything else.

[spulec]

Can you please give a minimal full test case that I can use to reproduce?

It is hard to tell from the above, but it seems like the code you are running inside of the lambda isn't using Moto which is the expected behavior. See more here: https://github.com/spulec/moto/issues/2260

As of now, we haven't implemented a way for the lambda execution to automatically be mocked. The issue above gives some workarounds.

[xiang-zhu]

@spulec here's a simplified example of the issue I'm seeing https://gist.github.com/xiang-zhu/d441289bc2715b3e6d8723d7d3cd31af

This test will fail with InvalidAccessKeyId error as botocore api call was not mocked properly.

If you move the from moto import mock_s3 line in test_example.py before the from example import list_buckets the mock will be properly applied.

The issue become harder to notice when you have multiple tests file, and import from code happening in other tests.

Search through README, turns out this is a known issue https://github.com/spulec/moto#what-about-those-pesky-imports

[spulec]

Yes, that is a known issue. I believe the original issue here may be something else though.

[zenithlight]

moto is injecting the botocore_stubber to botocore.handlers.BUILTIN_HANDLERS, but for my case, there's another import from code which includes a line of s3 = boto3.client('s3') call outside of functions, That import is before the import moto in my test, so it's causing boto creating the DEFAULT_SESSION without the extra botocore_stubber handler, and it was used when the AWS call in test supposed to be mocked.

@xiang-zhu I have also encountered this same issue. The problem is some of my test files are trying to hit the real S3, and some are trying to hit the moto version. In both test files, boto3.resource was created in the top-level so it can be reused across all tests.

As a result, whenever one test file that used the real S3 was run, all proceeding tests that used moto would fail. Creating boto3.resource within the test run function (instead of top level of the test file) indeed caused mocking to work, but I do not know how to explain to future test writers that their tests started breaking because they initialized boto3 resources at the top level.

Can you explain why initializing the resource in a function causes mocking to work?

[cscetbon]

@spulec I have the same issue even without having a global call in my module. I'm trying to run

import re
import boto3
from moto import mock_s3
from s3_upload_split import __version__, SplitUploadS3


@mock_s3
def test_handle_content():
    bucket = 'my-bucket'
    conn = boto3.resource('s3', region_name='us-east-1')
    conn.create_bucket(Bucket=bucket)
    prefix = 'd1/d2/d3/d4'
    regex = re.compile(r'(\d{4}-\d{2})-\d{2} \d{2}:\d{2}:\d{2}')

    iterator = [
        dict(dt="2020-02-10 10:04:10", level=10, name="cyril"),
        dict(dt="2020-06-02 16:04:10", level=21, name="cynthia"),
        dict(dt="2019-07-03 12:24:08", level=12, name="steve"),
        dict(dt="2018-01-06 11:14:17", level=17, name="oliver"),
    ]

    SplitUploadS3(bucket, prefix, regex, iterator).handle_content()

It uses my module https://github.com/cscetbon/s3-upload-split
You can see the client is created the class constructor https://github.com/cscetbon/s3-upload-split/blob/master/s3_upload_split/__init__.py#L34

Any idea what's going on ?