Botocore: Stubbing custom commands

[foreverc9]

When calling stubber.add_response with certain boto3 s3 client methods I have encountered "botocore.model.OperationNotFoundError".

import unittest

import boto3
from botocore.stub import Stubber

def setup_s3_client():
    return boto3.client('s3')

MOCK_BUCKET_NAME = 'test-bucket'

create_bucket_response = {
    'Location': '/' + MOCK_BUCKET_NAME,
    'ResponseMetadata': {
        'HTTPStatusCode': 200,
        'HostId': 'myHostId',
        'RequestId': '80678CD98525D782'
    }
}

upload_file_response = {
    'ServerSideEncryption': 'AES256',
    'ETag': 'string',
    'SSECustomerAlgorithm': 'string',
    'SSECustomerKeyMD5': 'string',
    'SSEKMSKeyId': 'string',
    'RequestCharged': 'requester'
}

class TestS3Bucket(unittest.TestCase):

    def setUp(self):
        self.client = setup_s3_client()
        self.stubber = Stubber(self.client)
        self.stubber.activate()

    def test_create_bucket(self):
        self.stubber.add_response(
            method='create_bucket',
            service_response=create_bucket_response,
            expected_params={'Bucket': MOCK_BUCKET_NAME})

        self.client.create_bucket(Bucket=MOCK_BUCKET_NAME)


    def test_upload(self):
        self.stubber.add_response(
            method='upload_file',
            service_response=upload_file_response,
            expected_params={'Bucket': MOCK_BUCKET_NAME, 'Filename': 'fake_source_uri', 'Key': 'test_key'})

        self.client.upload_file(Filename='fake_filename',
                                Bucket=MOCK_BUCKET_NAME,
                                Key='fake_key')

The test results are shown as follows.

       @instance_cache
    def operation_model(self, operation_name):
        try:
            model = self._service_description['operations'][operation_name]
        except KeyError:
>           raise OperationNotFoundError(operation_name)
E           botocore.model.OperationNotFoundError

env_name/lib/python3.5/site-packages/botocore/model.py:249: OperationNotFoundError
=========================================================================== 1 failed, 1 passed in 0.29 seconds ============================================================================

The add_response method does not seem to allow the 'upload_file' client method to be stubbed. I have also found this to be true with the 'download_file' client method.

[JordonPhillips]

That's because download_file and upload_file are customizations which live in boto3. They call out to one or many requests under the hood. Right now there's not a great story for supporting customizations other than recording underlying commands they use and adding them to the stubber. There's an external library that can handle that for you, though we don't support it ourselves.

[jnoring]

Just hit this too. It'd be really great if stubber could handle download_file and upload_file

[montaro]

@JordonPhillips Thanks for your reply. What's the external library that mocks the upload_file()?

[JordonPhillips]

placebo, though it runs in a different way than the stubber does: it records and replays.

[luck02]

I just found out that 'get_paginator' seem to be the same story. It is a little confusing for me at this point as it seemed the stubber was for boto3, the relationship between botocore and boto3 is a bit unclear to me. I'll check out placebo.

[crizzy9]

Anyone found a workaround for this yet?

[bosatsu]

In case anyone else comes across, this I found that for get_paginator, rather than adding a response with get_paginator as the service method, instead you should add a response for the service method that get_paginator would access. For example

Instead of:

cfn_stubber = Stubber(cloudformation_client)
cfn_stubber.add_response('get_paginator', expected_response, 'describe_stacks')

Do:

cfn_stubber = Stubber(cloudformation_client)
cfn_stubber.add_response('describe_stacks', expected_response)

The stubber does seem able to return a paginator object that includes the data passed from the expected_response.