Moto: Can Mock lambda's make calls to other mock resources?

[dbfr3qs]

Just a question really, but with Lambdas now executing within docker containers, is it possible for mocked lambdas to access other mocked resources, such as DynamoDB tables?

Also, do functioning environment variables get created when passed in to create_function?

Probably something @thehesiod can answer?

[dbfr3qs]

Hmm, looks like you can't access other resources from within the lambda, I'm getting

ClientError: An error occurred (UnrecognizedClientException) when calling the PutItem operation: The security token included in the request is invalid.

While trying to access a mocked DynamoDB table. Hard to debug though...

[dbfr3qs]

What I ended up doing was using the DynamoDB local application on another docker container. In my test setup code I stand up the container and create my table with data, etc, then I just use moto to stand up my lambda as per usual. I then run my tests without making any changes to my test code (just the setup code). I supply an endpoint_url parameter within my lambda's aws client constructor method which points to the dynamodb container like so:

client = boto3.client('dynamodb', aws_region='us-west-2', endpoint_url='172.17.0.2:8000')

Since the default network for docker containers is bridge, it all works. I'm happy to go into more detail if anyone is interested.

It got me thinking however. Perhaps we could consider implementing the dynamodb local application as a docker container into moto, like what has been done with the lambda execution environment. This would mean two things:

1) Mocked lambdas could talk to mocked DynamoDB tables
2) All of the extra stuff not yet implemented within DynamoDB we'd get for free (e.g.. proper FilterExpressions, KeyConditions, all the complex stuff that involves parsing and tokenising strings).

I haven't looked at the license for DynamoDB local so not sure if that's legit or not.

Does anyone have thoughts? @JackDanger @terrycain @thehesiod

[thehesiod]

Sorry on paternity leave, I wrote a blog post about how I do it: http://hesiod.blogspot.com/2017/10/mocking-aws-lambdas.html?m=1

[thehesiod]

Btw later we can do this easier when mocking via proxy is added. IIRC I added a issue for this for us to track

[thehesiod]

And environment variables are correctly passed now, it's one of the things I fixed.

[dbfr3qs]

Cool, when you have time, can you elaborate on what mocking via proxy will involve?

I used your blog as the basis for what I ended up doing with DynamoDB local.

[terrycain]

@thehesiod Doesn't boto use ssl by default so it would just do a CONNECT through the proxy and we'd see nothing? unless im missing something.

[dbfr3qs]

@terrycain use_ssl is is set to True by default except when the endpoint_url parameter is used in a client/session constructor.

So, would a high level proxy design go something like: set an http proxy environment variable within the lambda container, create a proxy either in a container or directly in memory listening on some port (8080 for example, which the http proxy variable set in the lambda container points to), run any other mocked services in moto's server mode? Just trying to get my head around it, I wouldn't mind trying to make a start.

[thehesiod]

Just thinking about this: https://github.com/boto/botocore/blob/develop/botocore/config.py#L59 so we wouldn't need to specify one per service. Then Moto would have a proxy server and route appropriately

[dbfr3qs]

Wouldn't that involve somehow setting the proxies parameter on any boto
session objects in use around a test (even clients that are created by code
under test)? For example, how would this parameter be set on code being
executed within a lambda, within moto's container, that creates clients to
access other services? Or am I on the wrong track?

On 4 November 2017 at 16:35, Alexander Mohr [e-mail]
wrote:

Just thinking about this: https://github.com/boto/botocore/blob/develop/
botocore/config.py#L59 so we wouldn't need to specify one per service.
Then Moto would have a proxy server and route appropriately

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spulec/moto/issues/1317#issuecomment-341868859, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIcGff-euvSR-z9FOtYKMMBjhx9Q1oAFks5sy9uXgaJpZM4QNWye
.

--
Chris Keogh
Ph 021 1616692

[thehesiod]

My thought is via an environment variable triggering a global wrapt like
currently done (would avoid an environment variable per endpoint). Or if
botocore natively supported a proxy environment var that would be the
cleanest.

On Nov 4, 2017 2:58 PM, "Chris" [e-mail] wrote:

Wouldn't that involve somehow setting the proxies parameter on any boto
session objects in use around a test (even clients that are created by code
under test)? For example, how would this parameter be set on code being
executed within a lambda, within moto's container, that creates clients to
access other services? Or am I on the wrong track?

On 4 November 2017 at 16:35, Alexander Mohr [e-mail]
wrote:

Just thinking about this: https://github.com/boto/botocore/blob/develop/
botocore/config.py#L59 so we wouldn't need to specify one per service.
Then Moto would have a proxy server and route appropriately

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spulec/moto/issues/1317#issuecomment-341868859, or
mute
the thread
z9FOtYKMMBjhx9Q1oAFks5sy9uXgaJpZM4QNWye>
.

--
Chris Keogh
Ph 021 1616692

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spulec/moto/issues/1317#issuecomment-341932712, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AD0P_WFRqqenm9NwI54mQuoNvR35DADeks5szN33gaJpZM4QNWye
.

[dbfr3qs]

Oh, yep. On your blog you talk about the wrapt code being run in each container, including the lambda container. How does that code get run exactly? There is a comment that says "run in start up code"...

[thehesiod]

Ah, basically had that part always run from my main() function

[dbfr3qs]

Ah, so for example, in the lambda execution container - you have that code running in the lambda itself? Sorry, I think I'm missing something important that I obviously don't understand. Can you point me in the direction of some code that might help explain how the code in your blog is integrated?

[dbfr3qs]

I also had the thought: would it be possible to mock lambdas that call other lambdas?

[terrycain]

I also had the thought: would it be possible to mock lambdas that call other lambdas?

@dbfr3qs If we can mock 1 lambda successfully then in theory multiple will just work.

Ah, basically had that part always run from my main() function

@thehesiod that does mean users need to make moto aware lambdas.

Just a random thought, couldn't we just run a dodgy wrapper script that (language dependant) patches whatever aws library is in use to point to moto url?

[dbfr3qs]

@terrycain that's where I was heading and that's what was confusing me. I was hoping we could find a way to do this without having to put anything extra into Lambda code specific to testing (as it is now, I am using the endpoint_url parameter to point to, for example, a container running the dynamodb service).

[thehesiod]

basically my main currently looks something like:

def main():
    test_mock_endpoints = {name: value for name, value in os.environ.items() if 
    name.endswith("_mock_endpoint_url")}
    if test_mock_endpoints:
        patch_boto()

instead if moto supported proxies, we could just have one environment variable and don't even need to do the check, and look like this:

def _wrapt_boto_create_client(wrapped, instance, args, kwargs):
    def unwrap_args(service_name, region_name=None, api_version=None,
                    use_ssl=True, verify=None, endpoint_url=None,
                    aws_access_key_id=None, aws_secret_access_key=None,
                    aws_session_token=None, config=None):

        boto_proxy_url = os.environ.get('{}_mock_boto_proxy_url'.format())
        if config is None and boto_proxy_url:
            scheme, path = boto_proxy_url.split('://')
            config = botocore.config.Config(proxies={scheme: path})

        return wrapped(service_name, region_name, api_version, use_ssl, verify,
                       endpoint_url, aws_access_key_id, aws_secret_access_key,
                       aws_session_token, config)

    return unwrap_args(*args, **kwargs

with main now always calling it:

def main():
   patch_boto()

however, after looking at botocore's source it appears environment variables are in fact supported, so none of that is needed, see: https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L289

following that, it seems like you can just set the environment variable: http_proxy, or https_proxy. However this would set it for ALL requests from the requests module, so the moto proxy server would need to support fall through. Either that or we request botocore's environment variable checks be more specific to botocore. What do you guys think?

[thehesiod]

@dbfr3qs ya that's what I'm thinking. Set http proxy environment variable for lambda function or in client process, botocore picks this up on each request, forwards request to appropriate endpoint...Similar as how endpoint_url currently works but with routing. I'm not 100% on details as I've never written a proxy server before but I think it should be pretty simple.

[terrycain]

Well when proxying HTTPS, most things issue a HTTP CONNECT, which will just sort of tunnel traffic through the proxy and I highty doubt we'll see anything, (https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_method)

As for HTTP, Moto pretty much works as a http proxy server, it tries to infer services from the urls anyway, we'd just need to pass through unknown urls (via a catch all handler) to the remote destination (probably would be nicer in asyncio).

As for getting lambdas to work, we can either ask people to include snippets in their code which does patching, or just attempt to patch known libraries ourselves.

The only other thing I thought of, was to set lambdas dns for amazon urls to that of moto, and somehow trust ssl

[dbfr3qs]

I was thinking we could add a fall through config for the proxy server, which would allow people to use other 3rd party docker containers (or even real services, whatever) for requests to non AWS services the moto proxy does not know about. Like a hosts file for the proxy or something that explicitly defines routes.

I think setting the HTTP_PROXY/HTTPS_PROXY env variables is a good idea, within the lambda container.

I once implemented a proxy server in golang and it was straight forward. I do remember having to mess around with HTTPS requests but it wasn't anything too challenging.

If we went this way that would make it easy to trigger other lambdas from within lambdas - the requests would be to moto just like any other AWS service.

So how does this sound:

• Add HTTP_PROXY, HTTPS_PROXY server environment variables to the lambda container (very straight forward).
• Add a lambda proxy class to moto which listens and forwards requests for AWS services to moto and is configurable to allows requests to other services to be routed according to defined routes (that would actually be pretty interesting because the traffic needs to get back to the lambda container don't forget).

We could try for asyncio a bit later depending on how we go, but I don't think it's necessary for POC/MVP (as it were).

Thoughts? Am I missing anything obvious?

[terrycain]

Step2, I have a feeling anything will just blow up when ssl is invalid when talking to moto, but other than that, sounds good. Have a crack at it and see how far you get, then we can decide on what bit needs fixing next

[dbfr3qs]

No problem. I'll have a go during the weekend.

On 9 November 2017 at 13:36, Terry Cain [e-mail] wrote:

Step2, I have a feeling anything will just blow up when ssl is invalid
when talking to moto, but other than that, sounds good. Have a crack at it
and see how far you get, then we can decide on what bit needs fixing next

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spulec/moto/issues/1317#issuecomment-343008955, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIcGfWw1qrj926Uj0vjYPzn5y38ftXGsks5s0kjygaJpZM4QNWye
.

--
Chris Keogh
Ph 021 1616692

[dbfr3qs]

So I started looking into this. Most of the example http proxy implementations I looked at all operate directly on the sockets library, a little bit lower than requests. Moto looks to patch the requests library (https://github.com/spulec/moto/blob/master/moto/packages/responses/responses.py#L309).

I am unsure how to "route" the traffic from a sockets based proxy to moto (at this point I'm talking moto not running in servermode). Anyone have any theories that might help?

[thehesiod]

Btw There are ways to validate self signed certs via env vars and public certs

[terrycain]

@thehesiod that sounds good, you mind elaborating :-)

[thehesiod]

this is what I use: SSL_CERT_FILE=/PRIVATE_ROOT.pem python3 setup.py bdist_wheel so I think if you set that env var you can specify which private root to trust. I think there are some was to do it programmatically as well.

[thehesiod]

btw back from paternity leave, let me know if there's anything else I can clarify. to recap: I think by setting the proxy env vars we can proxy boto calls w/o changing any code in the lambda...however that would cause all requests via the "requests" library to also be proxied...so I think either the moto proxy forwards non-AWS requests along, or, we ask botocore to add support for separate botocore specific proxy env vars (easy to add).

[dbfr3qs]

Cool. Where I got stuck was trying to work out how to implement the proxy
in moto that listens for incoming traffic from the lambda and routes
accordingly. Would it be as straight forward as just following what happens
when moto is running in server mode?

On 14 November 2017 at 10:28, Alexander Mohr [e-mail]
wrote:

btw back from paternity leave, let me know if there's anything else I can
clarify. to recap: I think by setting the proxy env vars we can proxy boto
calls w/o changing any code in the lambda...however that would cause all
requests via the "requests" library to also be proxied...so I think either
the moto proxy forwards non-AWS requests along, or, we ask botocore to add
support for separate botocore specific proxy env vars (easy to add).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spulec/moto/issues/1317#issuecomment-344064228, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIcGfcOGWezI4gEtXlJid_i_90uEEVgPks5s2LRrgaJpZM4QNWye
.

--
Chris Keogh
Ph 021 1616692

[thehesiod]

hmm, so thinking this through, if we assume proxy mode is going to replace the old server mode, we'll have one endpoint to control all services. I believe if we configure it as a transparent proxy, it can just natively call the underlying methods in moto and return the response...so I think the proxy should act like a a set of the old server-mode endpoints...however just one port is exposed. So I'd start by seeing how the current server mode calls the moto methods, and then call them directly from the proxy. thoughts?

[thehesiod]

and for any services not implemented return 404 or something like that?

[thehesiod]

there are some examples of transparent proxy servers. If this can be py 3.4+ personally I'd try with asyncio so we can start moving in that direction. Thinking about making moto handle multiple requests in parallel is going to be complicated...I have a feeling that in server mode it's probably already not quite right as you can have multiple threads working on the same service at the same time currently (ex: SNS triggering lambda, and using lambda endpoint at the same time).

[terrycain]

Yeah in server mode, you can get things quite confused if you hammer it.

So server mode essentially runs flask, with a ton of url regex's with some glue code to pass the request to the responses.py files.

So from what I've read so far. Summing up:

• a transparent proxy, so that anything non aws still works fine.
• ssl would be good too, would mean possibly 0 changes to code if we can tell python and preferably other languages the CA (and if so we could probs just insert some ca cert into a lambda with SSL_CERT_FILE). We'd also need to tell the lambda some amazon domains resolve to whatever ip the moto server is on.
• having done some aiohttp server stuff for work, that would probably handle a https and http proxy fine I think. (Would also make lambda invoke_async work)

[thehesiod]

I'm not sure flask is even needed Sorry this was unrelated

[jeznag]

Is there a good way to handle this now? Maybe we can draw inspiration from how Localstack does it:

Additionally, the following read-only environment variables are available:

LOCALSTACK_HOSTNAME: Name of the host where LocalStack services are available. This is needed in order to access the services from within your Lambda functions (e.g., to store an item to DynamoDB or S3 from Lambda). The variable LOCALSTACK_HOSTNAME is available for both, local Lambda execution (LAMBDA_EXECUTOR=local) and execution inside separate Docker containers (LAMBDA_EXECUTOR=docker).

[thehesiod]

good no, but possible yes: http://hesiod.blogspot.com/2017/10/mocking-aws-lambdas.html?m=1
That's what I use for our multi aws service unittests.

What complicates this is that AWS endpoints typically are one host per service, ex: s3.amazonaws.com. That's why I proposed doing it at the proxy level so we could re-direct requests on a per host basis. I suppose another way would be through a hosts file. Theoretically the moto endpoint could check the HOST header to redirect as appropriate. That would probably be easier.

If botocore supported setting endpoint_url via environment variable it would be a lot easier.

There's a lot of possibilities :)

[jacobisaliveandwell]

It would be great if this got some renewed attention :)

[spulec]

Going to close this in favor of https://github.com/spulec/moto/issues/1108