Moto: test_lambda.py:test_invoke_requestresponse_function test failing os x/linux since 1.1.20

[dbfr3qs]

Hi there, since moto 1.1.20 the above test has started to fail on my machine (OS X 10.11) and our Ubuntu 16.04 linux build agents, both Python 2.7 and 3.5. Any test that we have which uses invoke_lambda is now erroring out. There does not appear to be a clean version of moto I can pin to because of other changes that were made that were also included in this release that my tests rely on. Would it be possible to get this addressed ASAP?

I believe this is the commit which caused it:

https://github.com/spulec/moto/commit/9008b852995146b85296174108fc94c5f727bcb5#diff-2df06bdd95e3c78b9335da9f260a3472

Here is the stack trace:

~/D/W/moto (master↑3|…) $ nosetests tests/test_awslambda/test_lambda.py:test_invoke_requestresponse_function -s
Traceback (most recent call last):
  File "/Users/chris.keogh/Documents/Workspace/moto/moto/awslambda/models.py", line 291, in _invoke_lambda
    with _DockerDataVolumeContext(self) as data_vol:
  File "/Users/chris.keogh/Documents/Workspace/moto/moto/awslambda/models.py", line 94, in __enter__
    for vol in self._lambda_func.docker_client.volumes.list():
  File "/usr/local/lib/python2.7/site-packages/docker-2.5.1-py2.7.egg/docker/models/volumes.py", line 92, in list
    resp = self.client.api.volumes(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/docker-2.5.1-py2.7.egg/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/docker-2.5.1-py2.7.egg/docker/api/volume.py", line 38, in volumes
    return self._result(self._get(url, params=params), True)
  File "/usr/local/lib/python2.7/site-packages/docker-2.5.1-py2.7.egg/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/docker-2.5.1-py2.7.egg/docker/api/client.py", line 189, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(61, 'Connection refused'))
E
======================================================================
ERROR: test_lambda.test_invoke_requestresponse_function
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/chris.keogh/Documents/Workspace/moto/moto/core/models.py", line 70, in wrapper
    result = func(*args, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/moto/tests/test_awslambda/test_lambda.py", line 81, in test_invoke_requestresponse_function
    base64.b64decode(success_result["LogResult"]).decode('utf-8'))
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[JackDanger]

@thehesiod Can you take a look at this? Is there a way we can ship the lambda features without requiring people to be running Docker?

[dbfr3qs]

@JackDanger @thehesiod Thanks. I'm looking at the code now, I was a bit surprised that we now have running docker as a dependency and would recommend against this, particularly as it only seems to impact the invoke lambda function. I'm guessing this is why the test is passing in travis.

[thehesiod]

It's possible but require a lot of work as we'd need to be able to replicate the lambda environment. The reason we use a docker container is so that the tests run just as they would in AWS. @dbfr3qs is it not possible to have your test environment have docker support? If it's travis you just need to run w/ docker support enabled.

[dbfr3qs]

Firstly, thanks for replying so promptly!

Can I ask that this be better documented? Something somewhere that states that you now need to have a copy of lambic/lambda available and the docker service running? Maybe in the main README.md? Would it be possible to include some way of automatically checking/downloading that dependency when setting up development for moto itself?

However, my tests used to pass without the need for a docker image, I feel quite strongly that this should be an optional feature. Spare a thought for those of us who are using build systems within corporate structures where getting access to something like Docker is often completely out of our hands.

[thehesiod]

great idea for docs, thanks! I'll create a PR for this. This was my first feature to moto so bear with us :) Here's some of the reasons why this was done:

Previously lambdas were simply "exec"'d. This had several drawbacks... First: lambdas could corrupt your moto environment. Likewise since they ran in the same environment as the moto server they could not match the environment specified in the lambda (ex: run JS lambda would have never worked, or run 3.6 lambda from 2.7 moto server..these should all now work). In order to replicate this feature the client have to have a variety of tools installed on the user's system (node, python3.6, python2.7, or have one: docker.

I also highly believe in code-reuse. Since someone already solved this issue I believe we should take advantage and build upon that work. That benefits both parties. Further docker is widely used in production environments.

You shouldn't need to have the lambic/lambda images locally as they should get automatically pulled the image is not available. If that is not the case let us know.

Hmm, about automatically warning client if docker is missing I'm not sure how that would work...and further it's only required if you're going to use lambdas so we'd have to come up with something for that scenario, special exception when starting server/wrapping calls? Warning when installing moto?

So overall I think the pros highly outweigh the cons. Are you in a situation where you cannot have docker installed on the test system?

[thehesiod]

btw @dbfr3qs why are you running the moto tests? oh, I'm guessing this was just to point out that lambda's weren't working for you

[JackDanger]

@thehesiod If we could check for the existence of Docker and, if it's present and import docker works, load in the lambda functionality that'd be ideal. Then most users most of the time would keep their dependencies slim and then if they want to use lambda they have to do this one extra step. That seems like the right tradeoff to me. What do you think?

[dbfr3qs]

I run the tests because I contribute to moto, usually around lambda and dynamodb.

The moto test for invoke_lambda passes but with this:

{"errorMessage": "Handler 'handler' missing on module 'lambda_function'"}

..Traceback (most recent call last):
  File "/Users/chris.keogh/Documents/Workspace/moto/moto/awslambda/models.py", line 322, in _invoke_lambda
    'lambda invoke failed output: {}'.format(output))
Exception: lambda invoke failed output: START RequestId: 74cefacd-0ee8-4b71-8c62-0fb44483ea6e Version: $LATEST
failsauce: Exception
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 3, in lambda_handler
    raise Exception('failsauce')
Exception: failsauce

END RequestId: 74cefacd-0ee8-4b71-8c62-0fb44483ea6e
REPORT RequestId: 74cefacd-0ee8-4b71-8c62-0fb44483ea6e Duration: 0 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 14 MB

{"stackTrace": [["/var/task/lambda_function.py", 3, "lambda_handler", "raise Exception('failsauce')"]], "errorType": "Exception", "errorMessage": "failsauce"}

And the tests for my application which use invoke_lambda still fail:

{"errorMessage": "Unable to import module 'lambda_function'"}

ETraceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/moto/awslambda/models.py", line 321, in _invoke_lambda
    'lambda invoke failed output: {}'.format(output))
Exception: lambda invoke failed output: START RequestId: aefde1ec-2f8c-4c37-b30c-902569eedfc5 Version: $LATEST
Unable to import module 'lambda_function': No module named lambda_function
END RequestId: aefde1ec-2f8c-4c37-b30c-902569eedfc5
REPORT RequestId: aefde1ec-2f8c-4c37-b30c-902569eedfc5 Duration: 0 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 14 MB

{"errorMessage": "Unable to import module 'lambda_function'"}

E
======================================================================
ERROR: test_invalid_server_class (tests.unit.tests_frontend.test_unit_SRE_ReportCard_handler_Validate_ServerClass.TestReportCardHandler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/chris.keogh/Documents/Workspace/report-card-lambdas/tests/unit/tests_frontend/test_unit_SRE_ReportCard_handler_Validate_ServerClass.py", line 20, in test_invalid_server_class
    self.it_should_return_invalid()
  File "/Users/chris.keogh/Documents/Workspace/report-card-lambdas/tests/unit/tests_frontend/test_unit_SRE_ReportCard_handler_Validate_ServerClass.py", line 33, in it_should_return_invalid
    result = GetReports.valid_server_class(self.serverclass)
  File "/Users/chris.keogh/Documents/Workspace/report-card-lambdas/frontend/SRE_ReportCard_handler.py", line 40, in valid_server_class
    response = func.invoke(GET_SERVERCLASSES_FUNCTION_NAME, { 'action': 'get_serverclasses' })
  File "/Users/chris.keogh/Documents/Workspace/report-card-lambdas/frontend/SRE_ReportCard_handler.py", line 28, in invoke
    return json.loads(resp.decode())
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[thehesiod]

@JackDanger ya that's easy, but based on @dbfr3qs he's going to need this functionality because he's working on lambda functionality. So I'm guessing that won't help in this case. Plus then I bet we'll get a ton of bugs logged asking why lambda functionality is not available.

@dbfr3qs in your first example, that's coming from the test_invoke_lambda_error which explicitly causes the lambda to fail with that exception (negative test). Can you post your testcase?

[JackDanger]

So I'm guessing that won't help in this case. Plus then I bet we'll get a ton of bugs logged asking why lambda functionality is not available.

It may not help for this particular case but I recently realized how many different dependencies the docker library pulls in. For example, anybody using Moto in their application now has to bundle moto, docker, and websockets-client. For companies that need to audit all of the dependencies they use that's asking a lot of our users. Having the default be to not support Lambda (and print a big, obvious warning banner if they try to @mock_lambda showing users what they need to do) and allowing folks to opt-in to Docker would be kinder, I think.

[thehesiod]

btw @dbfr3qs the previous implementation of lambdas was incorrect in terms of file names, see https://github.com/spulec/moto/commit/9008b852995146b85296174108fc94c5f727bcb5#diff-eb9175e47e0880ffa6b4ae52862d55b6R21. Make sure you embed a file called "lambda_function.py" in your zip file (based on your previous exception stack). This is most likely your issue.

[JackDanger]

I'm feeling this strongly after spending much of yesterday writing XML code so we can remove the GPL'd dicttoxml from our app. A Moto user wasn't able to do their job because of licensing problems. This is the kind of problem we have when folks like @thehesiod, @dbfr3qs, and @terrycain made a project this successful! :)

(exactly the kind of problem I want to have more of)

[thehesiod]

ok created a POC here: https://github.com/spulec/moto/pull/1237

[terrycain]

I'll have to PR less ha :wink:

[thehesiod]

@terrycain you and me both =) I really hope we keep the lambda running in docker, it's really useful for my testcase. Pretty cool all the stuff I have daisy chained together with it. Basically have an outer unittest, running two containers that trigger a separate lambda ;)

[terrycain]

I do quite like that docker is there now, I have a few lambdas coming up the pipeline and would benefit from this.

Pretty cool all the stuff I have daisy chained together with it. Basically have an outer unittest, running two containers that trigger a separate lambda ;)

Lol nice, its great that stuff like that actually works and works well.

[JackDanger]

@thehesiod If you were to write a blog post showing a simple case of how you use lambda in Moto that would be so, so useful to Python developers everywhere. Basically, nobody I know has any way to test their lambda code anywhere. This is a huge feature!

[terrycain]

@JackDanger Once batch is done, I was thinking of spamming lots of docs too, as there isnt much docs on how to poke the backends of moto whilst testing... leaving people to running other boto commands to describe any changes they want to see. And probably show lots of examples, people like examples

[dbfr3qs]

@thehesiod thanks for the tip on the lambda function name, that fixed my tests! You have no idea how grateful I am! 👍

@JackDanger I think that's an excellent compromise. After working through it, I only had to make one small change which @thehesiod pointed out to get my tests to run again with Docker. I'm feeling a lot less negatively towards it, but I do think we need some clear documentation around it.

@JackDanger @thehesiod @terrycain thanks for your help. This is a great project, and I feel privileged to be involved.

[dbfr3qs]

Real quick, unrelated thought... @thehesiod is there a way to pass environment variables to the docker container running the lambda?

[thehesiod]

ok, wrote up something quick here: http://hesiod.blogspot.com/2017/10/mocking-aws-lambdas.html

@dbfr3qs yes! I fixed that in my PR as well. See in my blogpost. When you create the function you can specify the environment variables it should be run with

[dbfr3qs]

That is awesome!

[thehesiod]

btw with some work the MotoService can be largely eliminated by incorporating a shutdown and/or reset endpoint to the moto.server...also adding a context class helper and wait for service to come up. Sorry, I could go on for ideas, so many features, so little time :)

[terrycain]

+1 for context manager and more endpoints