Serverless-offline: Runs very slowly with AWS Python Lambda functions

[Jordan-Eckowitz]

I have setup a Serverless API Gateway with both Node.js (10.x) and Python (3.7) Lambdas.

I've compared simple endpoints in both languages that just return the string "hello world". It takes a few milliseconds with Node.js but about 2 seconds with Python to get a response.

When I run it from within a Docker container Node.js is as fast as outside the container. But, Python takes as long as 12 seconds to get a response.

Has anybody else had this issue with Python functions?

[dnalborczyk]

hey @Jordan-Eckowitz

I'm not a Python user, but serverless-offline is running against the serverless implementation, similar (or actually the same) as serverless invoke local ... you might have to look at their implementation, or look through the issues, or potentially file one. that is not to say that serverless-offline is not at fault and might need some perf improvements in that regard as well.

[dnalborczyk]

@Jordan-Eckowitz

just rewrote the (fairly buggy) python handling code, and it's running great locally. your specific issue should be fixed as well. we were spinning up a process (serverless invoke local), which in turn spun up another process (python). one of the sub processes is gone now, we're calling serverless invoke local directly.

[Jordan-Eckowitz]

Hey @dnalborczyk

Thanks for taking this on and re-opening the issue! I just tried updating to the latest v6 alpha release and its still the same for me.

It takes ~2s to return a hello-world response when running the API regularly and ~16s when running it within a Docker container.

As before, the Node.js response time is the same whether or not running via Docker - the same hello-world response takes ~5ms.

[dnalborczyk]

@Jordan-Eckowitz

it wasn't released yet. it was released with version alpha.15 and 16. could you try the latest (next) again? according to my measurements it's about ~ 25 times faster: https://github.com/dherault/serverless-offline/pull/785#issuecomment-522382513

closing as this should be resolved.

btw, it still has some rough edges to be ironed out. some feedback would be welcome!
e.g. python print does not work yet, windows might not work yet, error handling etc.

[Jordan-Eckowitz]

I tried updating to v6.0.0-alpha.20 and the python endpoints are now failing.

Error: Command failed with exit code 1 (EPERM): python3 -u /Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py index handler
    at makeError (/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/node_modules/execa/lib/error.js:58:11)
    at handlePromise (/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/node_modules/execa/index.js:112:26)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)
  command:
   'python3 -u /Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py index handler',
  exitCode: 1,
  exitCodeName: 'EPERM',
  stdout: '',
  stderr:
   `Traceback (most recent call last):\n  File "/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py", line 75, in <module>\n    module = import_module(args.handler_path.replace('/', '.'))\n  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 983, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked\nModuleNotFoundError: No module named 'index'`,
  all:
   `Traceback (most recent call last):\n  File "/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py", line 75, in <module>\n    module = import_module(args.handler_path.replace('/', '.'))\n  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 983, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked\nModuleNotFoundError: No module named 'index'`,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false,
  signal: undefined }
offline: Failure: Command failed with exit code 1 (EPERM): python3 -u /Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py index handler
Error: Command failed with exit code 1 (EPERM): python3 -u /Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/src/handler-runner/invoke.py index handler
    at makeError (/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/node_modules/execa/lib/error.js:58:11)
    at handlePromise (/Users/jeckowitz/Development/datalabs-api/node_modules/serverless-offline/node_modules/execa/index.js:112:26)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

[dnalborczyk]

@Jordan-Eckowitz just tried a couple combinations with https://github.com/dherault/serverless-offline/tree/master/__tests__/integration/python/python-big-json (I copied it out and changed the plugin-path in yml) serverless local install, global install, py in subfolder etc. all seem to work just fine. would you be able to create a small repository I could look into?

[Jordan-Eckowitz]

Hi @dnalborczyk ,

Thanks again for all the help - you're an absolute legend! I've re-created the issue and saved it in this repo:

https://github.com/Jordan-Eckowitz/serverless-offline-python-3

Please let me know if there's anything else you need from me.

[dnalborczyk]

😄 thank you!

alrighty, just debugged your issue. unfortunately I don't know why this is working in v5, but I can tell you why it's _not_ working in v6. it's this condition:

const argv = require('yargs').argv

module.exports.pyHandler = () => {
    if (argv._.find(arg => arg === "local")) return 'src/test/get/index.handler'
    return 'index.handler'
}

local is on the argv object (not argv._, which is an array) so you would have to check there, e.g.
if (argv.local != null) return ...

for a quick test you could just do:
const argv = require('yargs').argv

module.exports.pyHandler = () => {
return 'src/test/get/index.handler'
}

or move src/test/get/index.handler into the yml file: handler: src/test/get/index.handler

[Jordan-Eckowitz]

Great debugging! I also don't understand why the args would no longer be within the _ array. I've got it working using your recommendations. It works perfectly (and fast) for very simple endpoints.

Most of my endpoints, however, are not returning results anymore. It thinks for about a second and the call then ends without any logging information other than the message No JSON in the terminal. The status code being returned is still 200.

They're primarily Lambdas that retrieve SQL data via AWS boto3. All the endpoints are working with v5.

[dnalborczyk]

I've got it working using your recommendations. It works perfectly (and fast) for very simple endpoints.

Good to hear!

Most of my endpoints, however, are not returning results anymore. It thinks for about a second and the call then ends without any logging information other than the message No JSON in the terminal. The status code being returned is still 200.

I'm guessing you have some print(...) in your python code. that's one of the things I still need to look into. I'm trying to avoid the JSON detection stuff which was previously also being used.

stay tuned!

[Jordan-Eckowitz]

Ah yeah, that's it - I commented out the print statements and it worked like a charm! Thanks again, I'll look out for the updates as they roll out.

[dnalborczyk]

@Jordan-Eckowitz ah, nice!

I'm guessing the easiest would be just to assign to something specific here: https://github.com/dherault/serverless-offline/blob/master/src/handler-runner/invoke.py#L90

sys.stdout.write(json.dumps(result))

something like this, the javascript equivalent of { payload: result }, which we can then parse on the other side. unfortunately I have never written a single line of Python, and have to look every little thing up.

do you know the equivalent of the following in python by any chance?

const data = { payload: result }
JSON.stringify(data)

... ? ....
sys.stdout.write(json.dumps(data))

[dnalborczyk]

nm, I suppose it's something like this:

data = {
  "payload": result
}

sys.stdout.write(json.dumps(data))

[dnalborczyk]

@Jordan-Eckowitz

the fix should be available in the latest alpha version. so you should be able to still print(..) in your python code. let me know if you notice any problems. just a reminder, there's still some things needed to be cleaned up, e.g. error handling and stuff like that.

btw, you should be able to use websockets with python and ruby now as well. I haven't tested it myself yet, but I hooked everything needed up.

[Jordan-Eckowitz]

Hey @dnalborczyk - I tried it earlier and working without any issues - great job!