Serverless-webpack: SLS webpack invoke (v2) + SLS invoke local (V3) don't terminate once the response has been sent

[btaylor-ihr]

This is a Bug Report

Description

I'm trying to run sls webpack invoke -f some_function inside a docker container, but the command hangs and I have to stop the machine in another terminal window. Hitting Ctrl+C/D doesn't work. I'm getting Docker to do it as it's part of a Jenkins pipeline, and that's how other devs are doing this in my company.

I expect sls webpack invoke -f some_function to terminate once the callback has been executed. I'm not running it with the --watch flag.

I've tried upgrading to 2.2.2, but I get the same issue (still using sls webpack invoke -f some_function even though the docs say it's been disabled). I cannot run sls invoke local as I get SyntaxError: Unexpected token import as webpack hasn't actually done anything

My serverless.yml has the following plugins

plugins:
  - serverless-webpack
  - serverless-offline (Even with this turned off, it makes no difference)
  - serverless-jest-plugin (Even with this turned off, it makes no difference)

That is the exact order, and I've seen mention of webpack needing to be first, and it already is

Additional Data

Serverless-Webpack 2.2.0 (And 2.2.2)
Webpack 3.5.4
Serverless-Offline 3.15.3
Serverless 1.20.2
OS: OS X (local) / Ubuntu (docker)

Stack Trace

(function (exports, require, module, __filename, __dirname) { import { successResponse } from '../helpers'
                                                              ^^^^^^
SyntaxError: Unexpected token import
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at AwsInvokeLocal.invokeLocalNodeJs (/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:156:33)
    at AwsInvokeLocal.invokeLocal (/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:114:19)
    at AwsInvokeLocal.tryCatcher (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

webpack.config.js

const path = require('path')
const slsw = require('serverless-webpack')

require('source-map-support').install()

module.exports = {
  entry: slsw.lib.entries,
  devtool: 'source-map',
  target: 'node',
  externals: [
    /aws-sdk/ // Available on AWS Lambda
  ],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        loader: 'babel-loader'
      }
    ]
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js'
  }
}

babelrc

{
  "presets" : [
    [
      "env",
      {
        "presets": [
          "stage-3"
        ],
        "targets": {
          "node": "6.10"
        },
        "useBuiltIns": true,
        "modules": false,
        "loose": true
      }
    ]
  ],
  "plugins":[
    [
      "transform-async-to-module-method",
      {
        "module": "bluebird",
        "method": "coroutine"
      }
    ]
  ]
}

[btaylor-ihr]

I'm investigating the context. That might be the cause. I'm throwing an exception for testing, and that terminates just fine... Hmm...

[HyperBrain]

I experienced the issue yesterday too - and did not change the serverless-webpack version at all on my computer. Before it worked.
It seems that one of the dependencies causes the issue - candidates are serverless-offline or webpack.
I'll try to investigate too, which dependency it is.

Update below!

[HyperBrain]

BTW: I experienced the change in behavior with the 3.0.0 version of serverless-webpack which uses serverless invoke local, and a completely different approach.

[HyperBrain]

I tested it with two different projects of ours (with 3.0.0-rc.2). One project terminates correctly after showing the lambda results, one project does not. It looks for me now that the node main loop waits for an empty state and thus does not end the node process correctly. The tests were done in a bash shell, so the problem is independent from Docker.

I used serverless invoke local --function=<function> --path=<test event json> (the same should happen with v2 and webpack invoke)

The project that terminates correctly does not use any global lingering objects, the other one that does not terminate, uses a global REDIS connection that still executes at the end.
In AWS Lambda the behavior is that we configure the lambda to exit as soon as the final callback is called and not wait for the event loop to be empty.
I'm pretty sure that that's the core of the problem - the plugin must exit as soon as the terminating lambda callback has been called and the handler exits.

[HyperBrain]

Imo, the best approach to find and solve the problem would be to use V3 as there is much less plugin code involved and the problem can be found and isolated much better. Afterwards it should be backported to V2.

[btaylor-ihr]

Thanks for the quick response! I posted this before I left work, so I'll have a look at this in the morning.

The lingering objects might actually be a big clue. I'm not closing my db connection because I've been running into bugs if I do close the connection. I'm new to Promises and async/await. I'm not closing my database connection because there are queries that are somehow firing after the db gets closed. Serverless offline holds my hand there (somehow), but maybe that's all I need to fix :)

I'll update you tomorrow

[HyperBrain]

In my opinion lingering objects are often used (especially for database pools and connections), so this should be fixed in the plugin and not in the user code. I'll check if I can find a proper way to make the plugin behave as expected. Nevertheless you can also look into the plugin's code and check if it returns from the handler and really gets stuck at the very end.

[HyperBrain]

I investigated the problem. There is a major flaw in Serverless' invoke local (and thus webpack invoke as they use the same semantics).

The problem is, that the call of the handler does not await its end, but just calls it and expects a promise.
The termination only works correctly if your handler returns a promise and does not use the callbacks (which is an either/or).

The code in Serverless (aws/invokeLocal/index):

    return lambda(event, context, callback);

lambda is the "required" handler function export. As you see here, the function will return immediately if you use a strict callback pattern in your handler (and according to the AWS docs MUST NOT return a promise)! That leads to the immediate trigger of the after:invoke:local event, even before the handler has finished.

The solution for this issue would be, to check if the result of the lambda handler is a promise or not. In case it's a promise, just return it, otherwise return a promise that resolves on the callback parameter.
This makes sure that the event lifecycles are awaited correctly and the promise resolves or rejects on the callback calls in case the handler uses callbacks.

As V3 only uses invoke local, this must be fixed in Serverless first. We can decide later if we port the fix from Serverless into the old V2 branch into the similar implementation.

/cc @pmuens @eahefnawy
I can duplicate this as issue into the Serverless repo and provide a PR in a short time. I think this must be fixed soon as it breaks invoke local - regardless if with or without the plugin.

[eahefnawy]

@HyperBrain thanks for the mention, somehow it stood out of my 200+ notifications 😊 ... I think I understand what you mean. Would be great if you could migrate this issue to our repo so that we can keep track of it internally in our sprint board ✨

[HyperBrain]

@eahefnawy Will do :) The fix is nearly done already - I managed to get the event lifecycle in order now. There is still a hang, but I'm on that. The hang is related to dangling/lingering globals in the user's code, but should also be handled by local invoke. Expect a complete fix after this weekend - you know, unit tests take the longest 😄

[btaylor-ihr]

I apologise for not being able to do much on my side. Other things have been coming up at work. Though I can confirm that closing the database does terminate correctly :) Thank you so much for that tip

Personally, if you're going to fix it, perhaps just stick it in 3.x, as it's currently only in RC phase, so you can introduce breaking changes there. People might already have workarounds in place for 2.x

[hassankhan]

Hi @btaylor-ihr, any chance you could confirm the issue is resolved with the latest version of this plugin? 😄