Serverless-webpack: JavaScript heap out of memory when packaging many functions

[daniel-cottone]

This is a Bug Report

Description

I'm in the process of trying to upgrade serverless-webpack version from 2.2.3, where I do not experience the following issue. Our serverless configuration has package: invididually: true set, and about 40 functions. When I try to upgrade to a later version of serverless-webpack and run sls webpack, the build will run for about a minute and then I get the following error:

lambda:daniel.cottone $ npm run build

> [email protected] build /Users/daniel.cottone/Projects/expert-api/lambda
> sls webpack --stage dev

Serverless: Bundling with Webpack...
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using [email protected] and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json

<--- Last few GCs --->

[42611:0x104001600]    55964 ms: Mark-sweep 1405.7 (1508.8) -> 1405.7 (1508.8) MB, 1721.0 / 0.0 ms  allocation failure GC in old space requested
[42611:0x104001600]    57889 ms: Mark-sweep 1405.7 (1508.8) -> 1405.5 (1487.3) MB, 1923.4 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1923 ms) last resort 
[42611:0x104001600]    59801 ms: Mark-sweep 1405.5 (1487.3) -> 1405.4 (1486.8) MB, 1903.6 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x37341f01ba79 <JS Object>
    1: set [native collection.js:~247] [pc=0x29d828934f21](this=0x332730f95301 <a Map with map 0x23d2df14319>,p=0x3dd499abec41 <String[11]: MediaSource>,x=0x2589b9b1c819 <a SymbolObject with map 0x399abfecde11>)
    2: /* anonymous */(aka /* anonymous */) [/Users/daniel.cottone/Projects/expert-api/lambda/node_modules/typescript/lib/typescript.js:~23166] [pc=0x29d828ba5830](this=0x37341f002241 <...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/usr/local/bin/node]
 8: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 9: 0x29d827e840bd
10: 0x29d828934f21
11: 0x29d828ba5830
12: 0x29d827e86bbb
13: 0x29d828f85beb
Abort trap: 6

If I change my serverless config to not package individually, package: individually: false then this error goes away. I have tested this with version 3.0.0 and the latest, 4.1.0 with the same results. Don't have this issue with 2.2.3.

Additional Data

• Serverless-Webpack Version you're using: 4.1.0
• Webpack version you're using: 3.10.0
• Serverless Framework Version you're using: 1.24.0
• Operating System: macOS 10.12.6
• Stack Trace (if available): see above

[HyperBrain]

Hi @daniel-cottone ,
thanks for reporting. This can be something with your configuration. SLS-webpack since 3.0.0 requires that you use slsw.lib.entries for your entry definitions and have the function handlers declared correctly in your serverless.yml in case you use individual packaging.

Can you post the function definitions from your serverless.yml and the webpack config file?

[daniel-cottone]

Hey @HyperBrain thanks for quick response. Here's the webpack configuration:

var path = require('path');
var slsw = require('serverless-webpack');
var Webpack = require('webpack');

module.exports = {
    entry: slsw.lib.entries,
    resolve: {
        extensions: ['.ts', '.js', '.json']
    },
    target: 'node',
    output: {
        libraryTarget: 'commonjs',
        path: path.join(__dirname, '.webpack'),
        filename: '[name].js'
    },
    externals: [{
        'aws-sdk': 'aws-sdk',
        'mysql2': 'mysql2',
        'sqlite3': 'sqlite3',
        'tedious': 'tedious',
        'pg-native': 'pg-native'
    }],
    module: {
        loaders: [
            { test: /\.ts(x?)$/, loader: 'ts-loader' }
        ]
    }
};

The definitions for all 40 functions is too large to post, but I'll post an example:

users-list:
    handler: src/handler/UserHandler.getUsers
    events:
        - http:
            path: users
            method: get

They pretty much all look the same, I've clipped out VPC, authorizer, and environment config. I'm pretty confident that they're all configured correctly.

[HyperBrain]

The handlers look good. However, version 2.x did not support individual packaging (in fact it only copied the whole artifact per function).
So you should, as next step, add node externals to your webpack configuration to let the externals be automatically determined by webpack, so that individual packaging can make use of it:

// webpack config
const nodeExternals = require('webpack-node-externals');
...
  externals: [ nodeExternals() ]
...

Additionally, webpack > 3.0.0 now uses a module: rules structure instead of module: loaders. You should change that too.

Please also check if you have set custom: webpackIncludeModules: true in your serverless.yml.

Then do a serverless package to test, if it works. You'll find the zip packages that would be uploaded in the .serverless directory.

If aws-sdk should be packaged, you can either put it into your devDependencies or use

# serverless.yml
custom:
  webpackIncludeModules:
    forceExclude:
      - aws-sdk

to keep it outside of your packages.

[daniel-cottone]

I've made your suggested changes to webpack externals and have added the webpackIncludeModules configuration to serverless custom config; I still seem to be experiencing the same problem though.

[daniel-cottone]

It also appears to be related to the fact that there are so many functions in this serverless project; if I comment out all but 5 then sls package works.

[HyperBrain]

Hmmm... that sounds like a memory leak somewhere when using individual packaging.
We also have a project with more than 30 functions which works, but I did not check how the memory consumption is there (i.e. if we're about to hit a limit).

What you can try is, to increase node's heap memory limit (which is at 1.7GB by default) with:
node --max-old-space-size=4096 node_modules/serverless/bin/serverless package to 4GB and check if it then passes with the full amount of functions.

If that works, we have to find out, where exactly the memory leak comes from and if it can be fixed by reusing objects.

[daniel-cottone]

That definitely seems to be the problem. I got much further along, looks like about 50% of the way through. If I bump it up to 12GB then the process finishes after about 8-10 minutes.

[HyperBrain]

Good to know - thanks for testing this 👍 . Can you adjust the title of the issue to reflect that this will happen with many functions? Then it's more clear how to reproduce it and we can find a solution.

Is the workaround using the increased heap ok for you as long as there's no real fix?

[daniel-cottone]

Sure thing. I think the 12GB heap size is probably a bit much; in addition to that it seems to run significantly slower than our build does currently. I'll just opt to not make use of individual packaging for now. If/when this does get fixed I can turn it on then.

[HyperBrain]

The slower runtime is expected, because it takes each webpack compile's output to determine the modules that are really needed for each function and assembles only these for the function package. That takes some time (when using --verbose you should see the exact steps including their timing).
The longer build outweighs the better startup behavior (if the lambdas are cold started) and if some big dependencies are only used by one function.

[daniel-cottone]

@HyperBrain That makes sense, thanks!

[BobbieBarker]

I tried rolling back versions until I found one that didn't experience this issue. I got to 2.2.2, at which point my webpack config didn't work anymore.

[HyperBrain]

@BobbieBarker Thanks for the investigation 👍
Support for individual packaging is available since 3.0.0. Versions prior to that (2.x) where just 1.x versions that I released with the most important fixes (the project was quite dead when I took it over). But these old versions did not do invidivual at all.

So I'm quite sure that the memory leak is somewhere in the individual packaging part (maybe the file copy). Did it also happen for you with a serverless package?
Does anyone here know, if there is a good node performance analyzer (profiler), that can track the heap and the GC (best would be graphically), so that I can see when it starts to allocate objects?

[HyperBrain]

I did some experiments with node's internal profiler node --trace_gc serverless package --verbose
with a project having 20+ functions (JS project).

The outcome is, that there seem to be no critical object remnants (or leaks) in the npm install or copy steps. The only step where memory consumption increases (but is always cleaned up by the GC) is the actual zipping of the function packaged.

This behavior matches the log above: It crashed for you at the webpack step! And it seemed to have loaded the ts-loader multiple times. For my tested JS project, the memory showed roughly the same fill state before and after the webpack run.

So for finding the root issue, we should concentrate on the webpack step and especially typescript. Did you experience the same issue without using typescript with projects that have many functions?
It seems that the webpack compile itself runs out of memory here.

[HyperBrain]

I thought a bit about the issue. A workaround could be that the plugin would run the compiles in batches of some functions at once. However I do not know, if the webpack library will free the allocated resources after the compile again. But it could be worth a try.

According to the crash trace it already happened after 7 compiled - if every ts-loader line is for one function - and was at 1500 MB.
[42611:0x104001600] 55964 ms: Mark-sweep 1405.7 (1508.8) -> 1405.7 (1508.8) MB, 1721.0 / 0.0 ms allocation failure GC in old space requested

The first try should be to disable some plugins in the webpack.config and check if the ts-loader might allocate all the memory.

[Birowsky]

I hit this too after setting

package:
  individually: true

I don't think I can declare anything else of significance other than having only 9 functions. Do ask tho, I'll check whatever necessary. Here's my webpack:

const {resolve} = require('path'); 
const slsWebpack = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');


module.exports = {
  target: 'node',
  devtool: 'inline-source-map',
  entry: slsWebpack.lib.entries,
  externals: [nodeExternals()],
  output: {
    libraryTarget: 'commonjs',
    path: resolve('builds/dist'),
    filename: '[name].js'
  },
  resolve: {
    extensions: ['.ts', '.js']
  },
  module: {
    rules: [
      loader({
        test: /\.ts$/,
        use: {
          loader: 'ts-loader',
          options: { }
        }
      }),
      loader({
        test: /\.graphqls$/,
        use: {
          loader: 'graphql-tag/loader',
        }
      })
    ]
  }
};



function loader(config) {
  return Object.assign(config, {
    // exclude: [/node_modules/,/builds/, /test/],
  });
}

These might be useful:

"ts-loader": "3.1.1",
"serverless": "1.25.0",
"serverless-webpack": "4.2.0",
"webpack": "3.10.0",
"serverless-plugin-cloudfront-lambda-edge": "1.0.0",

The output after running sls package:

Serverless: Bundling with Webpack...

<--- Last few GCs --->

[52295:0x103000000]    68990 ms: Mark-sweep 1407.7 (1499.7) -> 1407.6 (1477.7) MB, 2150.7 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2151 ms) last resort 
[52295:0x103000000]    71182 ms: Mark-sweep 1407.6 (1477.7) -> 1407.6 (1477.7) MB, 2191.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1e68da628799 <JSObject>
    1: bind(aka bind) [/Users/Birowsky/Projects/Personal/###obfuscated###/node_modules/typescript/lib/typescript.js:~21002] [pc=0x3a6d9eaeaaef](this=0x307b27782311 <undefined>,node=0x3624801cc691 <NodeObject map = 0x1a660d8c8f59>)
    2: forEachChild [/Users/Birowsky/Projects/Personal/###obfuscated###/node_modules/typescript/lib/typescript.js:~12719] [pc=0x3a6...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 4: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 5: v8::internal::TranslationBuffer::CreateByteArray(v8::internal::Factory*) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 6: v8::internal::compiler::CodeGenerator::PopulateDeoptimizationData(v8::internal::Handle<v8::internal::Code>) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 7: v8::internal::compiler::CodeGenerator::FinalizeCode() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 8: v8::internal::compiler::PipelineImpl::FinalizeCode() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
 9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
12: v8::internal::StackGuard::HandleInterrupts() [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/Birowsky/.nvm/versions/node/v8.6.0/bin/node]
14: 0x3a6d9e7846fd
Abort trap: 6

[Birowsky]

An update: it works when I set transpileOnly: true for ts-loader.

[HyperBrain]

@Birowsky Thanks for the info 🥇 . @BobbieBarker , @daniel-cottone can you confirm, that this setting also works for you?

[daniel-cottone]

@HyperBrain That setting does appear to be working for me. I'll look into using fork-ts-checker-webpack-plugin to maintain type checking. Thanks!

[Birowsky]

@daniel-cottone please share your thoughts after u succeed. I was thinking on doing a single tsc --noEmit before deploying, but maybe your approach is more rational.

[daniel-cottone]

@Birowsky Seems to work. Only gripe I could have is that the type checking doesn't fail fast; if you would prefer to check types before you even start the build, which could take some time, then maybe tsc --noEmit is a better option. For now I'm going to stick with just using the plugin

[VuBui83]

@daniel-cottone I've been dealing with the same issue for a couple weeks now. Using fork-ts-checker-webpack-plugin will spawn a thread per function to type check. I'm wondering if fork-ts-checker is smart enough to do just the type check for the specific lambda or it just type checks the entire project since it's based on tsconfig.json. My project has 20+ functions, fork-ts-checker spawns 20+ threads just for type checking. It works but I don't think it's necessary.

[VuBui83]

@HyperBrain with transpileOnly: true, it starts to crash around 30+ functions

[daniel-cottone]

@HyperBrain @VuBui83 I've also experienced the same problem; setting transpileOnly: true makes a huge difference but I still get crashes around 30 functions. I've also gone the route of manually type checking with tsc --noEmit rather than using fork-ts-checker-webpack-plugin.

[HyperBrain]

Maybe a solution would be to provide a PR for the ts-checker plugin that limits the number of spawned processes when using multi-compiles in webpack.

[HyperBrain]

Did someone here try https://github.com/webpack-contrib/thread-loader in combination with ts-loader or does that make no difference?

[VuBui83]

I did, still crashed with these loaders

 module: {
        rules: [
            {
                test: /\.tsx?$/,
                use: [
                    { loader: 'cache-loader' },
                    {
                        loader: 'thread-loader',
                        options: {
                            // there should be 1 cpu for the fork-ts-checker-webpack-plugin
                            workers: require('os').cpus().length - 1,
                        },
                    },
                    {
                      loader: 'ts-loader',
                      options: {
                        happyPackMode: true,
                        transpileOnly: true
                      }
                    }
                  ]
             }
        ]
    }

Serverless: Bundling with Webpack...

<--- Last few GCs --->

[4920:0x391c8b0]   175834 ms: Mark-sweep 1155.1 (1502.1) -> 1155.0 (1503.1) MB, 695.4 / 0.0 ms  allocation failure GC in old space requested
[4920:0x391c8b0]   176489 ms: Mark-sweep 1155.0 (1503.1) -> 1154.9 (1450.6) MB, 655.1 / 0.0 ms  last resort GC in old space requested
[4920:0x391c8b0]   177153 ms: Mark-sweep 1154.9 (1450.6) -> 1154.9 (1437.6) MB, 663.7 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1f3b69da5501 <JSObject>
    1: /* anonymous */ [/mnt/c/AccountServices/node_modules/webpack/node_modules/webpack-sources/node_modules/source-map/lib/source-node.js:~342] [pc=0x3dab59846f57](this=0x3597b2d8c389 <JSGlobal Object>,chunk=0x2c2f5e74b4c1 <String[60]\:     xxx('_applySerializers: excludeFields', excludeFields);\n>,original=0x3added1b95b9 <Object map = 0x17ffa9905dd1>)
    2: SourceNode_walk [/mnt/c...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x11f155c [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
 6: 0xdf2313 [node]
 7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
 8: 0x3dab58f842fd
Aborted (core dumped)

[grumpy-programmer]

Hi everyone,
I spend couple of hours trying to debug this problem. And my conclusion is memory leak in webpack or something else below webpack. I tried with ts-loader, awesome-typescript-loader, thread-loader, cache-loader, happypack, fork-ts-checker-webpack-plugin in any combination.

I wrote test webpack-test.js to debug only webpack, and try in every possible way to lost references to preform GC. Screenshot from node-gc-viewer below.

I see possible workaround, but it's nasty... Invoke child node process (but please not like fork-ts-checker-webpack-plugin) to compile ts with webpack or ... fix webpack 😄

My setup:

tsconfig.json

{
  "compilerOptions": {
    "sourceMap": true,
    "target": "es6",
    "types": [
      "node"
    ],
    "moduleResolution": "node"
  }
}

webpack.config.js

module.exports = {
    entry: {},
    target: 'node',
    output: {
        libraryTarget: 'commonjs2',
        path: '<absolute patch to project>/.webpack',
        filename: '[name].js',
    },
    module: {
        rules: [
            {
                test: '\.ts(x?)$',
                use: [
                    {
                        loader: 'ts-loader'
                    }
                ],
            }
        ]
    },
    externals: {
        'crypto': true,
        'aws-sdk': true
    },
    resolve: {
        extensions: [
            '.js',
            '.jsx',
            '.json',
            '.ts',
            '.tsx'
        ],
        alias: {
            'handlebars': 'handlebars/dist/handlebars.js'
        }
    }
};

webpack-test.js

const webpackConfig = require('./webpack.config');

const entries = {
    'src/handler/AuthorizerHandler': './src/handler/AuthorizerHandler.ts',
    'src/handler/yyy/LoginHandler': './src/handler/yyy/LoginHandler.ts',
    'src/handler/xxx/GetAllHandler': './src/handler/xxx/GetAllHandler.ts',
    'src/handler/xxx/GetOneHandler': './src/handler/xxx/GetOneHandler.ts',
    'src/handler/xxx/CreateHandler': './src/handler/xxx/CreateHandler.ts',
    'src/handler/xxx/UpdateHandler': './src/handler/xxx/UpdateHandler.ts',
    'src/handler/xxx/DeleteHandler': './src/handler/xxx/DeleteHandler.ts',
    'src/handler/zzz/GetAllHandler': './src/handler/zzz/GetAllHandler.ts',
    'src/handler/zzz/GetOneHandler': './src/handler/zzz/GetOneHandler.ts',
    'src/handler/zzz/CreateHandler': './src/handler/zzz/CreateHandler.ts',
    'src/handler/zzz/UpdateHandler': './src/handler/zzz/UpdateHandler.ts',
    'src/handler/zzz/DeleteHandler': './src/handler/zzz/DeleteHandler.ts',
    'src/handler/zzz/PublishHandler': './src/handler/zzz/PublishHandler.ts',
    // 'src/handler/qqq/GetAllHandler': './src/handler/qqq/GetAllHandler.ts',
    // 'src/handler/qqq/GetOneHandler': './src/handler/qqq/GetOneHandler.ts',
    // 'src/handler/qqq/CreateHandler': './src/handler/qqq/CreateHandler.ts',
    // 'src/handler/qqq/UpdateHandler': './src/handler/qqq/UpdateHandler.ts',
    // 'src/handler/qqq/DeleteHandler': './src/handler/qqq/DeleteHandler.ts',
    // 'src/handler/aaa/GetAllHandler': './src/handler/aaa/GetAllHandler.ts',
    // 'src/handler/aaa/GetOneHandler': './src/handler/aaa/GetOneHandler.ts',
    // 'src/handler/aaa/CreateHandler': './src/handler/aaa/CreateHandler.ts',
    // 'src/handler/aaa/UpdateHandler': './src/handler/aaa/UpdateHandler.ts',
    // 'src/handler/aaa/DeleteHandler': './src/handler/aaa/DeleteHandler.ts'
};


const queue = [];

for (const key of Object.keys(entries)) {
    const value = entries[key];

    queue.push([key, value]);
}

let working = false;

let webpack = null;
let compiler = null;
let config = null;
const configJson = JSON.stringify(webpackConfig);

const interval = setInterval(intervalF, 1000);

function intervalF() {
    if (working) {
        return;
    }

    if (queue.length === 0) {
        console.log('DONE!');
        clearInterval(interval);
        return;
    }

    working = true;

    const [key, value] = queue.pop();


    config = null;
    webpack = null;
    compiler = null;

    config = JSON.parse(configJson);

    config.module.rules[0].test = new RegExp(config.module.rules[0].test);

    config.entry[key] = value;

    console.log(config);

    webpack = require('webpack');
    console.log('COMPILING', key, value);

    compiler = webpack(config, (err, stats) => {
        working = false;
        console.log('COMPILED', key, value);
    });
}

package.json

"devDependencies": {
    "@types/aws-lambda": "0.0.22",
    "@types/handlebars": "^4.0.36",
    "@types/jsonwebtoken": "^7.2.5",
    "@types/node": "^8.0.57",
    "@types/uuid": "^3.4.3",
    "awesome-typescript-loader": "^3.4.1",
    "aws-sdk": "^2.176.0",
    "cache-loader": "^1.2.0",
    "fork-ts-checker-webpack-plugin": "^0.3.0",
    "happypack": "^4.0.1",
    "serverless": "^1.25.0",
    "serverless-domain-manager": "^2.0.2",
    "serverless-dynamodb-local": "^0.2.26",
    "serverless-kms-secrets": "^1.0.2",
    "serverless-offline": "^3.16.0",
    "serverless-webpack": "^4.0.0",
    "thread-loader": "^1.1.2",
    "ts-loader": "^2.3.7",
    "typescript": "^2.5.2",
    "webpack": "^3.6.0"
  },
"dependencies": {
    "handlebars": "^4.0.11",
    "jsonwebtoken": "^8.1.0",
    "uuid": "^3.1.0"
  }

[Birowsky]

Much appreciated effort, Grumpy! When somebody fixes this, instead of all my lambdas weighing 30MB each, most of them will go below 1MB. So trust me, I appreciate efforts like this.

[dashmug]

Most feasible workaround for this right now is simply to turn off individual packaging.

[dashmug]

I think changing the title to "JavaScript heap out of memory when _packaging_ many functions" makes more sense now that it has been isolated to just the packaging process and not the deployment process.

[dashmug]

Has anyone tried if webpack v4.0.0 can fix this?

[TimonVS]

@dashmug I tried the RC two days ago and it didn’t fix the problem for me. But I’d like to hear other people’s experience.

[janicduplessis]

@dashmug Webpack 4.0.0 doesn't fix it for me. My project uses babel and the issue seems to happen only when enabling source maps (devtool: 'source-map').

[HyperBrain]

While preparing version 5.0.0, I recognized that we use ts-node to enable support for TS webpack configuration files. Currently ts-node is referenced as ^3.2.0 in the package.json of the plugin, but I saw that there is already a ^5.0.0 version of ts-node available. Does anybody know if I can upgrade it in the plugin's package.json without breaking anyone's projects or should I keep it at the current version?

[dashmug]

YMMV, but I'm currently testing what's in this article about using cache-loader and thread-loader.

Initial results are fine so far though I have only tested on my MacBook with 16GB of RAM and will still have to test on our CI which only has 3GB RAM :-).

Working config so far...

'use strict'

const os = require('os')
const path = require('path')
const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin')
const slsw = require('serverless-webpack')
const webpack = require('webpack')
const nodeExternals = require('webpack-node-externals')


module.exports = {
    context: __dirname,
    entry: slsw.lib.entries,
    target: 'node',
    output: {
        libraryTarget: 'commonjs2',
        path: path.join(__dirname, 'build'),
        filename: '[name].js',
    },
    module: {
        rules: [{
            test: /\.ts$/,
            use: [{
                loader: 'cache-loader',
            }, {
                loader: 'thread-loader',
                options: {
                    // There should be 1 cpu for the
                    // fork-ts-checker-webpack-plugin
                    workers: os.cpus().length - 1,
                },
            }, {
                loader: 'ts-loader',
                options: {
                    // IMPORTANT! use happyPackMode mode to speed-up
                    // compilation and reduce errors reported to webpack
                    happyPackMode: true,
                },
            }],
        }],
    },
    externals: [nodeExternals()],
    resolve: {
        extensions: ['.ts', '.js'],
    },
    devtool: 'source-map',
    plugins: [
        new ForkTsCheckerWebpackPlugin({ checkSyntacticErrors: true }),
    ],
}

[grumpy-programmer]

@dashmug as far as I remember fork-ts-checker-webpack-plugin compile typescript to javascript fast and spawn thread to check errors. Each of the spawned check threads runs with default 2048 MB memory limit and starts immediately without any queue mechanism. So in the worst case memory usage is lambda count * memory limit.

[dashmug]

@grumpy-programmer It's a workaround that worked on my local but didn't work on our CI environment (AWS CodeBuild using 3GB). I had to bump up the RAM to 7GB for it to work.

[HyperBrain]

@grumpy-programmer
I just inspected the code of https://github.com/Realytics/fork-ts-checker-webpack-plugin to see if there can be any changes done to restrict the amount of processes spawned.

What I've found there is const division = parseInt(process.env.WORK_DIVISION, 10); which seems to control the amount of worker processes spawned for the plugin.

Can anyone of you try to set process.env.WORK_DIVISION to a smaller value (maybe 2) and check if the memory consumption still explodes with bigger services? Additionally I found that it uses process.env.MEMORY_LIMIT to set the Node VM heap size per worker, which could be an additional screw to get it under control.

[Dashue]

Bought a new laptop with I8 quad core and 16 gb of ram and this issue is happening more often than on my I5 duo with 8 gb of ram??

[akleiber]

I have the same problem but without TS. Disabling sourcemaps helps, but can't be a solution. :(
Any hints how to optimize memory consumtion for sourcemap creation?

[HyperBrain]

@akleiber Is this a quite big project where it happens?
You could try to set devtool: "nosources-source-map" to prevent embedding the whole sources into the source maps but only the line numbers. In most cases this is fully sufficient and might reduce the memory consumption.

However, there are some issues in the webpack repository about the OOM issues in combination of source maps. According to this recent comment https://github.com/webpack/webpack/issues/4727#issuecomment-373692350 it should be solved in the latest source-map module and should be used with the latest webpack version.

What is the webpack version you use?

[aequasi]

cache-loader and thread-loader significantly helped for me

[Maxwell2022]

Thanks @Birowsky

transpileOnly: true

Definitely something wrong with ts-loader, setting the transpileOnly option to true we went from 9 minutes deployment time to 2 minutes and got rid of the CALL_AND_RETRY_LAST error.
Our setup:

• webpack v4
• 5 Lambda functions
• package.individually = true

[markl-vesper]

I've started to hit extremely long times for webpack to complete and also the javascript heap memory.

My stack has 14 functions

I've upgraded my t2 instance for now but will look at adjusting the heap as I saw above but I'm really concerned about how long it takes to perform the webpack (30 mins at minimum)

I've upgraded to [email protected] & [email protected]

My webpack.config.js is below

const StatsWriterPlugin = require("webpack-stats-plugin").StatsWriterPlugin;
const webpack = require("webpack");
const BabiliPlugin = require("babili-webpack-plugin");

module.exports = {
    target: "node",
    externals: [/aws-sdk/],
    plugins: [
        new StatsWriterPlugin(),
        new BabiliPlugin(),
    ],
    module: {
        rules: [
            {
                test: /src\/.*\.js$/,
                exclude: [
                    /node_modules/,
                    /test/],
                loader: "babel-loader",
                options: {
                    presets: [
                        [
                            "env",
                            {
                                target: { node: 6.10 }, // Node version on AWS Lambda
                                useBuiltIns: true,
                                modules: false,
                                loose: true,
                            },
                        ],
                        "es2015",
                        "stage-0",
                    ],
                    plugins: ["transform-runtime" ]
                },
            },
        ],
    }
};

and my serverless package section looks like

package:
 individually:                   true
 include:
   - src/**/*

 exclude:
   - ./*
   - test/**
   - node_modules/**

I'm no expert in node or webpack so any tips or ideas on how to increase the performance of the packaging would be greatly appreciated. I'm getting around it for now by deploying functions individually but if I need to deploy the whole stack I'm kissing a lot of time goodbye.

Appreciated
Thanks

[markl-vesper]

Hi All

I ran the serverless package command while increasing the heap. It completed OK

output below for 1st module

$ node --max-old-space-size=8192 /home/ec2-user/.nvm/versions/node/v7.10.1/lib/node_modules/serverless/bin/serverless package
Serverless: Bundling with webpack...
Child
    Time: 519248ms
                               Asset       Size  Chunks                    Chunk Names
    src/marlin/captureDeviceAlarm.js    1.52 MB       0  [emitted]  [big]  src/marlin/captureDeviceAlarm.js
                          stats.json  107 bytes          [emitted]
      [26] ../base/src/log.js 432 bytes {0} [built]
      [51] external "aws-sdk" 42 bytes {0} [not cacheable]
      [66] ../base/node_modules/babel-runtime/core-js/json/stringify.js 95 bytes {0} [built]
     [104] ../base/src/helper.js 3.53 kB {0} [built]
     [121] ../base/node_modules/babel-runtime/regenerator/index.js 49 bytes {0} [built]
     [122] ../base/node_modules/babel-runtime/helpers/asyncToGenerator.js 906 bytes {0} [built]
     [338] ./src/marlin/captureDeviceAlarm.js 394 bytes {0} [built]
     [339] ../base/node_modules/bole/bole.js 5.11 kB {0} [built]
     [344] ../base/node_modules/bole-console/lib/index.js 3.23 kB {0} [built]
     [346] ../base/src/device.js 29.1 kB {0} [built]
     [484] ../base/src/user.js 6.55 kB {0} [built]
     [628] ../base/src/sierra.js 5.69 kB {0} [built]
     [629] ../base/src/notifications.js 10.4 kB {0} [built]
     [634] ../base/database/marlin-config.js 6.52 kB {0} [built]
     [635] ../base/node_modules/axios/index.js 40 bytes {0} [built]
        + 645 hidden modules

Do I need to be concerned about the +645 hidden modules? it that why its taking so long perhaps?

[BobbieBarker]

fwiw I implemented the changes that @dashmug mentioned in his post and it looks like my current project is back in business.

[markl-vesper]

I'm finding much better performance by increasing the heap by using

node --max-old-space-size=4096 node_modules/serverless/bin/serverless package

I only ever do a full deploy with increased heap when a new function is created otherwise I now just use sls deploy function when updating a single function

I'd still love to know more about my question re +645 hidden modules and if that indicates a setup or config issue or is normal??

[rdsedmundo]

I don't even understand why this is an issue here. I have 7 functions, but all of them are very small. The overall size of the project is a very small project, I run projects much bigger with webpack with the same loaders (and more stuff) and almost never fall on this heap errors (the last I remember was back on webpack 1), so I don't think the solution here should be focused on changing the loaders configurations, but on the way that serverless-webpack is executing webpack.

[DWboutin]

An update: it works when I set transpileOnly: true for ts-loader.

It did it for me :D Ty

[andrewrothman]

I have a serverless project with a lot of functions 75+. I've been trying many of the answers in this thread, with no luck. I am using a new i7/16GB MacBook Pro which started spinning its fans and needed a restart twice from this issue. My build is not passing through CI and I do not want to go back to https://github.com/prisma/serverless-plugin-typescript because it is using an outdated version of typescript and appears to be looking for maintainers. Does anybody have any solutions to this problem?

I have tried:

• setting transpileOnly: true on ts-loader
• using fork-ts-checker-webpack-plugin (this caused the fans/slowness requiring restarts)
• setting serverless package.individually: true

Most of the time I get the heap out of memory error. I very much appreciate the hard work that has gone into this open source project and thank all the contributors/maintainers, but this seems like a serious issue for using this plugin in production. Is there anything else I should try?

Serverless: Bundling with Webpack...

<--- Last few GCs --->

[1588:0x102801c00]    29214 ms: Mark-sweep 1399.7 (1455.6) -> 1393.8 (1440.1) MB, 985.5 / 0.0 ms  (+ 0.2 ms in 210 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1095 ms) (average mu = 0.133, current mu = 0.100) allocat[1588:0x102801c00]    30184 ms: Mark-sweep 1393.8 (1440.1) -> 1393.8 (1440.1) MB, 968.8 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 969 ms) (average mu = 0.067, current mu = 0.001) allocation

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1ba3da45be3d]
Security context: 0x0104dda1e6e1 <JSObject>
    1: bindWorker(aka bindWorker) [0x104954f2fd1] [/Users/andrew/var/api/node_modules/typescript/lib/typescript.js:~28846] [pc=0x1ba3daa40b88](this=0x01049c2826f1 <undefined>,node=0x0104678e4b01 <IdentifierObject map = 0x1047aa7d7d1>)
    2: bind(aka bind) [0x104954f2ed1] [/Users/andrew/var/api/node_modules/typescript/lib/typescript.js:~28756] [pc=0x1ba3dad3...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003b125 node::Abort() [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 2: 0x10003b32f node::OnFatalError(char const*, char const*) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 3: 0x1001a8e85 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 4: 0x1005742a2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 5: 0x100576d75 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 6: 0x100572c1f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 7: 0x100570df4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 8: 0x10057d6b8 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
 9: 0x10057d70f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
10: 0x10055fac5 v8::internal::Factory::NewLoadHandler(int) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
11: 0x1005e62cd v8::internal::LoadHandler::LoadFromPrototype(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Smi>, v8::internal::MaybeHandle<v8::internal::Object>, v8::internal::MaybeHandle<v8::internal::Object>) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
12: 0x1005ea737 v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
13: 0x1005ea392 v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
14: 0x1005f0fc1 v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/andrew/.nvm/versions/node/v10.15.0/bin/node]
15: 0x1ba3da45be3d 
zsh: abort      SLS_DEBUG=* npx sls deploy --stage blue

[dashmug]

@andrewrothman The workaround that worked for my project is by turning off package.individually: true. I get bigger deployment bundles but at least everything works.

I still would want to package functions individually to get more optimized bundles but it is not my priority at the moment.

One thing I would try is to use babel (and babel-loader) for transpiling Typescript instead of awesome-typescript-loader or ts-loader. If you don't have any other option, maybe you can try this out.

[rdsedmundo]

I endorse @dashmug's answer here. package.individually not set helps with this problem. When it's true what I realized is that the plugin will run webpack multiple times, for each handler you have. This easily bomb the memory out as you can imagine.

[Birowsky]

Could serializing the jobs be an intermediate workaround?

[cj179johnCheckit]

Any updates on this particular issue. We have to separate out the typescript compilation and only doing package in webpack to bypass the problem.

[akleiber]

I don't even understand why this is an issue here. I have 7 functions, but all of them are very small. The overall size of the project is a very small project, I run projects much bigger with webpack with the same loaders (and more stuff) and almost never fall on this heap errors (the last I remember was back on webpack 1), so I don't think the solution here should be focused on changing the loaders configurations, but on the way that serverless-webpack is executing webpack.

Yes that. @HyperBrain is it necessary that webpack is run in parallel for each function? Is this behaviour changeable? If yes would it be okay for you if we'd provide a PR? Maybe an option that allows to configure if webpack is run in parallel or sequentially.

[HyperBrain]

The plugin utilizes webpack's multi-compile mode, which performs much
better optimization-wise, but webpack itself is invoked only once and does
the compile internally!

As far as I know, the behavior can be configured in the webpack.conf, as it
is a webpack specific thing. And I know that there are issues with the
various ts loaders which behave incorrectly. We should check, if the issues
are still open (e.g. for ts-loader) or fixed.

Invoking webpack sequentially would IMO extend compile times extremely.

On Fri, Apr 26, 2019 at 8:55 AM Andreas Kleiber [e-mail]
wrote:

I don't even understand why this is an issue here. I have 7 functions, but
all of them are very small. The overall size of the project is a very small
project, I run projects much bigger with webpack with the same loaders (and
more stuff) and almost never fall on this heap errors (the last I remember
was back on webpack 1), so I don't think the solution here should be
focused on changing the loaders configurations, but on the way that
serverless-webpack is executing webpack.

Yes that. @HyperBrain https://github.com/HyperBrain is it necessary
that webpack is run in parallel for each function? Is this behaviour
changeable? If yes would it be okay for you if we'd provide a PR? Maybe an
option that allows to configure if webpack is run in parallel or
sequentially.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/serverless-heaven/serverless-webpack/issues/299#issuecomment-486948019,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKEZXXTJNYQP6J25MDOOE3PSKRN7ANCNFSM4EHSFFPA
.

[akleiber]

With multi-compile mode you mean that serverless-webpack "multiplies" the webpack config for each function - like so: https://webpack.js.org/configuration/configuration-types/#exporting-multiple-configurations

I could not find anything else that sounds like multi-compile mode.
Can you point me to the right line - I guess something here is responsible https://github.com/serverless-heaven/serverless-webpack/blob/master/lib/packageModules.js

[mogusbi]

I'm using a combination of fork-ts-checker-webpack-plugin, cache-loader and thread-loader to compile 11 typescript lambda functions but I'm getting this error;

(node:145) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit

I'm now stuck because I can no longer deploy any of my functions

Here's my Webpack config

const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin');
const {cpus} = require('os');
const {join} = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');

module.exports = {
  context: __dirname,
  entry: slsw.lib.entries,
  externals: [
    nodeExternals()
  ],
  mode: slsw.lib.webpack.isLocal ? 'development': 'production',
  module: {
    rules: [
      {
        test: /\.ts(x?)$/,
        use: [
          {
            loader: 'cache-loader'
          },
          {
            loader: 'thread-loader',
            options: {
              workers: cpus().length - 1
            }
          },
          {
            loader: 'ts-loader',
            options: {
              happyPackMode: true
            }
          }
        ]
      }
    ]
  },
  output: {
    filename: '[name].js',
    libraryTarget: 'commonjs',
    path: join(__dirname, '.webpack')
  },
  plugins: [
    new ForkTsCheckerWebpackPlugin({
      checkSyntacticErrors: true
    })
  ],
  resolve: {
    extensions: [
      '.js',
      '.jsx',
      '.json',
      '.ts',
      '.tsx'
    ]
  },
  target: 'node'
};

[Plysepter]

What version of fork-ts-checker-webpack-plugin are you using? V 1.1.1 includes a fix for a regression when working with some other plugins: https://github.com/Realytics/fork-ts-checker-webpack-plugin/releases/tag/v1.1.1 and this may resolve your issue. I just encountered the same error with my webpack configuration and I was able to resolve it by updating my dependencies. I do not believe this is to do with serverless-webpack directly.

[mogusbi]

Updating to anything above version 0.5.2 leads to this error

Cannot read property 'options' of undefined

[aprat84]

In my case, I've got around 30 lambdas, and I have two problems:

• If I turn off individual packaging, then my package exceeds Lambda's ~250MB code limit
• If I turn it on, I get the error discuted in this issue (JS heap out of memory)

The only way I'm able to use individually packaging is turning on transpileOnly in ts-loader.

If I use fork-ts-checker-webpack-plugin, my machine dies as the plugin spawns like 30 workers in parallel and it eats my 16GB RAM/swap in few seconds...

IMHO the only solution is to compile all functions in series, one after the other, by default or with setting.
How's that going? Any ETA?

Thanks!

[alexsavio]

I have the same issue but not with webpack. This seems to be a Serverless Framework problem. I have 10 lambda functions in Python without dependencies, the dependencies are in 4 layers also in the same setup. If I turn off the plugins I have (python-requirements), I still get the same problem. In Linux the process gets killed half the way through after eating up all my RAM, in Windows defective .zip files are deployed without any warning.

[thewunder]

I had a similar issue on my linux build server. Turned out that installing libzip4 fixed the issue. Serverless uses an archive package that uses another package that falls back to a node implementation of zip if libzip isn't installed.

[weull]

Ran into the same situation in our project where we are using serverless-webpack to individually package 28 lambdas with typescript. We finally hit the same error - Javascript heap out of memory - that's already been reported.

Tried the PR from @asprouse - https://github.com/serverless-heaven/serverless-webpack/pull/517 - and can confirm that it fixed the issue for us. Any ETA on when this PR might be reviewed and merged?

[pettyalex]

This is still affecting my team, and https://github.com/serverless-heaven/serverless-webpack/pull/517 would fix it for us.

We've reverted back to not packaging individually because of excessive memory consumption from webpack's multiple compiler.

[wilson-burea]

Hi, Im having this same issue. I'm not using serverless webpack plugin, webpack file, neither typescript. When I deploy the service I got a JavaScript heap out of memory. I had remove package individually and it works, but I want to use that feature again. It will be good if anyone could solve this problem.

[RobBSage]

We were able to get round this issue setting a Node env variable on our cloud build server, and locally.

export NODE_OPTIONS=--max_old_space_size=8192

https://github.com/serverless/serverless/issues/6503

[592da]

any solution ?

[odykyi]

An update: it works when I set transpileOnly: true for ts-loader.

[soda0289]

I have implemented a fix (#570) that uses multiple process to compile functions when package individually is on. This guarantees that memory is cleaned up after every compile, since we kill the process, and can compile multiple functions at once. It improves performance by quite a bit in the testing I have done.

[mordka]

I have the same issue in a monorepo with 10+ services. Not using package: individually: true

Serverless:  
Serverless: Bundling with Webpack...

<--- Last few GCs --->

[3553371:0x3f36f40]    46484 ms: Mark-sweep 1387.3 (1424.8) -> 1386.9 (1425.3) MB, 488.2 / 0.0 ms  (average mu = 0.157, current mu = 0.009) allocation failure scavenge might not succeed
[3553371:0x3f36f40]    47368 ms: Mark-sweep 1387.6 (1425.3) -> 1387.1 (1425.8) MB, 880.4 / 0.0 ms  (average mu = 0.068, current mu = 0.004) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x34a9b6c5be1d]
Security context: 0x0d6352f9e6e9 <JSObject>
    1: declareSymbol(aka declareSymbol) [0x256504191261] [/home/user/API/node_modules/typescript/lib/typescript.js:~31012] [pc=0x34a9b7609231](this=0x2544629826f1 <undefined>,symbolTable=0x1f866b8c2b51 <Map map = 0x66207504359>,parent=0x2544629826f1 <undefined>,node=0x3d42a8b973e1 <NodeObject map = 0x7f62c5b6189>,includes=0,excludes=788968,isR...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8fa090 node::Abort() [node]
 2: 0x8fa0dc  [node]
 3: 0xb0052e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb00764 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xef4c72  [node]
 6: 0xef4d78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf00e52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf01784 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf043f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xecd874 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x116d9fe v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x34a9b6c5be1d 
Aborted

[corollari]

I ran into this problem as well, here's my experience with several of the alternatives discussed in this thread:

• Adding additional memory to the process worked for a while, but, when the complexity of my system grew, the system reached a point where I had to provision more than 12GB for the process not to trigger any faults (and I'd have had to keep increasing it whenever new functions were added).
• Applying #517 would let us compile more functions than without it but eventually we'd also get a fault. The number of functions we managed to compile depended on the memory allocated to the process, so eventually this would lead to the same problem of having to continually increase the memory forever.
• Applying #570 would solve our problem but would break serverless-offline, I managed to fix that for our system and submitted the fix in a PR (soda0289/serverless-webpack#2). Note that to get this to work we had to stop using webpack-node-externals on our webpack config, as that causes the config to be serialized with an empty array for externals which causes lambdas to compile seemingly fine but then fail when deployed.

Hope this is useful to someone and they don't have to spend a whole day on it like I did :smile:

[j0k3r]

Can someone confirme this has been improved or fixed by 5.4.0?

[miguel-a-calles-mba]

I have not seen improvements with 5.4.0. I was helping out a friend on his project and I had to rollback to 5.3.5 to see some stability with the out-of-memory issue.

I also had to roll back to an older webpack (4.46.0).

[coyoteecd]

@j0k3r I can confirm that the concurrency setting added in #681 works as intended after update to 5.4.0 (i.e. limits the number of concurrent compiles in the CI system thus effectively limiting the amount of necessary memory and avoiding the out-of-memory errors).

Note that in my case I run it with a value of 3 in the CI build; I have it configured in serverless.yml as follows:

custom:
  webpack
    [other settings]
    concurrency: ${opt:compile-concurrency, 6}

In CI, I deploy as follows:
serverless deploy --compile-concurrency 3

[ADrejta]

@j0k3r I can also confirm that setting the concurrency setting like described in #681 does do the trick in update 5.4.0

[imhazige]

I have not seen improvements with 5.4.0. I was helping out a friend on his project and I had to rollback to 5.3.5 to see some stability with the out-of-memory issue.

I also had to roll back to an older webpack (4.46.0).

Switch webpack back from 5 to 4 solve this problem for me.

[vicary]

I am the author of #681, my project is on-and-off dealing with 200 lambda functions.

Recent updates in minor versions introduced this again, subsequent builds in the same process does linear increases in bundle time. This is further confirmed when tested with thread-loader, the timer increases individually in each thread. Upgrading webpack from 5.11 to 5.37.1 slows down the increments, but, still, it is surely increasing gradually from 70s to 700s+ at the 50th entry.

Using the serverless-layers plugin and excluding with webpack-node-externals without using modulesFromFile options stops the build times of subsequent entries time from increasing.

My educated guess is that packages in node_modules contains side effects that webpack has no way to cleanup after bundling. Try to avoid having webpack to dip its toes into node_modules when Lambda Function Layers are available, otherwise pushing for https://github.com/serverless-heaven/serverless-webpack/pull/570 and helps rebasing maybe your only choice.

EDIT: Also make sure you read https://github.com/webpack/webpack/issues/6389 if you are thinking of downgrading to webpack 4.

[j0k3r]

Can someone confirm this has been improved or fixed by 5.5.1?

[vicary]

858 surely looks interesting, I'll give it a try next week.

[bxjw]

858 seems to have resolved it for us.

[vicary]

Yes, my team has been trying deployments in the last weeks. I am fairly confident that the problem is at least minimized to unnoticeable even for 200+ lambdas.

[jpascoe]

Adding --compile-concurrency 3 fixed problem for me

[daveykane]

@j0k3r I'm on 5.5.1 and still have this issue unfortunately

[jsefiani]

I'm experiencing the same issue with the latest versions of both serverless-webpack (5.5.1) and webpack (5.50.0). Really annoying.

PS I'm only using 1 function (NestJS API) and I constantly run into memory issues.

[janicduplessis]

This fix will only improve memory usage when packaging many functions, anything under ~8 functions probably won't make a difference since they will be packaged concurrently.

[omry-hay]

We still get those with version 5.5.1.