Serverless-webpack: Error: spawn E2BIG - ZIP bundling fails due to node_modules

[chriszirkel]

This is a Bug Report

Description

I originally created this issue in the serverless repo, but they suggested I should move it to serverless-webpack as it is related to a dependency of serverless-webpack. Original issue: https://github.com/serverless/serverless/issues/9415

---

I'm using the aws-nodejs-typescript template. When I deploy the application, it crashes with an spawn E2BIG error:

 Error ---------------------------------------------------

  Error: spawn E2BIG
      at ChildProcess.spawn (node:internal/child_process:403:11)
      at Object.spawn (node:child_process:573:9)
      at /Users/chris/Projects/Nebula/api/node_modules/bestzip/lib/bestzip.js:75:29
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:1802:20
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:248:13
      at wrapper (/Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:268:20)
      at iterateeCallback (/Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:421:28)
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:321:20
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:245:17
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:1792:24
      at /Users/chris/Projects/Nebula/api/node_modules/bestzip/lib/bestzip.js:49:7
      at processTicksAndRejections (node:internal/process/task_queues:75:11)

I found out that this error is related to building the .ZIP file which will be uploaded to S3. The spawn command will be invoked by bestzip (/Users/chris/Projects/Nebula/api/node_modules/bestzip/lib/bestzip.js:75:29) with too many arguments (more than 4.500 individual .js files). I added a consle.logright before the spawn and get the following output:

Serverless: Packing external modules: source-map-support@^0.5.19, neo4j-driver@^4.2.3, axios@^0.21.1, graphql@^15.5.0, graphql-tag@^2.12.4
command zip [
  '--quiet',
  '--recurse-paths',
  '../nebula-api.zip',
  'node_modules/.package-lock.json',
  'node_modules/@babel/runtime/helpers/applyDecoratedDescriptor.js',
  'node_modules/@babel/runtime/helpers/arrayLikeToArray.js',
  'node_modules/@babel/runtime/helpers/arrayWithHoles.js',
  'node_modules/@babel/runtime/helpers/arrayWithoutHoles.js',
  'node_modules/@babel/runtime/helpers/assertThisInitialized.js',
  'node_modules/@babel/runtime/helpers/AsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/asyncGeneratorDelegate.js',
  'node_modules/@babel/runtime/helpers/asyncIterator.js',
  'node_modules/@babel/runtime/helpers/asyncToGenerator.js',
  'node_modules/@babel/runtime/helpers/awaitAsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/AwaitValue.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorGet.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorSet.js',
  'node_modules/@babel/runtime/helpers/classCallCheck.js',
  'node_modules/@babel/runtime/helpers/classCheckPrivateStaticAccess.js',
  'node_modules/@babel/runtime/helpers/classCheckPrivateStaticFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/classExtractFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/classNameTDZError.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldGet.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldLooseBase.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldLooseKey.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldSet.js',
  'node_modules/@babel/runtime/helpers/classPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/classPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldSpecGet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldSpecSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/construct.js',
  'node_modules/@babel/runtime/helpers/createClass.js',
  'node_modules/@babel/runtime/helpers/createForOfIteratorHelper.js',
  'node_modules/@babel/runtime/helpers/createForOfIteratorHelperLoose.js',
  'node_modules/@babel/runtime/helpers/createSuper.js',
  'node_modules/@babel/runtime/helpers/decorate.js',
  'node_modules/@babel/runtime/helpers/defaults.js',
  'node_modules/@babel/runtime/helpers/defineEnumerableProperties.js',
  'node_modules/@babel/runtime/helpers/defineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/applyDecoratedDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/arrayLikeToArray.js',
  'node_modules/@babel/runtime/helpers/esm/arrayWithHoles.js',
  'node_modules/@babel/runtime/helpers/esm/arrayWithoutHoles.js',
  'node_modules/@babel/runtime/helpers/esm/assertThisInitialized.js',
  'node_modules/@babel/runtime/helpers/esm/AsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/asyncGeneratorDelegate.js',
  'node_modules/@babel/runtime/helpers/esm/asyncIterator.js',
  'node_modules/@babel/runtime/helpers/esm/asyncToGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/awaitAsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/AwaitValue.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorGet.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorSet.js',
  'node_modules/@babel/runtime/helpers/esm/classCallCheck.js',
  'node_modules/@babel/runtime/helpers/esm/classCheckPrivateStaticAccess.js',
  'node_modules/@babel/runtime/helpers/esm/classCheckPrivateStaticFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/classExtractFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/classNameTDZError.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldGet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldLooseBase.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldLooseKey.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldSet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldSpecGet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldSpecSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/esm/construct.js',
  'node_modules/@babel/runtime/helpers/esm/createClass.js',
  'node_modules/@babel/runtime/helpers/esm/createForOfIteratorHelper.js',
  'node_modules/@babel/runtime/helpers/esm/createForOfIteratorHelperLoose.js',
  'node_modules/@babel/runtime/helpers/esm/createSuper.js',
  'node_modules/@babel/runtime/helpers/esm/decorate.js',
  'node_modules/@babel/runtime/helpers/esm/defaults.js',
  'node_modules/@babel/runtime/helpers/esm/defineEnumerableProperties.js',
  'node_modules/@babel/runtime/helpers/esm/defineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/extends.js',
  'node_modules/@babel/runtime/helpers/esm/get.js',
  'node_modules/@babel/runtime/helpers/esm/getPrototypeOf.js',
  'node_modules/@babel/runtime/helpers/esm/inherits.js',
  'node_modules/@babel/runtime/helpers/esm/inheritsLoose.js',
  'node_modules/@babel/runtime/helpers/esm/initializerDefineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/initializerWarningHelper.js',
  'node_modules/@babel/runtime/helpers/esm/instanceof.js',
  'node_modules/@babel/runtime/helpers/esm/interopRequireDefault.js',
  'node_modules/@babel/runtime/helpers/esm/interopRequireWildcard.js',
  'node_modules/@babel/runtime/helpers/esm/isNativeFunction.js',
  'node_modules/@babel/runtime/helpers/esm/isNativeReflectConstruct.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArray.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArrayLimit.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArrayLimitLoose.js',
  'node_modules/@babel/runtime/helpers/esm/jsx.js',
  **... 4541 more items**
]

Interestingly, this only happens when I include the npm package "neo4j-driver", which seems to have a lot of dependencies. However, I'm not sure if it is related to this package itself or I will run into this issue anyway when my application grows.

serverless.yml

import type { AWS } from "@serverless/typescript";

// import hello from "@functions/hello";
import resolver from "@functions/resolver";

const serverlessConfiguration: AWS = {
  service: "nebula-api",
  frameworkVersion: "2",
  custom: {
    webpack: {
      webpackConfig: "./webpack.config.js",
      includeModules: {
        forceExclude: "aws-sdk",
      },
    },
  },
  plugins: ["serverless-webpack"],
  provider: {
    name: "aws",
    runtime: "nodejs14.x",
    region: "eu-west-1",
    stage: "${opt:stage, 'dev'}",
    apiGateway: {
      minimumCompressionSize: 1024,
      shouldStartNameWithService: true,
    },
    environment: {
      AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1",
    },
    lambdaHashingVersion: "20201221",
  },
  // import the function via paths
  functions: {
    // hello,
    resolver,
  },
};

module.exports = serverlessConfiguration;

webpack.config.js

const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');
const TsconfigPathsPlugin = require('tsconfig-paths-webpack-plugin');

/*
This line is only required if you are specifying `TS_NODE_PROJECT` for whatever reason.
 */
// delete process.env.TS_NODE_PROJECT;

module.exports = {
  context: __dirname,
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  entry: slsw.lib.entries,
  devtool: slsw.lib.webpack.isLocal ? 'eval-cheap-module-source-map' : 'source-map',
  resolve: {
    extensions: ['.mjs', '.json', '.ts'],
    symlinks: false,
    cacheWithContext: false,
    plugins: [
      new TsconfigPathsPlugin({
        configFile: './tsconfig.paths.json',
      }),
    ],
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js',
  },
  optimization: {
    concatenateModules: false,
  },
  target: 'node',
  externals: [nodeExternals()],
  module: {
    rules: [
      // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
      {
        test: /\.(tsx?)$/,
        loader: 'ts-loader',
        exclude: [
          [
            path.resolve(__dirname, 'node_modules'),
            path.resolve(__dirname, '.serverless'),
            path.resolve(__dirname, '.webpack'),
          ],
        ],
        options: {
          transpileOnly: true,
          experimentalWatchApi: true,
        },
      },
    ],
  },
  plugins: [],
};

tsconfig.json

{
  "extends": "./tsconfig.paths.json",
  "compilerOptions": {
    "lib": ["ESNext"],
    "moduleResolution": "node",
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "removeComments": true,
    "allowJs": true,
    "sourceMap": true,
    "target": "ES2020",
    "outDir": "lib"
  },
  "include": ["src/**/*.ts", "serverless.ts"],
  "exclude": [
    "node_modules/**/*",
    ".serverless/**/*",
    ".webpack/**/*",
    "_warmup/**/*",
    ".vscode/**/*"
  ],
  "ts-node": {
    "require": ["tsconfig-paths/register"]
  }
}

NPX SLS DEPLOY output

Chris-Mac:api chris$ SLS_DEBUG=* npm run deploy

> [email protected] deploy
> npm run deploy:dev


> [email protected] deploy:dev
> npx sls deploy --stage dev

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command webpack
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Skipping variables resolution with old resolver (new resolver reported no more variables to resolve)
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
asset src/functions/resolver/handler.js 2.56 KiB [emitted] [minimized] (name: src/functions/resolver/handler) 1 related asset
runtime modules 937 bytes 4 modules
built modules 9.43 KiB [built]
  cacheable modules 9.23 KiB
    modules by path ./src/functions/resolver/*.ts 1.18 KiB
      ./src/functions/resolver/handler.ts 606 bytes [built] [code generated]
      ./src/functions/resolver/executePerspective.ts 605 bytes [built] [code generated]
    ./src/libs/graphqlApi.ts 709 bytes [built] [code generated]
    ./src/graphql/queries.ts 6.89 KiB [built] [code generated]
    ./src/aws-exports.js 478 bytes [built] [code generated]
  external "source-map-support/register" 42 bytes [built] [code generated]
  external "neo4j-driver" 42 bytes [built] [code generated]
  external "axios" 42 bytes [built] [code generated]
  external "graphql" 42 bytes [built] [code generated]
  external "graphql-tag" 42 bytes [built] [code generated]
webpack compiled successfully in 1251 ms
Serverless: Invoke webpack:package
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: source-map-support@^0.5.19, neo4j-driver@^4.2.3, axios@^0.21.1, graphql@^15.5.0, graphql-tag@^2.12.4
command zip [
  '--quiet',
  '--recurse-paths',
  '../nebula-api.zip',
  'node_modules/.package-lock.json',
  'node_modules/@babel/runtime/helpers/applyDecoratedDescriptor.js',
  'node_modules/@babel/runtime/helpers/arrayLikeToArray.js',
  'node_modules/@babel/runtime/helpers/arrayWithHoles.js',
  'node_modules/@babel/runtime/helpers/arrayWithoutHoles.js',
  'node_modules/@babel/runtime/helpers/assertThisInitialized.js',
  'node_modules/@babel/runtime/helpers/AsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/asyncGeneratorDelegate.js',
  'node_modules/@babel/runtime/helpers/asyncIterator.js',
  'node_modules/@babel/runtime/helpers/asyncToGenerator.js',
  'node_modules/@babel/runtime/helpers/awaitAsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/AwaitValue.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorGet.js',
  'node_modules/@babel/runtime/helpers/classApplyDescriptorSet.js',
  'node_modules/@babel/runtime/helpers/classCallCheck.js',
  'node_modules/@babel/runtime/helpers/classCheckPrivateStaticAccess.js',
  'node_modules/@babel/runtime/helpers/classCheckPrivateStaticFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/classExtractFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/classNameTDZError.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldGet.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldLooseBase.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldLooseKey.js',
  'node_modules/@babel/runtime/helpers/classPrivateFieldSet.js',
  'node_modules/@babel/runtime/helpers/classPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/classPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldSpecGet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateFieldSpecSet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/classStaticPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/construct.js',
  'node_modules/@babel/runtime/helpers/createClass.js',
  'node_modules/@babel/runtime/helpers/createForOfIteratorHelper.js',
  'node_modules/@babel/runtime/helpers/createForOfIteratorHelperLoose.js',
  'node_modules/@babel/runtime/helpers/createSuper.js',
  'node_modules/@babel/runtime/helpers/decorate.js',
  'node_modules/@babel/runtime/helpers/defaults.js',
  'node_modules/@babel/runtime/helpers/defineEnumerableProperties.js',
  'node_modules/@babel/runtime/helpers/defineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/applyDecoratedDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/arrayLikeToArray.js',
  'node_modules/@babel/runtime/helpers/esm/arrayWithHoles.js',
  'node_modules/@babel/runtime/helpers/esm/arrayWithoutHoles.js',
  'node_modules/@babel/runtime/helpers/esm/assertThisInitialized.js',
  'node_modules/@babel/runtime/helpers/esm/AsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/asyncGeneratorDelegate.js',
  'node_modules/@babel/runtime/helpers/esm/asyncIterator.js',
  'node_modules/@babel/runtime/helpers/esm/asyncToGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/awaitAsyncGenerator.js',
  'node_modules/@babel/runtime/helpers/esm/AwaitValue.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorGet.js',
  'node_modules/@babel/runtime/helpers/esm/classApplyDescriptorSet.js',
  'node_modules/@babel/runtime/helpers/esm/classCallCheck.js',
  'node_modules/@babel/runtime/helpers/esm/classCheckPrivateStaticAccess.js',
  'node_modules/@babel/runtime/helpers/esm/classCheckPrivateStaticFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/classExtractFieldDescriptor.js',
  'node_modules/@babel/runtime/helpers/esm/classNameTDZError.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldGet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldLooseBase.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldLooseKey.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateFieldSet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/esm/classPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldDestructureSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldSpecGet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateFieldSpecSet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateMethodGet.js',
  'node_modules/@babel/runtime/helpers/esm/classStaticPrivateMethodSet.js',
  'node_modules/@babel/runtime/helpers/esm/construct.js',
  'node_modules/@babel/runtime/helpers/esm/createClass.js',
  'node_modules/@babel/runtime/helpers/esm/createForOfIteratorHelper.js',
  'node_modules/@babel/runtime/helpers/esm/createForOfIteratorHelperLoose.js',
  'node_modules/@babel/runtime/helpers/esm/createSuper.js',
  'node_modules/@babel/runtime/helpers/esm/decorate.js',
  'node_modules/@babel/runtime/helpers/esm/defaults.js',
  'node_modules/@babel/runtime/helpers/esm/defineEnumerableProperties.js',
  'node_modules/@babel/runtime/helpers/esm/defineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/extends.js',
  'node_modules/@babel/runtime/helpers/esm/get.js',
  'node_modules/@babel/runtime/helpers/esm/getPrototypeOf.js',
  'node_modules/@babel/runtime/helpers/esm/inherits.js',
  'node_modules/@babel/runtime/helpers/esm/inheritsLoose.js',
  'node_modules/@babel/runtime/helpers/esm/initializerDefineProperty.js',
  'node_modules/@babel/runtime/helpers/esm/initializerWarningHelper.js',
  'node_modules/@babel/runtime/helpers/esm/instanceof.js',
  'node_modules/@babel/runtime/helpers/esm/interopRequireDefault.js',
  'node_modules/@babel/runtime/helpers/esm/interopRequireWildcard.js',
  'node_modules/@babel/runtime/helpers/esm/isNativeFunction.js',
  'node_modules/@babel/runtime/helpers/esm/isNativeReflectConstruct.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArray.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArrayLimit.js',
  'node_modules/@babel/runtime/helpers/esm/iterableToArrayLimitLoose.js',
  'node_modules/@babel/runtime/helpers/esm/jsx.js',
  ... 4541 more items
]

 Error ---------------------------------------------------

  Error: spawn E2BIG
      at ChildProcess.spawn (node:internal/child_process:403:11)
      at Object.spawn (node:child_process:573:9)
      at /Users/chris/Projects/Nebula/api/node_modules/bestzip/lib/bestzip.js:75:29
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:1802:20
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:248:13
      at wrapper (/Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:268:20)
      at iterateeCallback (/Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:421:28)
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:321:20
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:245:17
      at /Users/chris/Projects/Nebula/api/node_modules/async/dist/async.js:1792:24
      at /Users/chris/Projects/Nebula/api/node_modules/bestzip/lib/bestzip.js:49:7
      at processTicksAndRejections (node:internal/process/task_queues:75:11)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              15.3.0
     Framework Version:         2.39.0 (local)
     Plugin Version:            4.5.3
     SDK Version:               4.2.2
     Components Version:        3.9.1

undefined

/Users/chris/Projects/Nebula/api/node_modules/path2/posix/common.js:11
                throw new TypeError('Arguments to path.common must be strings');
        ^
TypeError: Arguments to path.common must be strings
    at normalize (/Users/chris/Projects/Nebula/api/node_modules/path2/posix/common.js:11:9)
    at module.exports (/Users/chris/Projects/Nebula/api/node_modules/path2/posix/common.js:21:9)
    at anonymizeStacktracePaths (/Users/chris/Projects/Nebula/api/node_modules/serverless/lib/utils/telemetry/anonymize-stacktrace-paths.js:7:26)
    at resolveErrorLocation (/Users/chris/Projects/Nebula/api/node_modules/serverless/lib/utils/telemetry/resolve-error-location.js:24:10)
    at module.exports (/Users/chris/Projects/Nebula/api/node_modules/serverless/lib/cli/handle-error.js:152:32)

Installed version

Framework Core: 2.39.0 (local)
Plugin: 4.5.3
SDK: 4.2.2
Components: 3.9.1

Additional Data

  "devDependencies": {
    "@serverless/typescript": "^2.23.0",
    "@types/aws-lambda": "^8.10.71",
    "@types/node": "^14.14.25",
    "json-schema-to-ts": "^1.5.0",
    "serverless": "^2.39.0",
    "serverless-webpack": "^5.3.5",
    "ts-loader": "^8.0.15",
    "ts-node": "^9.1.1",
    "tsconfig-paths": "^3.9.0",
    "tsconfig-paths-webpack-plugin": "^3.3.0",
    "typescript": "^4.1.3",
    "webpack": "^5.20.2",
    "webpack-node-externals": "^2.5.2"
  }

[j0k3r]

I knew we were going to have that issue soon or later. Let me sum up the problem.

We switched to bestzip in 5.4.0 in an optimization point of view. Bestzip check if the zip command exist and use it instead of an internal way of creating zip in nodejs. It can improve zip time a lot!

The downside, is that we broke two things:

• an existing option excludeRegex because we called the bestzip with the wildcard * to include every files
• and when using wildcard bestzip doesn't include dot files by default

Both were fixed in #780 and released in 5.4.2 by listing files to put in the zip and give it to bestzip (so we can include dot files and also exclude file using the excludeRegex option). But the list can be very long, as you experienced, and broke the command.

We can count the number of files to zip and then, if the list is too big (but we need to determine the size...), switch to wildcard stuff. Which means it'll work for people which aren't using the excludeRegex option. For people using that option with too much files to package, it'll break again ...

I don't have a perfect solution right now.
Could you try locking serverless-webpack to 5.3.5 and test it again?

[chriszirkel]

I pinned the version to 5.3.5 with npm i --save-dev --save-exact [email protected] and it did work to deploy.

[j0k3r]

Ok so maybe, if the file list is too long, we might force to use the _internal way of creating zip in nodejs_ instead of using the zip command.

But what could be the limit of that files? 1k, 2k, 5k? 🤷‍♂️

[chriszirkel]

I'm not familiar with the mechanics of the zip command, but would there be a possibility to create an initial zip and then add the node_modules in chunks of e.g. 1000 files?

[james-relyea]

Encountered this issue as well and was looking into a possible fix. I think the list of files to zip has to be under the limit of ARG_MAX (at least on a *nix-based OS).

```
~/project:master [!?$]$ getconf ARG_MAX
262144
````

256kb.

[j0k3r]

Looks like there were around 5k files "only" in OP issue, far away from the limit you are talking about

[james-relyea]

ARG_MAX pertains to the actual byte count of the total arguments string length rather than the count of individual arguments themselves.

So with 5k files and each argument entry in that array starting off with node_modules/, they've got a minimum of 13 bytes each. If the average argument length was 60 bytes for 5,000 items, that would break the limit at 300,000.

[dankelleher]

We had this issue too and pinning to 5.3.5 fixed it. we had a relatively small node_modules, but these things get big fast! As a stop-gap, perhaps setting the source list to '*' if no excludeRegex is set would solve the problem in the 90% case, buying time for a better solution? I tried that on a fork and it worked.

[pveller]

Just stumbled upon it myself after a simple npm update. Traced it, reported a note here - https://github.com/serverless-heaven/serverless-webpack/pull/777#issuecomment-832335138 - and went to open a new issue only to see it has already been posted. This is indeed the problem of command line length to nativeZip() and you don't need too many packages. Something like date-fns and lodash even with modularized imports will package them all and create a command line half-mile long :)

Downgrade to 5.4.0 indeed solved the problem.

It would be nice to be able to opt-in for * instead of file list or to force nodeZip() that by the looks of it individually adds files vs. shelling out to native zip. It's probably much slower but don't know how else you can accommodate folders that start with . when simple * omits them. Maybe there's a better native zip way to say I want truly everything including the . path?

[j0k3r]

I hit the bug as well and provide a fix #813
It works in all cases I tested locally.

Could you guys review it and test it?
Thanks 🙏