Serverless-webpack: indirect aws-sdk dependency is always included in artifacts resulting in 10MB+ artifacts

[tommedema]

This is a Bug Report

Description

For bug reports:

• What went wrong?
  aws-sdk is a built-in dependency on Lambda, therefore it should not be added to the bundle of functions that import aws-sdk, yet this does happen. Even when excluding aws-sdk using the forceExclude option. This results in each function being at least 10MB (the package itself is 25MB).

• What did you expect should have happened?
  For aws-sdk to not be in the artifact zip files node_modules folder.

• What was the config you used?
  I tried several variations. This is my webpack.config.js:

const slsw = require('serverless-webpack')
const nodeExternals = require('webpack-node-externals')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  devtool: 'source-map',
  stats: 'errors-only',
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.js$/,
        loader: 'babel-loader',
        include: __dirname,
        exclude: /node_modules/
      }
    ]
  }
}

And these are the variations of my serverless.yml that I tried; all failed to exclude aws-sdk:

plugins:
  # transpile functions' ES7 to ES5 for provider runtime compatibility
  - serverless-webpack

package:
  individually: true

custom:

  # allows for the exclusion of external modules to webpack
  webpackIncludeModules:
    forceExclude:
      - aws-sdk

I also tried:

  webpackIncludeModules:
    forceExclude:
      - node_modules/aws-sdk

I also tried:

package:
  individually: true
  exclude:
    - node_modules/aws-sdk

And finally:

package:
  individually: true
  exclude:
    - aws-sdk

Additional Data

• Serverless-Webpack Version you're using: 4.1.0
• Webpack version you're using: 3.9.1
• Serverless Framework Version you're using: 1.24.1
• Operating System: OSX

[tommedema]

Also, I tried this with aws-sdk being both a devDependency and a normal dependency. It's included in both cases.

These are my deps:

"devDependencies": {
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-source-map-support": "^1.0.0",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-es2015": "^6.24.1",
    "babel-preset-stage-3": "^6.24.1",
    "serverless-scriptable-plugin": "^0.6.0",
    "serverless-webpack": "^4.0.0",
    "standard": "^10.0.3",
    "uuid": "^3.1.0",
    "webpack": "^3.8.1",
    "webpack-node-externals": "^1.6.0",
    "shelljs": "^0.7.8",
    "aws-sdk": "^2.162.0"
  },
  "dependencies": {
    "babel-runtime": "^6.26.0",
    "cfn-response": "^1.0.1",
    "greenlock": "^2.1.18",
    "le-challenge-s3": "^1.0.1",
    "le-store-s3": "^1.0.1",
    "source-map-support": "^0.5.0"
  }

Note that le-challenge-s3 and le-store-s3 have aws-sdk defined as a dependency. Of course, it should still be excluded.

[tommedema]

Even setting this in webpack.config.js did not help:

module.exports = {
  ...
  externals: [nodeExternals(), /aws-sdk/],
  ...
}

[tommedema]

I just confirmed that this is due to indirect dependencies on aws-sdk. E.g. if I change:

import s3StoreFactory from 'le-store-s3'
import s3ChallengeFactory from 'le-challenge-s3'

to:

import s3StoreFactory from './no-op-challenge' // 'le-store-s3'
import s3ChallengeFactory from './no-op-challenge' // 'le-challenge-s3'

The package size will go from 10MB to 1MB.

Of course, indirect dependencies on aws-sdk should be excluded too. Is there any way to force this or should this be fixed at a lower level in serverless-webpack?

It seems that other build tools have "solved" this with a deepExclude, see https://github.com/nfour/serverless-build-plugin/issues/37

[HyperBrain]

Hi @tommedema ,
thanks for reporting. Yes, this is, because forceExclude excludes first level dependencies. As your 2 dependencies themselves introduce the aws-sdk it is not possible to exclude the aws-sdk installed from there. I'll explain why, below.

In general, this declaration is the correct one to exclude dependencies (from your tries above):

custom:
  # allows for the exclusion of external modules to webpack
  webpackIncludeModules:
    forceExclude:
      - aws-sdk

Workaround - Solution I

But you can do a workaround, to transform the aws-sdk into a first level dependency.
If you bundle the two le-* dependencies, they will become first level dependencies of the deployment package and the aws-sdk exclusion definition mentioned above will effectively exclude it.
Just declare:

externals: [nodeExternals( { whitelist: [ 'le-challenge-s3', 'le-store-s3' ] ) } ) ]

Then the modules will be bundled and aws-sdk is now a first level dependency. The exclude declaration now will lead to a proper exclusion.

Deep exclusion

I had a look at the deepExclude and the build module (source code) you mentioned above. It does not work that way. I'm pretty sure that it will eliminate just the aws-sdk folder and leaves all dependencies of the aws-sdk itself untouched.
The reason is, because NPM5 does a module flattening and optimization after install, that means that all dependencies of the aws-sdk are strayed beneith the /node_modules directory, and do not reside under node_modules/le-*.s3/node_modules anymore.
Even if you now delete the aws-sdk directory, you'll miss all dependencies of the aws-sdk which makes up nearly all of the sdk's size.
With NPM 5 (and other modern optimizing module packagers), you just cannot predict anymore, what exactly is part of a dependency just by looking at the directory structure. (See #239 for a discussion about that). So the directory structure must be treated as opaque and cannot be used to make any decisions.

You can verify that just by doing a npm install aws-sdk with an empty package.json. Then inspect the node_modules folder:

$ ls node_modules/
aws-sdk/    crypto-browserify/  isarray/   punycode/     url/     xmlbuilder/
base64-js/  events/             jmespath/  querystring/  uuid/
buffer/     ieee754/            lodash/    sax/          xml2js/

Of course, indirect dependencies on aws-sdk should be excluded too. Is there any way to force this or should this be fixed at a lower level in serverless-webpack?

So this cannot be solved in a correct way easily, if ever. Only NPM could solve that by having a switch/option to exclude packages - regardless where they appear.

Solution II

The most correct way is, that modules that require the aws-sdk define them as peerDependencies. So the le-* dependencies should have aws-sdk as peerDependency, which in turn requires the using module (your service) to have it as either "devDependecy" or "dependency" to fulfill the requirement.
Then the forceExclude will work as expected.
This is the same as with serverless-webpack and the webpack dependency.
As long as the modules have the sdk in their production dependencies it could additionally come to version conflicts and even duplications of the aws-sdk.

[tommedema]

Much appreciated, thanks again @HyperBrain.

I understand the situation and why it's hard to resolve. I will for now use externals: [nodeExternals( { whitelist: [ 'le-challenge-s3', 'le-store-s3' ] ) } ) ] and will consider automating this process in the future.

Not sure if this is very maintainable though, because it means that all these whitelisted packages will be bundled inside the webpack bundle and processed etc. I guess this could cause issues in the future, e.g. if they use binaries.

[craigtsmith]

I found a workaround described here: https://github.com/serverless-heaven/serverless-webpack/issues/306#issuecomment-420426295 for anyone struggling with this that can't change the library that requires aws-sdk

[danrivett]

Just adding a comment for those who may come to this late like me and was also stuck with aws-sdk being included.

It took me quite a few different attempts (due to my ignorance) and I didn't get it working until I read the source code of this plugin and found the correct syntax that works for me is:

custom:
  webpack:
    includeModules:
      forceExclude:
        - aws-sdk

Just thought it was worth a post in case anyone else got stuck (I had forceExclude directly under webpack 🙄 )

[KillDozerX2]

Ran into this recently and no the config in serverless.yml did not work. I had to update the webpack config by adding aws-sdk in the externals.