Serverless-webpack: AWS-SDK Included in deployment package

[mzuniga84]

aws-sdk is being included in my deployment package even after I set forceExclude in serverless.yml.

• In my serverless.yml I have:
  ```
  custom:
  webpackIncludeModules: # enable auto-packing of external modules
  forceExclude:
  
  
  
  • "aws-sdk"
  
  

* In my webpack config file I have
`externals: [nodeExternals()]`

* When running the deploy process (sls deploy) my output looks like this:

Serverless: Bundling with Webpack...
Hash: a35bbeb42d11a50e378b
Version: webpack 3.9.1
Time: 872ms
Asset Size Chunks Chunk Names
functions/processor/index.js 39.5 kB 0 [emitted] functions/processor/index
functions/processor/index.js.map 61.8 kB 0 [emitted] functions/processor/index
[0] ./functions/processor/index.js 7.28 kB {0} [built]
[1] external "aws-sdk" 42 bytes {0} [not cacheable]
[2] external "clone" 42 bytes {0} [not cacheable]
[3] external "axios" 42 bytes {0} [not cacheable]
[4] external "bluebird" 42 bytes {0} [not cacheable]
[5] ./functions/processor/lib/schemas/ingest.js 2.9 kB {0} [built]
[6] ./functions/processor/lib/parser-templates/default.js 26 kB {0} [built]
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: clone@^2.1.1, axios@^0.17.1, bluebird@^3.5.1

* In my package.json aws-sdk is a dev dependency:

"devDependencies": {
"aws-sdk": "^2.162.0",
"babel-core": "^6.25.0",
"babel-loader": "^7.1.1",
"babel-plugin-transform-runtime": "^6.23.0",
"babel-polyfill": "^6.23.0",
"babel-preset-env": "^1.6.0",
"eslint": "^4.12.1",
"eslint-config-airbnb-base": "^12.1.0",
"eslint-plugin-import": "^2.8.0",
"serverless-plugin-tracing": "^2.0.0",
"serverless-webpack": "^4.1.0",
"webpack": "^3.9.1",
"webpack-node-externals": "^1.6.0"
}
```

Still, somehow the package is making it into my deployment file

Any help would be appreciated.

Thank you in advance

[HyperBrain]

Hi @mzuniga84 , thanks for asking.

It seems that the plugin did not recognize the forceExclude setting (otherwise it should have printed exluding aws-sdk.
I'm not sure if it's related to the quoting. To be sure remove the quotes around aws-sdk:

custom:
 webpackIncludeModules: # enable auto-packing of external modules
   forceExclude:
     - aws-sdk

and check the indentation of webpackIncludeModules. In your pasted code there is only one space.

Then do a serverless package --verbose and check the output again. If it does not exclude it, please paste the verbose output.

[mzuniga84]

Hello @HyperBrain, Thank you for the quick response!

I apologize for my delay.

The indentation in my serverless.yml is correct, I made a mistake when I pasted the code in my first comment. I removed the quotes around aws-sdk, so now it looks like this:

custom:
  webpackIncludeModules: # enable auto-packing of external modules
    forceExclude:
      - aws-sdk

After doing a serverless package --verbose the output is:

➜ serverless package --verbose
Serverless: Removing /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/.webpack
Serverless: Using multi-compile (individual packaging)
Serverless: Bundling with Webpack...
Hash: dbf3c8ff516b3b757136
Version: webpack 3.9.1
Time: 911ms
                           Asset     Size  Chunks             Chunk Names
    functions/processor/index.js  41.4 kB       0  [emitted]  functions/processor/index
functions/processor/index.js.map  65.1 kB       0  [emitted]  functions/processor/index
   [0] ./functions/processor/index.js 9.19 kB {0} [built]
   [1] external "aws-sdk" 42 bytes {0} [not cacheable]
   [2] external "clone" 42 bytes {0} [not cacheable]
   [3] external "axios" 42 bytes {0} [not cacheable]
   [4] external "bluebird" 42 bytes {0} [not cacheable]
   [5] external "@olympusat/olyplat-file-parser" 42 bytes {0} [not cacheable]
   [6] external "@olympusat/oly-fs" 42 bytes {0} [not cacheable]
   [7] ./functions/processor/lib/schemas/zoomin-ingest.js 2.9 kB {0} [built]
   [8] ./functions/processor/lib/parser-templates/zoomin-to-rightv.js 26 kB {0} [built]
Serverless: Fetch dependency graph from /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/package.json
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: clone@^2.1.1, axios@^0.17.1, bluebird@^3.5.1, @olympusat/olyplat-file-parser@^2.1.7, @olympusat/oly-fs@^3.0.1
Serverless: Package took [3221 ms]
Serverless: Copy modules: /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/.webpack/processor [1676 ms]
Serverless: Prune: /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/.webpack/processor [1110 ms]
Serverless: Zip function: /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/.webpack/processor [3323 ms]
Serverless: Packaging service...
Serverless: Remove /Users/mzuniga/projects/distribution/in-zoomin-vod-to-rightv/.webpack
Serverless: Tracing ENABLED for function "ds-zoomin-vod-to-rightv-dev-processor"

The package is created under .serverless/processor.zip. When I unzip the package I see aws-sdk in the node_modules.

Looks like the forceExclude is not being applied, but I'm not sure why.

Thank you again

[HyperBrain]

@mzuniga84 Thanks for the reply.
According to

Serverless: Packing external modules: clone@^2.1.1, axios@^0.17.1, bluebird@^3.5.1, @olympusat/olyplat-file-parser@^2.1.7, @olympusat/oly-fs@^3.0.1

It does not install the aws-sdk actively, nor does it detect the aws-sdk as a 1st level dependency. Otherwise it had explicitly logged the exclusion.

One idea I have is, that the dependency detected by webpack ([1] external "aws-sdk" 42 bytes {0} [not cacheable]) comes in with another dependency (nth level). Can you check if one of the packages listed in the packaging line contains the aws-sdk? The plugin does an explicit install of them, so also all their dependencies are installed.
A solution for such a problem is, to just bundle the dependency that references the aws-sdk by using it in the node-externals whitelist option like this:

nodeExternals({ whitelist: [ 'the-dep-that-refs-aws-sdk' ] })

The second idea is, that the package-lock is corrupted and npm just installs it because it is listed as prod dependency there (however that come). Can you try to delete your package-lock.json and do a npm install to force it to be recreated? Especially if you use npm >= 5.5 you might run into various instability bugs of npm.

[mzuniga84]

@HyperBrain

I tried whitelisting the package but this didn't solve the issue, I also tried reverting my npm binary to 5.3.0 (was 5.6.0) and deleting/recreating package-lock.json, this didn't work either. I do see one of the packages I'm using requires aws-sdk, I was assuming the forceExclude option was supposed to work like an exclude deep, removing the dependency from the bundle even if is a dependency of a dependency.

In any case is not a big deal. Everything is working fine in my deployed lambdas. My only concern is the bundle size but that's a minor issue, at least for me anyways.

Thank you again

[HyperBrain]

@mzuniga84 Thanks for the feedback.

Yes, currently the forceExclude only excludes 1st level dependencies, so in case it is fetched through another dependency, it will not be excluded.
However, this is a valid request, and I'm thinking about how a deep exclusion could be added.

[mzuniga84]

@HyperBrain

It can be tricky to balance efficiency with performance. A solution could be to remove the package(s) specified from the root node_modules right before bundling. In this case if a dependency requires an specific version of the said package(s) they will still be included under their own node_modules

[HyperBrain]

Removing anything from node_modules won't help at all, as the contents in there are optimized by npm and flattened. If you remove the aws-sdk folder, you'll only catch the sdk's files, but not its dependencies.

That's the reason, why the sls-webpack packaging only relies on npm and does not do any file changes manually within node_modules. You just cannot determine what belongs to a dependency manually.
NPM might even optimize modules used by multiple dependencies and move these to the root of node_modules. For that reason even the assumption that everything still is within the dependencie's folder is wrong.

[mzuniga84]

@HyperBrain Good point. I was looking at it from my specific example where I don't have a lot of dependencies and only a few packages are making it to the root node_modules, but you are right. It wouldn't be efficient especially when the "excluded" packages have a lot of specific dependencies

[HyperBrain]

However there might be a multi-pass approach possible. Currently npm ls is used to fetch the dependency tree with a depth of 1 (i.e. only direct dependencies).
I will do some tries with increasing the level to fetch nth level dependencies.

As soon as I have these, I just have to patch the dependencies' package.json in the very final compiled node_modules folders and run an npm prune again. If prune does not have bugs in that area, it could lead to the intended results. Pruning is the least costly phase in the packaging either (one can check that with --verbose). So for force excludes it would just add a few seconds - if the approach works.

[mzuniga84]

@HyperBrain

Great. I'm not too familiar with developing serverless plugins but if there anything I can help with just let me know.

Thank you again for your awesome work and quick responses

[HyperBrain]

Closing this in favor of #306

[danrivett]

Just adding a comment for those who may come to this late like me and was also stuck with aws-sdk being included. It took me quite a few different attempts (due to my ignorance) and I didn't get it working until I read the source code of this plugin and found the correct syntax that works for me is:

custom:
  webpack:
    includeModules:
      forceExclude:
        - aws-sdk

Just thought it was worth a post in case anyone else got stuck (I had forceExclude directly under webpack 🙄 )

[danrivett]

As a follow-up to my above comment, I found that although it did work for some of my services, as as soon as I added a dependency on the uuid package, aws-sdk got reincluded into the package. This increased the zipped package by 10mb and the unzipped package by almost 50mb. As this was completely unnecessary, I dug into this a bit more.

I'm quite a novice here, so others please correct me, but this is what I found:

I found that during an sls package the temporary /.webpack/service/node_modules/ still contained aws-sdk. Just deleting it caused an error during sls package because there was a .webpack/service/node_modules/.bin/uuid file that was symlinked to ../aws-sdk/node_modules/uuid/bin/uuid.

I don't know why this is happening - why uuid is symlinked to uuid under aws-sdk rather than directly under node_modules, and perhaps that's something that could be fixed??

But until then I added the following packaging option which packages up my services without including aws-sdk:

packagerOptions:
  scripts:
    - rm node_modules/.bin/uuid && rm -r node_modules/aws-sdk

This does not seem ideal, and likely to break in future as other dependencies are added and for some reason symlink to under aws-sdk, but my initial testing shows no adverse affects.

I am wary of this as I don't fully understand the consequences, so I welcome any feedback, including whether there's a bug or limitation with serverless-webpack which could address this symlinking issue.