Serverless-webpack: Deployment missing git packages

[garyrutland]

This is a Bug Report

Description

Git packages not being included on deployment.
In package.json I have a package included via git "my-package": "git+ssh://[email protected]/my-git-package.git#master".
This has been working fine for months but all of a sudden it would seem that my package is not being bundled on deployment, but working fine when invoking locally.

If I download the function from Lambda I can see that my package is missing from the node_modules folder.

Config

.babelrc

{
  "plugins": [
    "transform-runtime"
  ],
  "presets": [
    "es2015",
    "stage-3"
  ]
}

serverless.yml

...
plugins:
  - serverless-webpack

custom:
  webpackIncludeModules: true
...

webpack.config.js

const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  // Since 'aws-sdk' is not compatible with webpack,
  // we exclude all node dependencies
  externals: [nodeExternals()],
  // Run babel on all .js files and skip those in node_modules
  module: {
    rules: [{
      test: /\.js$/,
      loader: 'babel-loader',
      include: __dirname,
      exclude: /node_modules/,
    }],
  },
};

Additional Data

• Serverless-Webpack Version you're using: 4.1.0
• Webpack version you're using: 3.10.0
• Serverless Framework Version you're using: 1.24.1
• Operating System: OSX

$ serverless deploy --stage=##REMOVED##
Serverless: Bundling with Webpack...
Time: 600ms
          Asset     Size  Chunks             Chunk Names
handlers/get.js  5.47 kB       0  [emitted]  handlers/get
   [0] ./handlers/get.js 2.54 kB {0} [built]
   [1] external "babel-runtime/regenerator" 42 bytes {0} [not cacheable]
   [2] external "babel-runtime/helpers/asyncToGenerator" 42 bytes {0} [not cacheable]
   [3] external "aws-sdk" 42 bytes {0} [not cacheable]
   [4] external "my-package" 42 bytes {0} [not cacheable]
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, aws-sdk@^2.165.0, git+ssh://[email protected]/my-git-package.git#master
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..........
Serverless: Stack update finished...
Service Information
service: ###REMOVED###
stage: ###REMOVED###
region: ###REMOVED###
stack: ###REMOVED###
api keys:
  None
endpoints:
  GET - ###REMOVED###
functions:
  get: ###REMOVED###
Serverless: Removing old service versions...

[HyperBrain]

Hey @garyrutland ,

thanks for raising the issue. In general it sounds strange especially because the logs show Serverless: Packing external modules: babel-runtime@^6.26.0, aws-sdk@^2.165.0, git+ssh://[email protected]/my-git-package.git#master what means that the package actually got installed (at least locally for the package).
Can you try to do a serverless package instead of a deploy and check the generated zip files in .serverless and tell me, if it is included there?

There were no changes in serverless-webpack that I remember that could have changed anything there. At work we also use github references and everything works as expected.

[HyperBrain]

Oh - can you also check if the commit that is locked in package-lock.json points to an existing commit? I once had an issue where I did a rebase of a branch, and the project that referenced the branch had a commit reference in the package-lock that did not exist anymore after the rebase.
NPM sometimes has problems with these situations.

BTW: If you run invoke local, it will use the local node modules. So as long as the module is locally available, it will work. If you find out that it is the package-lock issue, we should improve the error detection in the packaging, as it then should have bailed out there.

[garyrutland]

Hi,

I just ran serverless package and then checked the zip file, my package still wasn't there.
I checked the commit and it does exist, however, I did then completely delete the node_modules folder as well as the package-lock.json file and re-run npm install but it still is not there.

This is very frustrating as our systems are now down and have been since I created this ticket.
Any help would be greatly appreciated.

[HyperBrain]

Hi @garyrutland . Unfortunately I'm unable to reproduce the issue. On our systems everything works well.
Can you please add --verbose to the commandline and then SLS_DEBUG=* serverless ... --verbose?
These two variations should show much more information and might help to find out what exactly is going on there.

[garyrutland]

I've also updated another service to the same libraries and getting the same issue there now.

$ SLS_DEBUG=* serverless deploy --stage=##REMOVED## --verbose
Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command webpack
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Removing ##REMOVED##/.webpack
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
Time: 625ms
          Asset     Size  Chunks             Chunk Names
handlers/get.js  5.47 kB       0  [emitted]  handlers/get
   [0] ./handlers/get.js 2.54 kB {0} [built]
   [1] external "babel-runtime/regenerator" 42 bytes {0} [not cacheable]
   [2] external "babel-runtime/helpers/asyncToGenerator" 42 bytes {0} [not cacheable]
   [3] external "aws-sdk" 42 bytes {0} [not cacheable]
   [4] external "my-package" 42 bytes {0} [not cacheable]
Serverless: Invoke webpack:package
Serverless: Fetch dependency graph from ##REMOVED##/package.json
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, aws-sdk@^2.165.0, my-package@git+ssh://[email protected]/my-git-package.git#master
Serverless: Package took [4661 ms]
Serverless: Copy modules: ##REMOVED##/.webpack/service [1879 ms]
Serverless: Prune: ##REMOVED##/.webpack/service [1181 ms]
Serverless: Zip service: ##REMOVED##/.webpack/service [2774 ms]
Serverless: Packaging service...
Serverless: Remove ##REMOVED##/.webpack
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - ##REMOVED##
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - GetLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - GetLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1512640597764
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1512640597764
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1512640597764
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - ##REMOVED##
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1512640542640
CloudFormation - DELETE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1512640542640
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - ##REMOVED##
Serverless: Stack update finished...
Serverless: Invoke aws:info
Service Information
service: ##REMOVED##
stage: ##REMOVED##
region: ##REMOVED##
stack: ##REMOVED##
api keys:
  None
endpoints:
  GET - ##REMOVED##
functions:
  get: ##REMOVED##

Stack Outputs
GetLambdaFunctionQualifiedArn: ##REMOVED##
ServiceEndpoint: ##REMOVED##
ServerlessDeploymentBucketName: ##REMOVED##

Serverless: Invoke aws:deploy:finalize
Serverless: Removing old service versions...

[HyperBrain]

I now created a new project (based on babel-dynamically-example), and added one of our GIT modules to it (and used it from the first function, so that it is actually needed). Then I did a serverless package.
I did the same with package: individually set to true and to false (as I saw that you use service packaging).

The log outputs for both tests are the same as yours, but it actually included the module in the zip's node_modules correctly exactly as expected.
I currently have npm 5.3.0 installed on my system, use SLS 1.24.1 and Node 6.10.1.

You mentioned that the failing behavior all appeared out of a sudden from one day to another. Were there any significant changes on the build system or in the git repo? I do not know how I can track this further on my side. I just cannot reproduce it 🤔 . Does it happen with any git+ssh repo, or only with some specific ones?

[garyrutland]

I'm using Node 8.9.2 and NPM 5.5.1, no idea if that makes a difference.
It's only one git repo I'm trying to include and haven't made any change there either (other than code).

[garyrutland]

I'm not packaging individually either, not sure where you got that from.

[HyperBrain]

Can you try to switch one of the "regular" dependencies to its git repo + tag to see if it also happens there?
E.g. "lodash": "git+ssh://[email protected]/lodash/lodash#4.17.4" (the correct tag for this example might be 4.17.4-npm, I'm not sure about that)
Otherwise a npm cache clean or cleaning the git cache would be a further option.
... or of course the NPM version.

[HyperBrain]

I saw that you do not use individual packaging (you can see that looking at the webpack folder in the logs: /service/ means no individual packaging) 😃

[garyrutland]

Looks like the cache clear has resolved it, thanks very much for the help.

[HyperBrain]

This was a tough one 😄

[garyrutland]

Hi @HyperBrain, I think I've been able to figure out how to replicate this and it seems that anything above NPM version 5.5 and there are issues.

[HyperBrain]

Hi @garyrutland , that's interesting. So it might be that there is a regression in NPM itself with git references and package-lock.json or the npm cache. We should further analyze the behavior without the plugin (just with combinations of a simple npm based project with git and package-lock) and check if the lock file or the cache gets somehow out of sync.

Is there already a similar issue in the npm repo? For file references I raised an issue too - because I found it in the context of the plugin.

[HyperBrain]

I think I found it: https://github.com/npm/npm/issues/19356

The plugin does a npm prune to remove not used packages per function. Due to the NPM bug, it will errorneously remove the git packages.

[DanSchick]

I'm still having this same issue, when I try to include a git module with serverless-webpack and then require() it in my handler, the module cannot be found. Any help would be very appreciated

[HyperBrain]

@DanSchick which npm version are you running? it should work with anything < 5.5

[sairathb]

I have the exact same setup and faced the same problem, I ran serverless package --verbose and @HyperBrain you are right, the prune is removing the git package. Is there a workaround for now so that I can go on with my project. Is there a way that I can stop the npm prune from running

[HyperBrain]

@sairathb The only way to skip npm prune is if you switch to service packaging temporarily until the npm issue is fixed. (set package: individually: false). This disables the per function optimization, but as long as you do not have huge dependencies that are only used in a single of your functions, this should be ok as workaround.
However, anyone visiting this issue should put a comment at the original npm issue here to put some pressure on it. They do not really seem to notice how critical the issue is.

[HyperBrain]

I investigated the changelogs for npm 5.7.1 and saw that the git fix as well as some prune fixes were in there. I also saw, that they did not comment the old open tasks and inform users that this might be closed.
So I'd propose that we do some tests with [email protected] to see if the problem is fixed. If it is, we can add a section for supported npm versions to the docs (<5.5 && >=5.7.1)