Serverless-webpack: Cannot use private packages on Codeship

[jake-lambert-volusion]

This is a Bug Report / Question

Description

• What went wrong?
  I am getting Error: Command failed: npm install just after Packing external modules: which includes a package from a private prefix (i.e. requires an auth token for the registry in .npmrc). I am able to run serverless package locally with no problem so maybe it is something particular to Codeship, however the initial npm install in our build process works fine, so I suspect either the .npmrc file isn't being seeing when serverless-webpack does npm install.

• What did you expect should have happened?
  A successful build.

• What was the config you used?

const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  entry: slsw.lib.entries,
  target: "node",
  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: [
          {
            loader: "babel-loader"
          }
        ]
      }
    ]
  }
};

• What stacktrace or error message from your provider did you see?

(node:5950) DeprecationWarning: Chunk.forEachModule: Use for(const module of chunk.modulesIterable) instead
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: [REMOVED - private scoped package], csv-parse@^2.2.0
[06:01:03] 'deploy:serverless:int' errored after 13 s
[06:01:03] Error: Command failed: npm install
npm ERR! code E404
npm ERR! 404 Not Found: [REMOVED]@https://registry.npmjs.org/[REMOVED]

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/rof/.npm/_logs/2018-04-27T06_01_03_547Z-debug.log

at ChildProcess.exithandler (child_process.js:273:12)
at ChildProcess.emit (events.js:180:13)
at ChildProcess.emit (domain.js:440:23)
at maybeClose (internal/child_process.js:936:16)
at Socket.stream.socket.on (internal/child_process.js:353:11)
at Socket.emit (events.js:180:13)
at Socket.emit (domain.js:440:23)
at Pipe._handle.close [as _onclose] (net.js:541:12)

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
• If there is additional config how would it look

Similar or dependent issue(s):

• #71 (mabye)

Additional Data

• Serverless-Webpack Version you're using:
  5.1.1
• Webpack version you're using:
  4.6.0
• Serverless Framework Version you're using:
  latest

• Operating System:

• Stack Trace (if available):

[HyperBrain]

Hi @jake-lambert-volusion . This is strange. We also use private scoped packages that need an authorization from .npmrc and it works correctly. We use Bamboo on a separate server for the deployment.

I think you're right that npm can somehow not find the .npmrc file. Is the user that executes the initial npm install the same that then runs the serverless deploy? I ask, because if you don't use a global .npmrc it will be fetched from the user's home directory.

[HyperBrain]

BTW: Can you try to let serverless be executed with SLS_DEBUG="*" serverless ... --verbose to get the maximum log out of it?

[jake-lambert-volusion]

@HyperBrain Thanks so much for the quick reply - very glad to hear that what we are trying to do can work.

It is the same user running npm install and serverless deploy (unless there is some weird codeship magic going on).

More debug output is below. Two additional details perhaps worth noting:

1. I am using gulp-serverless as well as serverless-webpack. We had gulp-serverless in place and I am trying to migrate a step at a time, rather than replacing our entire build process at once. Is this possible a problem source?
2. our .npmrc file is reading a token from an environment variable. Is it possible that gets lost in the course of spawning in gulp/serverless/serverless-webpack somewhere? Maybe .npmrc is being read but our token is undefined at that point so npm install fails?

[15:04:12] Starting 'deploy:serverless:int'...
[15:04:12] Deploying serverless project from path /home/rof/src/github.com/[REMOVED]
[15:04:13] Processing command deploy with options: { stage: 'int',
environment: 'integration'
verbose: true }
Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command prune
Serverless: Load command webpack
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Using configuration:
{
  "includeModules": true,
  "webpackConfig": "webpack.config.js",
  "packager": "npm",
  "packagerOptions": {},
  "packExternalModulesMaxBuffer": 204800
}
Serverless: Removing /home/rof/src/github.com/[REMOVED]/.webpack
Serverless: Using multi-compile (individual packaging)
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
Time: 814ms
Built at: 2018-04-27 15:04:20
Asset      Size  Chunks             Chunk Names
src/addOrUpdateRedirects.js  1.38 KiB       0  [emitted]  src/addOrUpdateRedirects
Entrypoint src/addOrUpdateRedirects = src/addOrUpdateRedirects.js
[0] external "@[REMOVED PRIVATE PACKAGE]" 42 bytes {0} [built]
[1] external "csv-parse/lib/sync" 42 bytes {0} [built]
[2] ./src/addOrUpdateRedirects.js 1.71 KiB {0} [built]
Serverless: Invoke webpack:package
Serverless: Fetch dependency graph from /home/rof/src/github.com/[REMOVED]/package.json
(node:5769) DeprecationWarning: Chunk.forEachModule: Use for(const module of chunk.modulesIterable) instead
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: [REMOVED PRIVATE PACKAGE], csv-parse@^2.2.0
[15:04:31] 'deploy:serverless:int' errored after 18 s
[15:04:31] Error: Command failed: npm install

[HyperBrain]

Is this possible a problem source?

Maybe. To eliminate that, you could setup a very small project that only uses serverless-webpack. The project should reference the private package in the dependencies and the sample code should use it somewhere. Then try to have this project built. You don't need to do a full deploy on your CI/CD. Just a serverless package instead of deploy should be sufficient, as it runs through the whole packaging.

A further hint for a possible interference of the two plugin is the deploy:serverless:int log line. This is not emitted by serverless-webpack, but maybe from the gulp plugin.

our .npmrc file is reading a token from an environment variable

serverless-webpack uses child_process.exec() and since version 5.1.2 child_process.spawn to execute npm. The environment is not altered and so the child process should inherit the full environment from the running serverless process. Of course I cannot state anything about the gulp plugin because we do not use it.

[jake-lambert-volusion]

deploy:serverless:int is our gulp task which calls serverless-gulp.exec.

I will try your suggestion and report back either way.

[jake-lambert-volusion]

Per https://github.com/npm/npm/issues/7995#issuecomment-175915766, adding echo "//registry.npmjs.org/:_authToken=\${NPM_AUTH_TOKEN}" > ~/.npmrc before sls package resolved it, so does indeed seem to be some kind of path resolution issue with my codeship setup and/or webpack config.

[HyperBrain]

@jake-lambert-volusion Nice find 👍 . Good to know that you were able to get it working.

[rpgreen]

Just experienced a similar issue using a private package from GitHub Package Registry. The local .npmrc did not work, but copying it to ~/.npmrc allowed serverless-webpack to succeed

[beanaroo]

If using yarn, specifying it as the packager is an alternative option. Please see: https://github.com/serverless-heaven/serverless-webpack/issues/71#issuecomment-553232334