Serverless-webpack: Cannot package external module directories referenced by require() with path

[otbe]

This is a Bug Report

Description

I try to require a precompiled binary:

require('../compiled/module')

I added this to my webpack config for the externals part to externalize the binary module:

function(context, request, callback) {
      if (/compiled/.test(request)) {
        return callback(null, true);
      }
      callback();
    }

The expected output is:

...
/* 16 */
/***/ (function(module, exports) {

module.exports = require("../compiled/module");

/***/ }),
...

For copying the necessary files I use the CopyWebpackPlugin as you suggested in another issue. Everything works from the webpack view fo point, but at the end of sls webpack it fails with a strange error message:

Serverless: WARNING: Could not determine version of module ..
Serverless: Packing external modules:a@^0.25.0, b@^2.0.3, .., [email protected]

  Error --------------------------------------------------

  Command failed: npm install
npm ERR! code EINVALIDPACKAGENAME
npm ERR! Invalid package name "..": name cannot start with a period

I tried to forceExclude the .. module but it didnt work :/

Is this a serverless-webpack issue or a serverless issue or am I doing something completely wrong?
My current workaround is

eval('require')('../compiled/module')

which works pretty good, but feels a bit hacky :)

Package versions:

{
    "serverless": "^1.24.1",
    "serverless-webpack": "^3.0.0",
    "webpack": "^3.6.0"
}

Thank you for this great project :)

[HyperBrain]

Hi @otbe ,
thanks for raising the issue. What is strange with the configuration is, that webpack seems to determine .. as dependency. This will of course lead to errors, because .. is indeed an invalid module name and cannot be handled correctly.

You see that here:

Serverless: WARNING: Could not determine version of module ..
Serverless: Packing external modules:a@^0.25.0, b@^2.0.3, .., [email protected]

Can you post your webpack config, or prepare a sample project, that reproduces the issue?

BTW: You can update to ^4.0.0. That is the newest version.

@boazdejong - Do you have an idea here? Can your fork of the copy plugin be used to achieve the required behavior?

[otbe]

@HyperBrain
thank you for your response :)
Upgraded to v4, but the error still occurs.

Here is my webpack config (its a small modified version of the aws-nodejs-typescript starter)

const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');
const CopyWebpackPlugin = require('copy-webpack-plugin');

module.exports = {
  entry: slsw.lib.entries,
  resolve: {
    extensions: ['.js', '.jsx', '.json', '.ts', '.tsx']
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js'
  },
  target: 'node',
  externals: [
    nodeExternals(),
    function(context, request, callback) {
      if (/compiled/.test(request)) {
        return callback(null, true);
      }
      callback();
    }
  ],
  plugins: [
    new CopyWebpackPlugin([{ from: 'compiled/module', to: 'compiled/module' }])
  ],
  module: {
    loaders: [{ test: /\.ts(x?)$/, loader: 'ts-loader' }]
  }
};

Related serverless.yml part:

  webpackIncludeModules:
    forceInclude:
      - module

I plan to open source my little project within the next 2 days, so you will be able to reproduce this error very soon.

Thanks

[otbe]

Hi @HyperBrain,

I published the related project at https://github.com/Mercateo/serverless-image-processor.
Just remove the eval('') part from https://github.com/Mercateo/serverless-image-processor/blob/master/src/Sharp.ts#L6 and you should get an error with sls webpack

[HyperBrain]

@otbe Thanks. I will check it soon 😄

[HyperBrain]

@otbe I just started testing with your demo project. I will try to isolate the issue and give you feedback in at least some days.

[HyperBrain]

I was able to reproduce it. The local compiled module is correctly determined by webpack as external:
[19] external "../compiled/sharp" 42 bytes {0} [not cacheable]

However the webpack plugin does not catch this special case. I'm thinking right now, if the right solution would not be, that the plugin would just copy the whole module folder into the correct location as soon as it encounters a local external pointed out by webpack. According to webpacks compile results the external is a needed dependency (exactly like the other standard NPM dependencies too).

One additional possible issue in your code:

  if (!isDevEnv()) {
    return require('../compiled/sharp');
  } else {
    return require('sharp');
  }

You determine which version (local or NPM) to use by a dynamic function isDev(). I wonder if setting a constant via the webpack DefinePlugin would not be better here. Then you'd set a constant IS_DEV via the plugin in your webpack config which is just accessible for webpack in the source (use if (IS_DEV) { ... there) depending on the serverless environment (stage, etc.).
The big advantage is, that for production builds, webpack will be able to strip out the dead-code in the if and remove the unused dependency from the externals. Otherwise the local compiled sharp version would be bundled, even if it is never used, because webpack cannot determine by 100% that the isDev() function will return false in all possible cases on a production deployment.

However, I will analyze and fix the local dependency issue and prepare a PR for that.

Note: The reason why your eval('require')('../compiled/module') fix works is, because this prevents webpack from reporting the module as external. This circumvents the bug because you manually copy the module into the package with the copy plugin.

[HyperBrain]

As soon as it is fixed all copy plugin uses or other workarounds should be removed.