Serverless-webpack: include/exclude not working (trying to include a binary with a function)

[bastibense]

This is a Bug Report

Description

Started with the aws-nodejs-typescript template.

I'm trying to include an executable/binary with the function package, so it can be executed on Lambda to do some work there. The binary would be called from within the function as you'd normally call command line binaries on linux.

I tried different variants of include/exclude config params, set individually to false and other tweaks to get the generated package/lambda function to include the binary, but without success.

Am I doing something wrong here, or is it a bug?

serverless.yml

service:
  name: myservice

package:
  individually: true

plugins:
  - serverless-webpack
  - serverless-offline

provider:
  name: aws
  runtime: nodejs6.10
  region: eu-west-1
  memorySize: 1024
  stage: dev


functions:

  processInputFile:
    handler: input-processor/handler.hello
    package:
      include:
        - bin/procbinary
    events:
      - http:
          method: get
          path: render-pdf

File Layout

• input-processor
  
  
  
  • bin
  
  
  • procbinary (my custom executable)
  
  
• node_modules
• tsconfig.json
• webpack.config.js
• package.json
• serverless.yml

Additional Data

• Serverless-Webpack Version you're using: 3.1.0
• Webpack version you're using: 3.6.0
• Serverless Framework Version you're using: 1.23.0
• Operating System: macOS 10.12.6 (16G29)

[HyperBrain]

Hi @bastibense , thanks for asking 👍

We had this discussion just recently. The solution that we came up with, is to use the proper webpack plugin (copy file), that will allow to copy files including permissions.

Please read the thread here https://github.com/serverless-heaven/serverless-webpack/issues/205

And the conclusion: https://github.com/serverless-heaven/serverless-webpack/issues/205#issuecomment-331312739

In general, webpack plugins should be used to include specific files (in one or the other way).

[HyperBrain]

@bastibense Did you manage to solve your problem by using the webpack plugin mentioned above?

[bastibense]

@HyperBrain Thanks for the follow-up. I will get some time to try it later today. Will let you know how it turns out. 😊

[bastibense]

I tried this by putting the following in my webpack.config.js:

plugins: [
  new WebpackPluginCopy([
    { from: 'input-processor/procbinary' }
  ])
]

When I run sls package and inspect the resulting ZIP archive

The binary is copied, as expected 👍 (It shows up in the webpack summary and I find it in the resulting ZIP file at the root of the archive) - didn't get to call it while running on lambda yet, though, working on that now.

I noticed the structure of the resulting ZIP is different now: the _handler.js_ is not at the root of the archive anymore. Instead it is placed in a subfolder (input-processor/handler.js). I assume this is not a huge problem as long as Lambda finds the handler.js and executes it correctly. Removing the "plugins" portion of the config results in a simple ZIP file with only the handler.js at the root.

Without plugin:

- handler.js

With plugin:

- input-processor
  - handler.js
- procbinary

This is copied for all function deployments. I have multiple functions and I just want one function to contain the binary (hence having individually: true).

How do I limit the copy plugin to only do its work for one function/copy the binary only for one function and not all?

Another thing I noticed is that despite using copyPermissions: true, the permissions of the binary are not set correctly.

[HyperBrain]

@bastibense According your function definition in serverless.yml, the handler should also be located in input-processor/handler.js without the plugin option - it should exactly resemble the service definition. Can you check if you have set filename: '[name].js' in the output section of the webpack config? This will make sure that the folder structure is exactly the same as if it were copied without the webpack plugin by SLS raw packaging.

Regarding that the binary should be only copied along with one function, I'll have to check. We have one project that also uses the copy plugin (for CSS files). Never really checked, if they are only packaged with one lambda 😄

@boazdejong Do you have any idea why the permissions are not copied correctly?

[bastibense]

@HyperBrain I'll try to do some more tests and put together some useful feedback for you. This might take a day or two due to a different project I'm finishing, so please bear with me.

Basically I'd try to create a vanilla serverless project from the nodejs-typescript template, run sls package and unzip the resulting ZIP. Now add the plugin (without adding the binary yet) and the structure should be different. I was able to toggle this on/off by adding and removing the entry in the plugins section of the serverless config.

I'll get back to you asap.

Thanks for the quick response!

[boazdejong]

@HyperBrain @bastibense I'm not sure what happened, but something went wrong when merging the changes. Just commited again and it should work now. Please update webpack-plugin-copy to 1.0.1

[HyperBrain]

@bastibense Did you have a chance to try @boazdejong 's latest version?

[boazdejong]

@HyperBrain @bastibense I have successfully copied a binary file with its permissions with [email protected]. Think this can be closed for now.

[HyperBrain]

Thanks for the feedback, and good to hear that it worked.

[patoncrispy]

@bastibense, did you figure out how to only copy the binary for a single function? I've been trying to solve this for a while now and haven't been able to get to the bottom of it... :(

[sepehr]

I wonder if there are any plans to make the inclusion/exclusions behavior as per serverless service definition? I've found many similar issues closed in favor of webpack-plugin-copy which doesn't seem to be actually a solution to this limitation.

The point is that plugin does not address the issue with per-function inclusions. It also has nothing to do with service/function-wide exclusions which remains a limitation. Plugins like serverless-plugin-common-excludes simply don't work with serverless-webpack.

Is there anything I'm missing here?

@HyperBrain

[HyperBrain]

Hi @sepehr , other plugins that exclude files are not comparable with serverless-webpack at all, as long as it concerns dependencies (node_modules). For serverless-webpack, webpack is the single source of truth and will determine which dependencies are needed, and which are not. Any manual intervention (with exception of the existing forceExclude and forceInclude settings) would make the optimization done by webpack undone.

Regarding "payloads" that are not part of the compiles, I agree that there should be a way to include arbitrary files into the output package, per function. This is yet missing. Excluding however, will not make any sense, because the compiled code only contains things that are bundled by webpack and thus referenced and needed. Currently the copy files plugin does not select per function - a proper solution would be to use the package: include settings of a function and configure the copy files plugin automatically per function compile.

So to answer your question - the other plugins are to "correct" Serverless' unconditional packaging in a meaningful way 😄 . Serverless-webpack does a completely separate packaging which is optimized to utilize webpack's optimization and dependency management, so they solve different problems.

I can try to provide a PR for the function-wise inclusion, as soon as I'm done with version 5.1.0.

[HyperBrain]

We should handle the per function include in a separate feature request. I will open one later.

[sepehr]

Hey Frank,

Thank you for the explanation. It's much clearer now. That'd be great if you could also add this explanation to the readme for the newcomers.

_"a proper solution would be to use the package: include settings of a function and configure the copy files plugin automatically per function compile."_

I tried finding out a way to utilize the service definition through slsw to no avail. I'm looking forward to seeing it live.

_"Excluding however, will not make any sense, because the compiled code only contains things that are bundled by webpack and thus referenced and needed."_

I see, however, one usecase might be to drop the unused assets & files within linked modules, e.g. *.md, docs/, test/ and even further reducing the size by utilizing something like serverless-plugin-reducer.

_"I can try to provide a PR for the function-wise inclusion, as soon as I'm done with version 5.1.0."_

That's even more appreciated.

[phillipsnick]

Know this is closed, but kept coming back to this issue when trying to find a solution. I'm using webpack too which results in the binary not being copied.

Install the plugin

npm install --save-dev webpack-plugin-copy

To copy binary add this to webpack.config.js, make sure to include permissions

  plugins: [
    new WebpackPluginCopy([
      {
        from: 'node_modules/rsvg-convert-aws-lambda-binary/vendor/rsvg-convert',
        copyPermissions: true
      }
    ])
  ]

Then to execute the binary

import childProcess from 'child_process';

// some handler logic

childProcess.spawn('./rsvg-convert', params);

Hope that saves someone the 2 hours it took me to figure out :)