Aws-cdk: @aws-cdk/aws-lambda-nodejs doesn't build functions with native modules

[thekevinbrown]

I'm trying to get a lambda set up with @aws-cdk/aws-lambda-nodejs.

I applied the change mentioned in #6274, and now I have a build error:

Error: Failed to build file at [my lambda function]: Error: 🚨 [project path]/node_modules/pg/lib/native/client.js:11:21: Cannot resolve dependency 'pg-native'

Reproduction Steps

Use "pg": "7.18.1", then in the source of the lambda:

import * as pg from 'pg';

Error Log

Error: Failed to build file at [my lambda function]: Error: 🚨 [project path]/node_modules/pg/lib/native/client.js:11:21: Cannot resolve dependency 'pg-native'

Environment

• CLI Version : 1.24.0
• Framework Version: 1.24.0
• OS : OS X 10.15.3
• Language : Typescript

Other

I'm familiar with how to work around this error in Webpack, but given that Parcel is a zero config tool, I'm not sure what to do about this error.

I think connecting to RDS databases from Node Lambdas will be a common use case, so this should be supported out of the box if possible, or at least have a documented solution.

---

This is :bug: Bug Report

[thekevinbrown]

Note: I found the following issues on the Parcel repo with no clear way to resolve the issue:

https://github.com/parcel-bundler/parcel/issues/4031
https://github.com/parcel-bundler/parcel/issues/3977
https://github.com/parcel-bundler/parcel/issues/2493

[thekevinbrown]

Alright, that's odd. Even though pg-native should be an optional install, running npm i --save pg-native has fixed the build issue...

I'll keep digging and see if I can find a more definitive answer.

[thekevinbrown]

Also:

just a feedback, tried to use parcel for serverless, didn't do too hot because the way parcel 1 greedily tries to take care of everything. bundle size with parcel: 13MB, bundle size with serverless-webpack: 31kb
i think that would be the main use

From: https://github.com/parcel-bundler/parcel/issues/2493

I'm seeing the same ~14mb with very little actual code. I suspect most of it is the aws-sdk.

[thekevinbrown]

Yeah, I'm having a lot of trouble with Parcel in general.

I'm doing:

import { TypeOrmLoggerAdapter } from '@eropple/typeorm-bunyan-logger';

new TypeOrmLoggerAdapter(logger),

And I get

"ReferenceError","errorMessage":"typeorm_bunyan_logger_1 is not defined","stack":["ReferenceError: typeorm_bunyan_logger_1 is not defined"

So I'm really not sure how Parcel works or why I'd get these errors. When I run the code unbundled with node or ts-node it works fine, but when it goes through Parcel, it stops working.

@jogold I noticed this: https://github.com/aws/aws-cdk/issues/1620#issuecomment-458902329 Any chance you could share an example repo with that approach? I'm not seeing how the webpack build is actually run in that setup.

[jogold]

I'm seeing the same ~14mb with very little actual code. I suspect most of it is the aws-sdk.

All the parcel ecosystem is available to you, I'm personally using parcel-plugin-externals which deals with this nicely.

So I'm really not sure how Parcel works or why I'd get these errors.

I don't know the module you're using @eropple/typeorm-bunyan-logger, but if you can get parcel to bundle it, it will work with @aws-cdk/aws-lambda-nodejs. If not, you should fallback to your own build method and use @aws-cdk/aws-lambda directly.

I'm not seeing how the webpack build is actually run in that setup.

The webpack build was run outside of the CDK. I don't have an example repo for this.

[thekevinbrown]

That makes sense, again I've only used webpack, so I'm not familiar with the Parcel ecosystem. I searched repeatedly for how to configure parcel to exclude dependencies and kept just finding "zero configuration!".

I think it'd be good to configure parcel as distributed by this module to exclude aws-sdk (and only aws-sdk) by default.

Even when you know to search for "parcel plugin", you still just find this page: https://parceljs.org/plugins.html which doesn't tell me how to configure which modules are included and which are excluded, so documenting this in our docs would be helpful for people I think.

[thekevinbrown]

I was able to build a WebpackCode asset that works with CDK, and my bundle size went from 14MB to 700KB.

• Most of the reduction in size was excluding aws-sdk (From 14MB down to 4MB)
• I'm using a monorepo, so TypeORM was included from two packages, which meant it went into the bundle twice. Aliased so that it always resolves to the same one (down to 2.2MB).
• TypeORM includes all the drivers for every database, and I was able to ignore all the ones I'm not using. (down to 700KB).

I'm not sure how I'd do that with Parcel. Given that cold boot times are important on Lambda, I feel like @aws-cdk/aws-lambda-nodejs should allow this type of configuration / optimisation if it doesn't already, and how to do this should be documented in the readme.

I'm happy to help contribute to this except that I don't know how to accomplish that with Parcel.

[eladb]

@jogold I think we should add support for building the function inside a Lambda docker image (like/with "sam build"). What do you think?

[jogold]

I think we should add support for building the function inside a Lambda docker image

Yes, it would be a nice addition. One of the problems with this is how to isolate the modules used by a single Lambda function (vs those used for CDK or other functions in the same module). The simple approach of installing what's in the package.json would result in a lot of unwanted packages in the Lambda zip file.

[alexpulver]

I think we should add support for building the function inside a Lambda docker image

Yes, it would be a nice addition. One of the problems with this is how to isolate the modules used by a single Lambda function (vs those used for CDK or other functions in the same module). The simple approach of installing what's in the package.json would result in a lot of unwanted packages in the Lambda zip file.

What about having multiple package.json files, one for infrastructure logic dependencies and another for application logic dependencies? I do this in Python and this way keep the Lambda ZIP file with application dependencies only.

[thekevinbrown]

Just to close the loop here, I ended up building my own webpack build step, and with further tweaking got the bundle down to 75.4KB. Things that I needed to do that I'm not sure how to do with Parcel:

• Used null loader to stub out a module that was being included but not used at runtime. Couldn't mark it as an external because it's being forcibly required, but it isn't actually being used.
• Configured .mjs files to load as javascript. (Not sure if this comes out of the box with Parcel or not.)
• Bundled the .node native modules that I generate during build on a lambci image with node-loader.
• Marked my dev dependencies as externals so they don't end up in the bundle (assumed this is possible using parcel-plugin-externals.
• Because this is a monorepo with multiple lambdas, I need to have different webpack configs for each, tweaking settings. Not sure how this'd work in Parcel either.

[jogold]

Regarding externals and natives, I think that it will be possible to offer a better API after #7169 gets merged.

[eladb]

@jogold will the docker build help here?

[jogold]

@jogold will the docker build help here?

Yes, on it.

[moltar]

I'm with @thekevinbrown here.

I think:

• parcel is too primitive and lacks configuration. It is great if you are building a simple web app, but it falls apart on large, backend-oriented projects.
  
  
  
  • using docker is way too heavy handed and is extremely slow.
  
  
  • It is also a pain to use, if you don't have Docker installed, e.g. in a CI environment.
  
  
  • I am not even sure if GH Actions supports running arbitrary Docker containers. It might, but I couldn't find any info on that.
  
  
  • Trying to test the infra via Jest basically runs all of the Docker bundling every time, which is extremely slow.
  
  

Overall DX here is very sluggish and full of errors.

I think a better approach would be to go for Webpack or Rollup, with a sane default configuration but allow consumers to supply config overrides.

You can see this being used successfully in projects like Nuxt. Where nuxt.config.ts file exposes props (simple) and hooks (advanced) for configuration of Webpack internals, but it is not necessary to touch them for most projects, and everything just works.

Likewise @aws-cdk/aws-lambda-nodejs could automatically fix many issues, like exclusion of aws-sdk, and ignoring pg-native module.

[babtsoualiaksandr]

Alright, that's odd. Even though pg-native should be an optional install, running npm i --save pg-native has fixed the build issue...

I'll keep digging and see if I can find a more definitive answer.

gyp ERR! node -v v12.16.1
gyp ERR! node-gyp -v v5.1.0