Sharp: Cannot use sharp with AWS Lambda

[KrishKayc]

Hi,

I am using sharp in AWS Lambda.

The problem is that the packaged zip file to lambda is more than 250MB size which is allowed limit for lambda.

The sharp folder inside node_modules alone takes 76MB which is mainly causing this issue.

The build machine uses amazonlinux:2 image which is similar to AWS Lambda execution environment and we do the npm install there.

Is there a way to solve this problem?

[lovell]

The latest version of sharp including all of its dependencies and vendored prebuilt binaries for Linux occupies 36MB uncompressed on disk, which represents 14% of 250MB.

$ du -sh node_modules/
du: cannot access 'node_modules/': No such file or directory

$ npm install sharp

> [email protected] install /tmp/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached /.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz

+ [email protected]
added 71 packages from 105 contributors and audited 161 packages in 2.908s
found 0 vulnerabilities

$ du -sh node_modules/
36M node_modules/

$ du -sh node_modules/sharp/
33M node_modules/sharp/

$ du -sh node_modules/sharp/vendor/
32M node_modules/sharp/vendor/

[bosgeluk]

I'm successfully using sharp in AWS Lambda, to prevent such a big upload I do have put Sharp inside a Layer, so my upload remains small. I've deployed with serverless.

[shehantissera]

I'm successfully using sharp in AWS Lambda, to prevent such a big upload I do have put Sharp inside a Layer, so my upload remains small. I've deployed with serverless.

Would it be possible to share how you managed to get it working? Is there any documentation somewhere to be referred? thanks

[KrishKayc]

@lovell : I'm using sharp v 0.22.1, In lamba which uses amazonlinux:2 image, sharp occupies 76MB. Is the size reduced in sharp v 0.23.4?

[bosgeluk]

@shehantissera

Sure, download the sample I've constructed.
https://gabrielschoolputten.s3.eu-central-1.amazonaws.com/shehan.zip

Read the serverless.yml in the layer folder and adjust accordingly
In the layer/lib folder: npm install
In the layer folder: sls deploy

Read the serverless.yml in the root and adjust accordingly
In the root: npm install
start a local proxy (it's a bonus ;-) ) node ./proxy/index.js

... happy trails

Paul Hoogendijk

Op vr 6 dec. 2019 om 10:48 schreef Shehan Tis [e-mail]:

I'm successfully using sharp in AWS Lambda, to prevent such a big upload I
do have put Sharp inside a Layer, so my upload remains small. I've deployed
with serverless.

Would it be possible to share how you managed to get it working? Is there
any documentation somewhere to be referred? thanks

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lovell/sharp/issues/1998?email_source=notifications&email_token=AAX2KZCIIKW23L7PD3B2FS3QXINWVA5CNFSM4JVUZPSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGDSUMY#issuecomment-562506291,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAX2KZFATQ5TSUDSTSCA7ILQXINWVANCNFSM4JVUZPSA
.

[lovell]

@KrishKayc sharp and all its dependencies has never been that big. My best guess would be that your custom packaging process, unrelated to sharp, is unable to deal with symlinks and is instead copying duplicate files. Check the node_modules/sharp/vendor/lib directory.

[shehantissera]

@shehantissera Sure, download the sample I've constructed. https://gabrielschoolputten.s3.eu-central-1.amazonaws.com/shehan.zip Read the serverless.yml in the layer folder and adjust accordingly In the layer/lib folder: npm install In the layer folder: sls deploy Read the serverless.yml in the root and adjust accordingly In the root: npm install start a local proxy (it's a bonus ;-) ) node ./proxy/index.js ... happy trails Paul Hoogendijk
Thank you, I will give it a try.

[lovell]

@KrishKayc I hope this information helped. Please feel free to re-open with more details if further assistance is required.

[snorkypie]

Thought I would shed some light on the situation of @KrishKayc since I also ran into this.

zip defaults to make copies of symlinks and thus bloats the size of the vendor directory. This is solved by passing the --symlinks flag to zip. Here's a before and after using that flag:

-rw-r--r-- 1 steevel users 13M Feb 20 09:42 layer-sharp.zip
-rw-r--r-- 1 steevel users 32M Feb 20 09:36 orig-layer-sharp.zip