Serverless: Deploy stops at "Installing dependencies for custom CloudFormation resources"

[mklueh]

I´m using this template https://github.com/msfidelis/serverless-architecture-boilerplate
and the issue has only occured with this template so far.

Creating new projects from scratch with the sls assistent and deploying works fine.

When running sls deploy it processes these 3 steps and then stops

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Installing dependencies for custom CloudFormation resources...

Running deploy with --verbose --force does not change anything and there are literally no additional exceptions or log messages thrown which makes it very hard to find the source of this issue.

Serverless v: 1.50.0
OS: Windows 10

[mardafirmansyah]

I also experienced the same problem, but with the error message:

EMFILE: too many open files, open 'C: \ Users \ marda \ AppData \ Local \ Temp \ tmpdirs-serverless \ 330d \ b435695edb9e63a1 \ resources \ node_modules \ tokopedia-api \ api node_modules \ caniuse-lite \ data \ regions \ GL.js'

 Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              12.8.1
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

Serverless v: 1.50.0
OS: Windows 10

Update:
After I tried to downgrade to the previous version (v1.49.0) it turns out that this message disappears and serverless runs normally. I think this is a bug for version v1.50.0

[medikoo]

@mklueh Great thanks for reporting, we'll look into that

[mvrska]

I also experienced the same problem, but with the error message:

EMFILE: too many open files, open 'C: \ Users \ marda \ AppData \ Local \ Temp \ tmpdirs-serverless \ 330d \ b435695edb9e63a1 \ resources \ node_modules \ tokopedia-api \ api node_modules \ caniuse-lite \ data \ regions \ GL.js'

 Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              12.8.1
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

Serverless v: 1.50.0
OS: Windows 10

Update:
After I tried to downgrade to the previous version (v1.49.0) it turns out that this message disappears and serverless runs normally. I think this is a bug for version v1.50.0

I'm getting a similar error with EMFILE: too many open files, also on win32 and serverless 1.50.0.
Unfortunately downgrading to 1.49.0 didnt help for me, but downgrade to 1.48.4 did remove the issue.

[GeneralistDev]

Same here on version 1.50.0 with the same EMFILE error as @mvrska, downgrading to 1.48.4 also worked for me. Windows 10 Version 10.0.18362 Build 18362.

[dested]

Chiming in, same story, 1.50.0 too many files, 1.48.4 deploys just fine.

Your Environment Information --------------------------- Operating System: win32 Node Version: 10.16.3 Framework Version: 1.50.0 Plugin Version: 1.3.8 SDK Version: 2.1.0

[mklueh]

@dested Downgrading to 1.48.4 has indeed worked for me too. Thank you

[Woojie]

This happens to me to, however i cannot downgrade below 1.49 as I wish to use eventBridge. Any suggestions would be great!

` Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: win32
Node Version: 12.9.0
Framework Version: 1.51.0
Plugin Version: 1.3.9
SDK Version: 2.1.0`

[francovp]

1.51.0 still has the same error.
Did they even try to fix the bug before the 1.51 release?

[GeneralistDev]

Looks to only affect windows so likely they aren't being affected thus making it less of a priority

[medikoo]

Work to be done is scheduled here: https://github.com/serverless/serverless/issues/6598.
It's in a queue, and if not with next minor release, I believe it should land with following one.

[pmuens]

Hey everyone,
thanks for reporting and confirming this issue. I just created https://github.com/serverless/serverless/issues/6557 to explore different solutions for this issue.

Unfortunately I don't have a Windows machine handy, so it would be super awesome if someone can checkout this PR and report back if this fixes the problem.

This issue seems to be related to https://github.com/serverless/serverless/issues/3249.

---

EDIT: apparently the current fix doesn't do anything at all. I took another deep dive into this and it seems like the zipping is the issue here (see: https://github.com/serverless/serverless/pull/6636#issuecomment-528360012). I'll work on an update on that front and post another comment when I have something to test.

[pmuens]

Hey everyone,
here's just a quick update: I've pushed a fix in #6636 in which I unified our zipping approaches. We're now using one zip util function for both, custom resource zipping and service zipping. This implementation uses graceful-fs when working with files in an async fashion so that the aforementioned bugs in Windows shouldn't happen anymore.

Would be really awesome if someone can test this branch out and confirm that this fixes the problem (I don't have a Windows machine). Thanks in advance!

[hakimio]

@pmuens Just tested: the fix works great on Windows. Would be nice if you could publish a new version soon.
For those who need the fix right now, here is a diff which can be applied using patch-package: github gist

BTW, @pmuens is there any reason you can't just create a VM on Azure to test fixes for Windows?

[pmuens]

Thanks for confirming @hakimio :+1:

Another update: I just created https://github.com/serverless/serverless/pull/6653 which has a smaller scope (we're about to release v1.52.0 soon and we want to ship this fix in that release). It would be nifty if some of you could test #6653 and confirm that this also fixes the problem (@hakimio do you think you could test that one as well 😬). Thanks in advance!

BTW, @pmuens is there any reason you can't just create a VM on Azure to test fixes for Windows?

Interesting. Haven't thought about that option 🤔

[hagaigold]

on windows 10 with 1.52.0 still stop after "Installing dependencies.." with this template https://github.com/AnomalyInnovations/serverless-nodejs-starter

Framework Core: 1.52.0
Plugin: 2.0.0
SDK: 2.1.1

[garethmcc]

@pmuens @medikoo Still seeing reports of this issue from users with latest version

[medikoo]

on windows 10 with 1.52.0 still stop after "Installing dependencies.." with this template

@hagaigold this template involves external plugins which interfere with packaging process. Do you observe same issue when not relying on those plugins?

[hagaigold]

@medikoo tried with "serverless create --template aws-nodejs" which work fine.

as the OP state, "--verbose", don't give any info on the nature of the error, so it a bit hard to track it down ;(

[medikoo]

@medikoo tried with "serverless create --template aws-nodejs" which work fine.

This one, actually installs this one: https://github.com/serverless/serverless/tree/master/lib/plugins/create/templates/aws-nodejs which doesn't involve any plugins

[hagaigold]

@medikoo, after replacing serverless-bundle plugin with serverless-webpack plugin, the deploy completed successfully.
if this issue is not a concern in this end, I can notify the plugin creator.

[Wintereise]

In my side, I'm seeing it with

plugins:
    - serverless-pseudo-parameters
    - serverless-stage-manager
    - serverless-stack-output
    - serverless-webpack
    - serverless-offline

1.52.0, win32

Same symptoms, no debug data. Deploy just stops after the package step.

[medikoo]

if this issue is not a concern in this end, I can notify the plugin creator.

Yes, best would be to coin, with which plugin issue occurs, and open the corresponding issue there

[garethmcc]

In my side, I'm seeing it with

plugins:
    - serverless-pseudo-parameters
    - serverless-stage-manager
    - serverless-stack-output
    - serverless-webpack
    - serverless-offline

1.52.0, win32

Same symptoms, no debug data. Deploy just stops after the package step.

@Wintereise The candidate plugins are probably serverless-stage-manager or serverless-stack-output. Disable each, in turn, to see if that is the problem.

[Wintereise]

@Wintereise The candidate plugins are probably serverless-stage-manager or serverless-stack-output. Disable each, in turn, to see if that is the problem.

I'll give that a try, reporting back in a while.

[mardafirmansyah]

I tried several times and generated random errors. if this is because of the plugin why when I downgrade to 1.49.0 this error disappears?

  Error --------------------------------------------------

  Error: EPERM: operation not permitted, unlink 'E:\Zenofa\api\tokopedia-api\.build\node_modules'


  Error --------------------------------------------------

  Error: EBUSY: resource busy or locked, unlink 'E:\Zenofa\api\tokopedia-api\.build\node_modules\typeorm\query-builder\transformer\PlainObjectToNewEntityTransformer.d.ts'


  Error --------------------------------------------------

  Error: EPERM: operation not permitted, unlink 'E:\Zenofa\api\tokopedia-api\.build\node_modules'


  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              12.8.1
     Framework Version:         1.52.0
     Plugin Version:            2.0.0
     SDK Version:               2.1.1

plugins:

• serverless-plugin-typescript
• serverless-domain-manager
• serverless-offline

[Wintereise]

I'll give that a try, reporting back in a while.

@garethmcc @medikoo Yeah, no bueno.

Here's what the verbose output looks like while this happens, sls 1.52.0, win32

I only had sls-webpack and sls-offline installed while running this test.

plugins:
    - serverless-webpack
    - serverless-offline

Exact versions from yarn.lock file

serverless-webpack@^5.3.1:
  version "5.3.1"

serverless-offline@^5.11.0:
  version "5.11.0"

Serverless: Invoke webpack:package
Serverless: Using package.json sections resolutions
Serverless: Fetch dependency graph from G:\<redacted>\manager\package.json
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: <long list of modules>
Serverless: Package took [4815 ms]
Serverless: Copy modules: G:\<redacted>\manager\.webpack\service [5 ms]
Serverless: Prune: G:\<redacted>\manager\.webpack\service [5470 ms]
Serverless: Run scripts: G:\<redacted>\manager\.webpack\service [1 ms]
Serverless: Zip service: G:\<redacted>\manager\.webpack\service [15941 ms]
platform-sdk fetching: GET https://api.serverless.com/core/tenants/<redacted><redacted>/applications/<redacted>
platform-sdk fetching: GET https://api.serverless.com/core/meta
Serverless: Packaging service...
Serverless: Remove G:\<redacted>\manager\.webpack
Serverless: Installing dependencies for custom CloudFormation resources...
<Radio Silence, runtime exits>

Framework Core: 1.52.0
Plugin: 2.0.0
SDK: 2.1.1

[medikoo]

@garethmcc @medikoo Yeah, no bueno.

@Wintereise Can you also try with the SLS_DEBUG=* and paste log? Also with which highest version exactly it works for you without issues, 1.49.0?

[Wintereise]

@Wintereise Can you also try with the SLS_DEBUG=* and paste log? Also with which highest version exactly it works for you without issues, 1.49.0?

1.49.0 works fine, yep. 1.50 has EMFILE issues, 1.51 has the same. 1.52 seems to succeed in packaging, but does nothing else.

As to pasting the log, I PMed it to you on the SLS Slack. Unfortunately, it doesn't say anything afterwards seemingly :(

[medikoo]

1.49.0 works fine, yep. 1.50 has EMFILE issues, 1.51 has the same. 1.52 seems to succeed in packaging, but does nothing else.

Ok, thank you! That clarifies that with EMFILE prevention we've fixed the issue just partially.
We'll look into it further

[hagaigold]

@medikoo i see another [maybe] issue which i think related to the packaging, but not sure if it related to that issue..

custom-resources.zip become very big because it "pack" inside all the project folder with his node_modules directory:

custom-resources.zip
  node_modules
    [project folder]
      .serverless
      node_modules
      ...

byw: someone else also notice this few hours ago - https://forum.serverless.com/t/custom-resources-zip-is-huge-node-js/9343

[medikoo]

@medikoo custom-resources.zip become very big because it "pack" inside all the project folder with his node_modules directory:

Indeed that definitely should not be the case. @pmuens this could be real source of this issue (?)

[mklueh]

Can we just add a .catch clause to

copyCustomResources(srcDirPath, tmpDirPath)
    .then(() => installDependencies(tmpDirPath))
    .then(() => createZipFile(tmpDirPath, zipFilePath))

in

https://github.com/serverless/serverless/blob/d48d2ee56d8d2d920289722804383511e0429f80/lib/plugins/aws/customResources/index.js

to see if and where it fails?

[medikoo]

to see if and where it fails?

If it would fail, the error would be propagated and exposed
(note it's return copyCustomResources(..)), issue is that operation hangs.

[bkmagnetron]

For me also the same issue but the what caused that seems to be different.
When I use the property existing: true for S3 event this issue happens. Otherwise it works fine.

[kziemski]

i get the same issue as well and 1.49.0 was the fallback.
windows 10 1.5# i utilize serverless-webpack and serverless-offline.
really wish i could update to 1.52.0 just saw lambda@edge support on 1.52

[ThomasEg]

I can confirm that it is resolved in 1.53.0...!

Thanks!

[naeemshaikh27]

I can still see this issue in1.53.0 while using existing: true for S3 event

[naeemshaikh27]

My bad, actually I hadn't updated the serverless framework version globally. npm uninstall -g serverless and then npm install -g serverless updated the framework to 1.53.0. The issue got resolved in this build

[uladkasach]

Still seeing this error with v1.53.0 :(

The build stops progressing at

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Installing dependencies for custom CloudFormation resources...

     Operating System:          linux
     Node Version:              10.16.3
     Framework Version:         1.53.0
     Plugin Version:            3.1.2
     SDK Version:               2.1.2
     Components Core Version:   1.1.1
     Components CLI Version:    1.2.3

here is the setup:

  doAwesomeThing:
    handler: dist/handlers/doAwesomeThing.handle
    events:
      - s3:
          bucket: bucket-name-${self:provider.stage}
          existing: true
          event: s3:ObjectCreated:*
          rules:
            - prefix: some_prefix/
            - suffix: .suffixed.csv

Any suggestions?

[medikoo]

@uladkasach above was Windows specific problem. What version of npm are you using? Also does it happen for you in all cases?

[uladkasach]

@medikoo ah, gotcha. Would you like me to open a separate issue?

I was using npm 6.12.0

Yeah, it happened in all cases: tested with sls v1.48, v.1.49, v1.50, v1.52, and v1.53. The only difference between a build succeeding and failing being the presence of that event trigger

Is there any other information needed that could help diagnose this?

[medikoo]

@medikoo ah, gotcha. Would you like me to open a separate issue?

Yes, it'll be good. It looks quite specific to your setup (maybe Node.js and npm version correlation). Do you observe same issue with npm v6.11.3 (it's one that's normally installed with Node.js v12) or 6.9.0 (one that goes with Node.js v10)?

[chadjvw]

I've reproduced this issue on Linux using yarn workspaces.

Best I can tell, executing a serverless command via yarn workspace <workspace name> package --stage.... causes the command to stop sometime after it prints out Serverless: Installing dependencies for custom CloudFormation resources....

I've found that changing this line from

.then(([cachedZipFilePath]) => fse.copyAsync(cachedZipFilePath, zipFilePath))

to

.then(([cachedZipFilePath]) => {fse.copyAsync(cachedZipFilePath, zipFilePath)})

fixes the issue (note the addition of {} around the copy command).

Why this fixes it I have no idea.

Root package.json

{
  "name": "root",
  "private": true,
  "version": "2.0.0",
  "workspaces": [
    "packages/*"
  ],
  "scripts": {
    "format": "prettier --config ./.prettierrc --ignore-path ./.prettierignore --write '**/*.{js,ts,json,yml,html,md}'",
    "type-check": "tsc -p ./tsconfig.json --noEmit --pretty false",
    "lint": "eslint -c ./.eslintrc packages/ --ext .ts,.js"
  },
  "devDependencies": {
    "@babel/cli": "^7.8.4",
    "@babel/core": "^7.8.4",
    "@babel/preset-env": "^7.8.4",
    "@babel/preset-typescript": "^7.8.3",
    "@types/aws-lambda": "^8.10.40",
    "@types/jest": "^25.1.1",
    "@types/jsonwebtoken": "^8.3.7",
    "@types/lodash": "^4.14.149",
    "@types/node": "^13.7.0",
    "@types/oracledb": "^4.2.0",
    "@types/superagent": "^4.1.5",
    "@types/uuid": "^3.4.7",
    "@typescript-eslint/eslint-plugin": "^2.19.0",
    "@typescript-eslint/parser": "^2.19.0",
    "babel-loader": "^8.0.6",
    "babel-plugin-lodash": "^3.3.4",
    "cache-loader": "^4.1.0",
    "eslint": "^6.8.0",
    "eslint-config-prettier": "^6.10.0",
    "eslint-plugin-import": "^2.20.1",
    "eslint-plugin-node": "^11.0.0",
    "eslint-plugin-prettier": "^3.1.2",
    "eslint-plugin-promise": "^4.2.1",
    "jest": "^25.1.0",
    "prettier": "^1.19.1",
    "serverless": "^1.62.0",
    "serverless-iam-roles-per-function": "^2.0.2",
    "serverless-plugin-epsagon": "1.6.2",
    "serverless-plugin-split-stacks": "^1.9.3",
    "serverless-plugin-stage-variables": "^1.9.5",
    "serverless-pseudo-parameters": "^2.5.0",
    "serverless-scriptable-plugin": "^1.0.5",
    "serverless-step-functions": "^2.17.0",
    "serverless-webpack": "^5.3.1",
    "terser-webpack-plugin": "^2.3.4",
    "ts-jest": "^25.1.0",
    "ts-loader": "^6.2.1",
    "tsconfig-paths-webpack-plugin": "^3.2.0",
    "typescript": "^3.7.5",
    "webpack": "^4.41.5"
  }
}

Workspace package.json

{
  "name": "@cloud/api",
  "version": "2.0.0",
  "private": true,
  "scripts": {
    "deploy": "serverless deploy",
    "package": "serverless package"
  },
  "dependencies": {
    "@cloud/base": "^2.0.0",
    "aws-sdk": "^2.591.0",
    "jsonwebtoken": "^8.5.1",
    "lodash": "^4.17.15",
    "spacetime": "^6.3.0",
    "superagent": "5.1.2"
  },
  "devDependencies": {
    "serverless-aws-documentation": "^1.1.0"
  }
}

Serverless Environment Info

  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              12.14.1
     Framework Version:         1.62.0
     Plugin Version:            3.3.0
     SDK Version:               2.3.0
     Components Core Version:   1.1.2
     Components CLI Version:    1.4.0