Serverless-next.js: serverless remove Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

[damo78]

Describe the bug

serverless remove in my CI/CD pipeline results in this error:

Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

Actual behavior

The remove job results in:

459$ serverless remove
460  0s › Components › Running .
461aNA
462  0s › Template › Removing .
463aNA
464  error:
465  Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'
466Require stack:
467- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js
468- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js
469- /usr/local/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js
470- /usr/local/lib/node_modules/serverless/bin/serverless.js
471    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
472    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
473    at Module.require (internal/modules/cjs/loader.js:952:19)
474    at require (internal/modules/cjs/helpers.js:88:18)
475    at Template.load (/usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js:108:24)
476    at async fn (/usr/local/lib/node_modules/serverless/node_modules/@serverless/template/utils.js:309:27)
477    at async Promise.all (index 0)
478    at async syncState (/usr/local/lib/node_modules/serverless/node_modules/@serverless/template/utils.js:333:3)
479    at async Template.remove (/usr/local/lib/node_modules/serverless/node_modules/@serverless/template/serverless.js:78:5)
480    at async Object.runComponents (/usr/local/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js:218:17) {
481  code: 'MODULE_NOT_FOUND',
482  requireStack: [
483    '/usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js',
484    '/usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js',
485    '/usr/local/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js',
486    '/usr/local/lib/node_modules/serverless/bin/serverless.js'
487  ]
488}
489  0s › Template › Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'
490Require stack:
491- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js
492- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js
493- /usr/local/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js
494- /usr/local/lib/node_modules/serverless/bin/serverless.js
495
Cleaning up file based variables
00:01
496ERROR: Job failed: exit code 1

Expected behavior

I can run the exact code locally with sls and sls remove and it provisions and removes as expected.

The sls command correctly provisions in the initial job, but the subsequent remove job fails as per the logs above.

Steps to reproduce

.gitlab-ci-yml

image: node:latest

before_script:
  - npm config set prefix /usr/local
  - npm install -g serverless
  - npm install
  - apt-get update && apt-get install -y python3 python3-pip  python3-setuptools groff less && pip3 install --upgrade pip && apt-get clean
  - pip3 --no-cache-dir install --upgrade awscli

stages:
  - Deploy
  - Remove

Deploy Staging:
  stage: Deploy
  environment: staging
  variables:
    SLS_NEXT_S3_STATE_BUCKET: my-state-bucket-staging
  script:
    - echo "Deploy Staging"
    - aws s3 sync s3://$SLS_NEXT_S3_STATE_BUCKET/.serverless .serverless --delete
    - serverless
    - aws s3 sync .serverless s3://$SLS_NEXT_S3_STATE_BUCKET/.serverless --delete
  rules:
    - if: $CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE != "schedule"

Remove Staging:
  stage: Remove
  environment: staging
  variables:
    SLS_NEXT_S3_STATE_BUCKET: my-state-bucket-staging
  script:
    - echo "Remove Staging"
    - aws s3 sync s3://$SLS_NEXT_S3_STATE_BUCKET/.serverless .serverless --delete
    - serverless remove
    - aws s3 rm s3://$SLS_NEXT_S3_STATE_BUCKET --recursive
  rules:
    - if: $CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE != "schedule"
      when: manual

serverless.yml (Note: I've removed the env variables below from the gitlab-ci.yml above for simplification

org: ${env.SLS_NEXT_ORG}
app: ${env.SLS_NEXT_APP}
name: ${env.SLS_NEXT_NAME}
stage: ${env.SLS_NEXT_STAGE}
domain: ${env.SLS_NEXT_DOMAIN}
fullDomain: '${subdomainStage.${stage}}${domain}'
subdomainStage:
  prod: ''
  staging: staging.
  dev: dev.
distribution: ${env.SLS_NEXT_DISTRIBUTION_MAIN}
bucketName: ${env.SLS_NEXT_BUCKET_NAME}

next-sls-app:
  component: '@sls-next/[email protected]'
  inputs:
    cloudfront:
      distributionId: ${distribution}
      priceClass: 'PriceClass_100'
      aliases: ['${fullDomain}']
    bucketName: ${bucketName}
    bucketRegion: 'eu-west-1'
    description: 'Lambda@Edge for ${org}-${app}-${name}-${stage}'
    name:
      defaultLambda: ${org}-${app}-${name}-${stage}-default
      apiLambda: ${org}-${app}-${name}-${stage}-api
    runtime:
      defaultLambda: 'nodejs12.x'
      apiLambda: 'nodejs12.x'

package.json (Note: I've tried all combinations of serverless being installed as dependency, and globally as per gitlab-ci.yml above, so here is an example of a local dependency)

{
  "name": "app",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "mysql": "^2.18.1",
    "next": "9.5.4",
    "next-auth": "^3.1.0",
    "react": "16.13.1",
    "react-dom": "16.13.1"
  },
  "devDependencies": {
    "serverless": "^2.8.0"
  }
}

Screenshots/Code/Logs

As above

Versions

• OS/Environment: node:alpine Gitlab-CI runner
• @sls-next/serverless-component version: 1.18.0-alpha.18
• Next.js version: n/a

Checklist

I have pinned the version in my serverless.yml, as per the README and I've reviewed all of the issues similar that I could find - most of which had issues with the deploy sls and not pinning versions. This one is similar, but definitely different in that I'm attempting to use sls remove and this is where the failure occurs. This issue existed in version 1.18.0-alpha.12 also; I upgraded and tested locally and through GitLab-CI before posting this, and the same results.

Also, I've reviewed the .serverless templates generated for the local environment and the s3 bucket state environment and they look the same (apart from stage and aws resources). The Template.json is as follows:

{
"components": {
"next-sls-app": "/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component"
}
}

It seems this is what is being looked at by the GitLab CI runner and cannot find it in the environment

[stan-sack]

im seeing this as well. i was on 1.17 which just broke randomly. tried 1.18 alpha and getting the same error. also downgraded to 1.16 and getting the same error as well

[damo78]

I’m now seeing this even on my deploy stage too. I have gone back as far as alpha.12 and I’m getting all sorts of similar errors

I’ve even reverted my original working master branch and that has stopped working

[dphang]

Sorry, I am not as familiar with the remove command but this is strange. Just for sanity, checked that @1.18.0-alpha.18 is published correctly to npm and I am able to install locally.

Could you please check what is the contents of root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/?

This line seems strange:

Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

I don't know why it is trying to load a nested @sls-next/serverless-component, it should loaded at the top level from dist/component.js. I checked /root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/ node and it should only be aws-cloudfront, aws-lambda, cloudfront, domain, lambda-at-edge, next-aws-cloudfront, s3-static-assets (those are internal dependencies).

I wonder if this could be issue with serverless components itself, maybe a recent change broke how serverless components are resolved (and this being on the serverless components beta may not help either). If you are using CI and serverless version is not pinned, since a CI workflow normally uses a clean environment, it would usually download latest version.

@danielcondemarin would you have any ideas?

[bhall2001]

My experience is the remove command does not remove the CloudFront distribution or lambda function(s). Those need to be removed manually. Here's the steps I use to remove a serverless-nextjs deployed site.

1. serverless remove from command line.

2. log in to aws console

3. navigate to CloudFront and disable the Distribution for the deployed serverless-nextjs component (be sure to select the correct distribution if you have many).

4. wait about 5 minutes

5. delete the CloudFront distribution for the deployed serverless-nextjs component (again make sure because there's no going back if you delete an incorrect distribution)

6. wait about 15 minutes. During this time your CloudFront distribution is deleted from edge locations

7. Now you can delete the Lambda function(s) created by the serverless-nextjs component. NOTE: If you get an error that the lambda function can not be deleted, you need to wait longer. The time does vary to delete all the CloudFront distributions. I find 15 minutes is an average time. There may be 1 or 2 lambda's -- one for content, one for api.

[stan-sack]

Just to clarify my earlier comment. I'm seeing this when i run serverless, not serverless remove

[dphang]

I see, I did test with the latest serverless versions (e.g 2.8.0 and 2.7.0) and it seems fine for both commands, both locally and on GitHub actions. Unfortunately I don't use GitLab so I can't help too much there, if someone else uses maybe they can help.

Though as mentioned this line is still strange either way:

Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

Not sure why it is trying to resolve the path like this, as the end @sls-next/serverless-component doesn't exist for me locally - the component is at top level. Maybe Serverless made an update that's breaking the resolution in certain environments? Perhaps you can try to deploy using an older serverless version. You can do something like:

npx [email protected]

See versions here: https://www.npmjs.com/package/serverless

[damo78]

Unfortunately I can’t do any further tests in the remove command at the moment because the deploy is not working anymore either, for any version of serverless or serverless next

@bhall2001 mentioned that the repo has moved? I’ll try again tomorrow to do some debugging to provide any useful info

[dphang]

Ok, thanks @damo78. I guess I was a bit confused - I thought the issue @bhall2001 mentioned is separate but semi-related issue - it's known the remove command may not work well because the CloudFront distribution takes a long time to disable and delete (this is an AWS bottleneck).

Also, the latest component to use is @sls-next/serverless-component, so I think you have it right there.

Sorry I was mistaken earlier, that @sls-next/serverless-component should exist in the component stored. I was previously looking at npm installation but the Serverless component installation is slightly different: E.g for my local deployment, in Template.json it is here:

"/Users/danielphang/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component"

So I suspect the path here got changed in GitLab, not sure what it would be since I don't use GitLab:

/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

If you can figure what it should be and update it in Template.json, it might help?

[stan-sack]

I was able to fix this by manually creating the directory before running serverless with: mkdir -p /root/.serverless/components/registry/npm/@sls-next/[email protected]. This obviously isn't a great solution but it should unblock everyone.

The issue seems to be that serverless doesnt create the directory before it tries to install the component and npm install <package> --prefix <dir> fails if <dir> doesnt exist. I think this is a serverless bug.

[wintvelt]

Thank you for this thread. I was hit by a (probably) related issue.
Since yesterday, I could not deploy anymore through CircleCI.
Deploy from local machine worked just fine, and I was lost why.

For those running into simular problems: I was able to fix by

1. Copying the directory from the CircleCI build log where it failed.
   In my case /home/circleci/.serverless/components/registry/npm/@sls-next/[email protected].

2. Adding a step in my CircleCI script just before npx serverless deploy step, that does
   mkdir - p [directory here].

This fixed the deployment through CI/CD for me.

[damo78]

Thanks everybody for their support on this. I will try out the manual directory creation in the pipeline as temporary fix.

To try answer some of the questions above, here is everything I've tried. Firstly it's worth noting that the deploy stage within GitLab was fully working up until last week on versions of serverless 2.6.0, 2.7.0 and 2.8.0, and also the serverless component 1.18.0-alpha.12, -alpha.16,-alpha.17 and -alpha.18. It was only the remove that wasn't working.

Then something changed and all combinations of the above whether installing serverless globally or locally to the GitLab directory have stopped working with the above "registry" directory error, including using npx. I went down a strand of thought thinking my node_modules caching was causing the problem, but cleared these and still it wasn't working. I then rebuilt the entire project starting from a new repo, various tests on the many versions again and still no luck.

Locally on my Mac it works as expected.

The deploy error on GitLab looks as described by others above, which is pretty much the same as I was getting at the remove stage prior to this happening:

48  $ serverless
49  0s › Components › Running .
50aNA
51  0s › Template › Deploying .
52aNA
53  0s › Template › Deploying ..
54aNA
55  0s › Template › Deploying ..
56aNA
57  0s › Template › Deploying ...
58aNA
59  0s › Template › Deploying ...
60aNA
61  0s › Template › Deploying 
62aNA
63  0s › Template › Deploying 
64aNA
65  0s › Template › Deploying 
66aNA
67  0s › Template › Deploying .
68aNA
69  error:
70  Error: Command failed: npm install @sls-next/[email protected] --prefix /root/.serverless/components/registry/npm/@sls-next/[email protected]
71npm ERR! code ENOENT
72npm ERR! syscall lstat
73npm ERR! path /root/.serverless/components/registry/npm/@sls-next
74npm ERR! errno -2
75npm ERR! enoent ENOENT: no such file or directory, lstat '/root/.serverless/components/registry/npm/@sls-next'
76npm ERR! enoent This is related to npm not being able to find a file.
77npm ERR! enoent 
78npm ERR! A complete log of this run can be found in:
79npm ERR!     /root/.npm/_logs/2020-10-23T21_48_45_484Z-debug.log
80    at ChildProcess.exithandler (node:child_process:309:12)
81    at ChildProcess.emit (node:events:327:20)
82    at maybeClose (node:internal/child_process:1048:16)
83    at Process.ChildProcess._handle.onexit (node:internal/child_process:288:5) {
84  killed: false,
85  code: 254,
86  signal: null,
87  cmd: 'npm install @sls-next/[email protected] --prefix /root/.serverless/components/registry/npm/@sls-next/[email protected]',
88  stdout: '',
89  stderr: 'npm ERR! code ENOENT\n' +
90    'npm ERR! syscall lstat\n' +
91    'npm ERR! path /root/.serverless/components/registry/npm/@sls-next\n' +
92    'npm ERR! errno -2\n' +
93    "npm ERR! enoent ENOENT: no such file or directory, lstat '/root/.serverless/components/registry/npm/@sls-next'\n" +
94    'npm ERR! enoent This is related to npm not being able to find a file.\n' +
95    'npm ERR! enoent \n' +
96    '\n' +
97    'npm ERR! A complete log of this run can be found in:\n' +
98    'npm ERR!     /root/.npm/_logs/2020-10-23T21_48_45_484Z-debug.log\n'
99}
100  0s › Template › Error: Command failed: npm install @sls-next/[email protected] --prefix /root/.serverless/components/registry/npm/@sls-next/[email protected]
101npm ERR! code ENOENT
102npm ERR! syscall lstat
103npm ERR! path /root/.serverless/components/registry/npm/@sls-next
104npm ERR! errno -2
105npm ERR! enoent ENOENT: no such file or directory, lstat '/root/.serverless/components/registry/npm/@sls-next'
106npm ERR! enoent This is related to npm not being able to find a file.
107npm ERR! enoent 
108npm ERR! A complete log of this run can be found in:
109npm ERR!     /root/.npm/_logs/2020-10-23T21_48_45_484Z-debug.log
111
Cleaning up file based variables
00:00
113ERROR: Job failed: exit code 1

[damo78]

So I suspect the path here got changed in GitLab, not sure what it would be since I don't use GitLab:

/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'

If you can figure what it should be and update it in Template.json, it might help?

Thanks for your help @dphang. I had checked the Template.json prior and I get a similar local and on pipeline dir path. The /root/.serverless.... is the GitLab local runner user as you'd find locally on your own machines with /Users/Daniel/etc...

My state was being sent/received from an S3 bucket from the GitLab runner and was also working fine thanks to following @bhall2001 's guide. But as described above, something went pop towards the end of last week

[wintvelt]

@damo78 I do not use GitLab (CircleCI instead), but the error surely seems like the one I am getting.
Also the behavior is same as mine: deploy from Local works fine, deploy through CI/CD fails.

What you could try: Add a step in your .gitlab-ci-yml.
My guess woulld be in the before_script, just after - npm install

That says:
- mkdir -p /root/.serverless/components/registry/npm/@sls-next/[email protected]

Which is the (missing) directory that cause the fail.

For me this solved it. Hope you have similar luck!

[damo78]

The temporary workaround does work for me so many thanks for getting me past this blocker. Obviously hard-coding this into the pipeline is not great but at least I can move on to the other elements I want to test whilst this gets looked into.

373   Deploy staging
374$ mkdir -p /root/.serverless/components/registry/npm/@sls-next/[email protected]
375$ aws s3 sync s3://$SLS_NEXT_S3_STATE_BUCKET/.serverless .serverless --delete
376$ serverless
377  0s › Components › Running .
378aNA
379  0s › Template › Deploying .
380aNA
381  0s › Template › Deploying ..
382aNA
383  0s › Template › Deploying ..
384aNA
385  0s › Template › Deploying ...
...
2865  next-sls-app: 
2866    appUrl:         https://randomString-blah.cloudfront.net
2867    bucketName:     my-bucket-origin
2868    distributionId: E2LH300EXAMPLE
2869  126s › next-sls-app › done
2870$ aws s3 sync .serverless s3://$SLS_NEXT_S3_STATE_BUCKET/.serverless --delete

I thought I'd see if this solution would work for the remove stage, i.e. creating the "registry" path. My setup uses and existing terraform CloudFront distro and S3 origin bucket, and locally I know I have to delete the S3 contents first otherwise serverless remove doesn't work. It also disables the CloudFront distro, which is quite good rather than deleting it so I can use my terraform to re-enable and keep the same identifiers.

Unfortunately, it didn't work for `remove with the same errors as before. I suspected as such given it was a cannot find module error, rather than directory missing problem.

415$ serverless remove
416  0s › Components › Running .
417aNA
418  0s › Template › Removing .
419aNA
420  error:
421  Error: Cannot find module '/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'
422Require stack:
423- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js
424- /usr/local/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js
425- /usr/local/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js
426- /usr/local/lib/node_modules/serverless/bin/serverless.js
...

It's weird how this works locally for me, but not on GitLab's runners. For now, I'll use the manual removal.

Do I need to raise a separate issue for the deploy problem vs. this remove problem? As they've been intertwined and are two very different problems?

[hugomn]

I'm facing the same issue when trying to remove a deployment using Circleci. I'm storing the state in S3, deploy works fine, but then when I try to remove it, I get this error:

mkdir -p /home/circleci/.serverless/components/registry/npm/@sls-next/[email protected]
serverless remove


  0s › Components › Running .
nfinityA
  0s › Template › Removing .
nfinityA
  error:
  Error: Cannot find module '/home/circleci/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'
Require stack:
- /snapshot/serverless/node_modules/@serverless/core/src/Component.js
- /snapshot/serverless/node_modules/@serverless/core/src/index.js
- /snapshot/serverless/node_modules/@serverless/cli/src/index.js
- /snapshot/serverless/bin/serverless.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:964:15)
    at Function.Module._resolveFilename (pkg/prelude/bootstrap.js:1346:46)
    at Function.Module._load (internal/modules/cjs/loader.js:840:27)
    at Module.require (internal/modules/cjs/loader.js:1024:19)
    at Module.require (pkg/prelude/bootstrap.js:1225:31)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Template.load (/snapshot/serverless/node_modules/@serverless/core/src/Component.js:108:24)
    at async fn (/snapshot/serverless/node_modules/@serverless/template/utils.js:309:27)
    at async Promise.all (index 0)
    at async syncState (/snapshot/serverless/node_modules/@serverless/template/utils.js:333:3) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/snapshot/serverless/node_modules/@serverless/core/src/Component.js',
    '/snapshot/serverless/node_modules/@serverless/core/src/index.js',

As you can see, I even tried to solution @wintvelt suggested, the folder is created successfully, but it still fails. It seems like serverless is trying to find the component package and can't find it.

I'm I missing something here?

[Meemaw]

Seeing this issue as well. This seems to be related https://github.com/serverless-nextjs/serverless-next.js/issues/504? cc @dphang @danielcondemarin

[dphang]

Honestly, I am not quite sure on this as I couldn't reproduce myself locally - so I still think it may be an issue with the serverless framework, maybe a new version from serverless v2 resolves the component npm package differently. If you are deploying in CI, running npx serverless will always use the latest version, since CI generally provisions a new container for each job run unless you are using self hosted CI.

We did not change anything related to component resolution recently.

I would suggest the following:

1. Pin the serverless cli version to one of the v1 versions before this started happening (e.g npx serverless@version)
2. Hopefully the .serverless state directory is not corrupted due to use of serverless cli v2. If it is, you may need to manually recreate, e.g do a dummy deployment using serverless v1, then copy your CF distribution/lambda/s3 into the state.

Sorry for the trouble - I think we are exploring other more flexible IaC options like CDK or CDK for Terraform as serverless generally doesn't manage state as well and anyway we have a lot of custom build/deploy logic we are doing ourselves. Hopefully this will work much better in the future.

[hugomn]

@dphang thanks for your reply! Even trying with old versions of serverless, nothing changed. One weird thing I've noticed is if I run serverless before running serverless remove it works fine. It seems like only syncing the .serverless folder from s3 and running serverless remove is not enough.

This doesn't work:

aws s3 sync s3://my-bucket/<< pipeline.parameters.environment >>/.serverless .serverless --delete
npx [email protected] remove

But this does:

aws s3 sync s3://my-bucket/<< pipeline.parameters.environment >>/.serverless .serverless --delete
npx [email protected]
npx [email protected] remove

And it fails with the following:

error:
  Error: Cannot find module '/home/circleci/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component'
Require stack:
- /home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js
- /home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js
- /home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js
- /home/circleci/.npm/_npx/154/lib/node_modules/serverless/bin/serverless.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:893:15)
    at Function.Module._load (internal/modules/cjs/loader.js:743:27)
    at Module.require (internal/modules/cjs/loader.js:965:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Template.load (/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js:108:24)
    at async fn (/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/template/utils.js:309:27)
    at async Promise.all (index 0)
    at async syncState (/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/template/utils.js:333:3)
    at async Template.remove (/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/template/serverless.js:78:5)
    at async Object.runComponents (/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js:218:17) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/core/src/Component.js',
    '/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/core/src/index.js',
    '/home/circleci/.npm/_npx/154/lib/node_modules/serverless/node_modules/@serverless/cli/src/index.js',
    '/home/circleci/.npm/_npx/154/lib/node_modules/serverless/bin/serverless.js'

Any idea? I'll keep it deploying before removing, but it's definitely not ideal. Thanks!

[dphang]

Unfortunately I don't use the remove command myself much, so not sure what's going on. @danielcondemarin maybe you know how the serverless component resolution works?

[Meemaw]

From my testing the problem is that npx serverless generates/installs components and injects paths into Template.json:

"components": {
    "app": "/home/runner/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/@sls-next/serverless-component"
}

When you are in the CI and you want to cleanup your resources in a different job (e.g. on PR merged/closed) you just want to run npx serverless remove, but this will fail because the components are not being installed before that (this is not an issue if you want to remove them at the end of the same CI job, e.g. after running E2E tests).

Workaround is to call npx serverless again that will install the components followed by npx serverless remove. I would expect npx serverless remove to have the same logic of installing the components/packages if required.

[dphang]

Ah, thanks for that, I guess that makes sense now.

Perhaps another workaround is to cache that directory in Template.json somewhere and then load it before calling serverless remove.

@danielcondemarin I'm not sure if there's a quick fix as it seems the problem is inherent to Serverless. We are planning to explore other proper IaC options, so we may not want to invest too much into fixing the state management issues.

[Meemaw]

I've tried syncing that folder to S3 as it is done for .serverless but it contains 1000+ (node_modules,...) files and is just takes too long so that is not really feasible.

[damo78]

Perhaps another workaround is to cache that directory in Template.json somewhere and then load it before calling serverless remove.

I've tried many combinations of caching (and s3 syncing) with GitLab CI:

• .serverless (from deploy stage to remove, with and without s3 state syncing)
• global and local npm packages, with and without using npx

All result in the same error when trying remove. Are you suggesting another caching option to try?

@danielcondemarin I'm not sure if there's a quick fix as it seems the problem is inherent to Serverless. We are planning to explore other proper IaC options, so we may not want to invest too much into fixing the state management issues.

I haven't wanted to write this, but I tend to agree with how much investment should be given to IaC and state-management in general should be given. I keep reading, and have followed much of the advice, to use terraform (or CDK) to provision and maintain infrastructure. Whilst almighty efforts have been given to try and work around serverless, serverless components general and this serverless next componentry, it feels like there's a combination of reinventing the wheel whilst also trying to merge multiple moving parts.

Before this I was looking to try this; next-aws-lambda-webpack-plugin, to transform the referenced CloudFormation/SAM spec into terraform. But I thought I'd gain the benefits of the serverless framework by using this serverless next component. Having researched further, and especially when I understood that serverless components weren't really the serverless framework as such anymore, I'm now not sure what benefits there are for infrastructure provisioning and maintenance left that isn't better suited elsewhere.

Personally I'd rather this bug went stale if I knew that in the near future I'd be able to utilise terraform (or CDK) end-to-end for IaC provisioning into AWS for Next JS. The semi-manual removal stage is no big deal for me at the moment. Workflow for using terraform with serverless infrastructure is already straight-forward enough to keep them separated and sharing, e.g. using SSM Param Store, or CI/CD pipeline outputs. Trying to wedge something else in-between has taken weeks' worth of effort that I feel could be much simpler architecturally.

Please, please do not take this as anything other than me agreeing with your conclusion on time investment for this bug @dphang. You are all doing outstanding work and I'll be continuing with this componentry for this current project to see how it pans out.

[damo78]

For others who run into this issue and are provisioning their CloudFront and S3 origin buckets via terraform, then you can still utilise a useful removal stage. What I found is that even though I didn't provision versioning on my S3 bucket, serverless creates multiple versions over time which means I could not easily use terraform to remove the infrastructure due to the bucket not being empty, and the simple --recursive deletion doesn't work.

I found this script useful in my serverless 'removal' stage which then allowed me to use my terraform removal stage to complete the infrastructure deletion.

echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket $SLS_NEXT_BUCKET_NAME | grep -E "^VERSIONS" | awk '{print "aws s3api delete-object --bucket $SLS_NEXT_BUCKET_NAME --key "$4" --version-id "$8";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh; echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket $SLS_NEXT_BUCKET_NAME | grep -E "^DELETEMARKERS" | grep -v "null" | awk '{print "aws s3api delete-object --bucket $SLS_NEXT_BUCKET_NAME --key "$3" --version-id "$5";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;

[dphang]

@damo78 thanks for your thoughts and feedback. For caching I thought something like this would work (example for GitHub Actions): https://github.com/actions/cache instead of syncing to S3. It zips it the whole directory up and stores it with GitHub, so it's faster than trying to sync many files individually to S3 or zipping it up yourself. Sorry to hear if you have already tried and it did not work.

RE: IaC, yea, I feel that Serverless in general is not really good for complex infra, but if you have simple Lambdas and configuration, it works fine. For example, I use regular Serverless Framework + a couple of plugins for a test API endpoints. After all I think originally it was used since there was just a few parts (e.g single Lambda with the original plugin), and then moved to Serverless Component. Then we also imported domain and aws-cloudfront from Serverless Components and made our own updates to those because the originals weren't maintained anymore.

Infrastructure state management is not as easy as it first appears to be, and we are kinda reinventing the wheel here by managing it ourselves via the AWS SDK (since Serverless components don't provide too much outside of Lambda functions), whereas proper IaC like Terraform/CDK would have already solved that plus probably gives way more flexibility (e.g Terraform allows you to manage state on S3, Terraform, GCP, etc.). It has given me trouble especially when managing certificates, Route53 records. Plus, anytime there are new features e.g on CloudFront we need to implement those additional configuration changes - for example for Brotli support, it needs to use the new caching policy, but we are still using the legacy one. Something like Terraform would already be quickly updated by the community. Plus, it will make it easier to support and manage more platforms besides AWS Lambda@Edge.

I think @danielcondemarin and I are looking to prototype CDK for Terraform as we can leverage TypeScript + all Terraform modules, probably providing the vast Terraform ecosystem + allowing more control by actually writing code to update configuration (vs. static configuration files from Terraform). I personally think it may be quite suited for this component since it needs to programmatically update infrastructure on each deploy - at the very least, uploading Lambda, S3 assets and updating the CloudFront Lambda function associations.

[damo78]

That would be awesome @dphang. I'd not heard that terraform had a CDK until now, so I've spent all night looking into this, as well as Pulumi. It seems the TF CDK is quite some way away from primetime, whereas Pulumi is very mature with a migration path from TF. However, given the size of the TF community and business adoption of TF so far, I suspect the TF CDK is probably going to come out on top. Good luck and thanks again!

[dphang]

@damo78, thanks, yeah it's relatively new (I think released earlier this year?) so we should keep it in mind and also consider Pulumi.