Aws-sam-cli: NODE_PATH is not set in the Container when in debug mode

[guijob]

Description

I was using sam 0.13 and I could debug my function using VS Code with no problems, after upgrading my lambdas do nodejs10.x (coming from 8.10) I had to update my sam version due compatbility, and now I no longer can debug my code, either in Ubuntu or macOS environment. I'm getting following error:

{
  "errorType": "Runtime.ImportModuleError",
  "errorMessage": "Error: Cannot find module 'aws-sdk'"
}

Does this package shouldn't be inside container?

Additional point:

• It runs normally without debbuging mode (without -d 5858).
• Steps provided used to work using SAM 0.13 with node8.10 Runtime.

EDIT: I've just tried using SAM 0.17.0 with node8.10 Runtime and it worked. But ,using node10.x it doesn't.

Steps to reproduce

project structure

node_modules/
template.yml
webpack.config.js
src/
-- lambdas/
---- records/
------ index.js

template.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: ["AWS::Serverless-2016-10-31", "CorsFixer"]
Description: >
  auditing-back-app

Globals:
  Function:
    Runtime: nodejs10.x
    Timeout: 60
    MemorySize: 128
  Api:
    Cors:
      AllowOrigin: "'*'" 
      AllowHeaders: "'Content-Type,Authorization'" 

Parameters:
  Environment:
    Type: String

  RecordsEndpointsFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: !Sub ${Environment}
      CodeUri: dist/records/records.zip
      Handler: index.handler
      MemorySize: 1536
      Timeout: 500
      Policies:
        - AWSLambdaBasicExecutionRole
        - AmazonDynamoDBFullAccess
        - AmazonS3FullAccess
        - AmazonSESFullAccess
        - AmazonSQSFullAccess
      Events:
        UserEndpoints:
          Type: Api
          Properties:
            Path: /users/{userId}/parkinglots/{parkingLotId}/records
            Method: any
            RestApiId: !Ref HttpApiGateway
        NewFileUploaded:
          Type: SQS
          Properties:
            Queue: !Sub arn:aws:sqs:${AWS::Region}:{hidden}:auditing-new-file-to-process-${Environment}
            BatchSize: 1
        RealTimeUpdater:
          Type: Schedule
          Properties:
            Schedule: rate(5 minutes)

webpack.config.js

const glob = require('glob');
const path = require('path');
const ZipPlugin = require('zip-webpack-plugin');

module.exports = (env, argv) => {
  console.log(`Buiding in ${argv.mode} mode`);

  const functionsToBuild = glob
    .sync('./src/lambdas/*/index.*')
    .map(x => x.match(/(?<=src\/lambdas\/).*/)[0]);

  return functionsToBuild.map(fxn => {
    const lambdaName = fxn.split('/')[0];
    const triggerName = fxn.split('/')[1];

    return {
      entry: path.join(__dirname, 'src', 'lambdas', lambdaName, triggerName),
      devtool: 'cheap-module-eval-source-map',
      output: {
        filename: 'index.js',
        path: path.join(__dirname, 'dist', lambdaName),
        libraryTarget: 'commonjs2'
      },
      module: {
        rules: [
          {
            test: /\.ts(x?)$/,
            use: [
              'ts-loader'
            ]
          },
        ]
      },
      resolve: {
        extensions: ['.ts', '.js']
      },
      optimization: {
        minimize: false,
        namedModules: true
      },
      plugins: [
        new ZipPlugin({
          path: path.join(__dirname, 'dist', lambdaName),
          pathPrefix: '',
          filename: `${lambdaName}.zip`
        })
      ],
      externals: {
        'aws-sdk': 'aws-sdk',
        'awslambda': 'awslambda',
        'dynamodb-doc': 'dynamodb-doc',
        'imagemagick': 'imagemagick'
      },
      target: 'node',
      node: {
        __filename: false,
        __dirname: false
      },
      stats: 'errors-only',
      bail: true,
    }
  });
}

package.json

{
  "name": "serverless-auditing-back",
  "version": "1.0.0",
  "description": "This is a sample template for sam-app - Below is a brief explanation of what we have generated for you:",
  "main": "index.js",
  "scripts": {
    "install": "find ./node_modules/* -mtime +10950 -exec touch {} \\;",
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "node_modules/.bin/webpack --config webpack.config.js --mode=development",
    "build-prod": "node_modules/.bin/webpack --config webpack.config.js --mode=production"
  },
  "keywords": [],
  "private": true,
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "glob": "^7.1.4",
    "terser-webpack-plugin": "^1.3.0",
    "webpack": "^4.35.0",
    "webpack-cli": "^3.3.5",
    "zip-webpack-plugin": "^3.0.0"
  },
  "dependencies": {
    "@types/node": "^12.0.10",
    "axios": "^0.19.0",
    "crypto": "^1.0.1",
    "jsonwebtoken": "^8.5.1",
    "moment": "^2.24.0",
    "moment-timezone": "^0.5.25",
    "sinesp-api": "^1.4.0",
    "ts-loader": "^6.0.4",
    "typescript": "^3.5.2",
    "uuid": "^3.3.2"
  }
}

launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Attach to SAM CLI",
      "type": "node",
      "request": "attach",
      "address": "localhost",
      "port": 5858,
      // From the sam init example, it would be "${workspaceRoot}/hello_world"
      "localRoot": "${workspaceRoot}/",
      "remoteRoot": "/var/task",
      "protocol": "inspector",
      "stopOnEntry": false
    }
  ]
}

commands:

npm run build
sam local invoke -d 5858 RecordsEndpointsFunction 

Observed result

Fetching lambci/lambda:nodejs10.x Docker container image......
2019-06-28 11:13:01 Mounting /tmp/tmpoVDOPI as /var/task:ro,delegated inside runtime container
2019-06-28 11:13:01 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 201
2019-06-28 11:13:01 http://localhost:None "GET /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/json HTTP/1.1" 200 None
2019-06-28 11:13:01 http://localhost:None "GET /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/json HTTP/1.1" 200 None
2019-06-28 11:13:02 http://localhost:None "POST /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/start HTTP/1.1" 204 0
2019-06-28 11:13:02 Setting up SIGTERM interrupt handler
2019-06-28 11:13:02 http://localhost:None "GET /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/json HTTP/1.1" 200 None
2019-06-28 11:13:02 http://localhost:None "POST /containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/attach?stream=1&stdin=0&logs=1&stderr=1&stdout=1 HTTP/1.1" 101 0
Debugger listening on ws://0.0.0.0:5858/6f13348c-d0c6-479b-92d8-7076ac53cd7c
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
2019-06-28T14:13:13.320Z undefined ERROR Uncaught Exception {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'aws-sdk'","stack":["Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'"," at _loadUserApp (/var/runtime/UserFunction.js:100:13)"," at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)"," at Object. (/var/runtime/index.js:36:30)"," at Module._compile (internal/modules/cjs/loader.js:698:14)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)"," at Module.load (internal/modules/cjs/loader.js:600:32)"," at tryModuleLoad (internal/modules/cjs/loader.js:539:12)"," at Function.Module._load (internal/modules/cjs/loader.js:531:3)"," at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)"," at startup (internal/bootstrap/node.js:283:19)"]}
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Init Duration: 10554.03 ms Duration: 0.00 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 86 MB
{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-sdk'"
}
2019-06-28 11:13:13 http://localhost:None "GET /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d/json HTTP/1.1" 200 None
2019-06-28 11:13:13 http://localhost:None "DELETE /v1.35/containers/0c56b87b7a0f7ee5597371a544ffc652a9836bed4eb31d8b6a34c469aeef885d?force=True&link=False&v=False HTTP/1.1" 204 0

Expected result

Run function normally

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Tried in Ubuntu 18.04.2 LTS and macOS Mojave 10.14.5
2. sam --version: SAM CLI, version 0.17.0

[sriram-mv]

does including the package in your package.json help? I'm wondering if other execution environments have them by default, whereas 10.x doesnt.

[guijob]

@TheSriram, yes it does. It works when installing directly in my project using npm install --save aws-sdk and removing aws-sdk from externals object in webpack.config.js. But it made building process become too slow and heavy, as I had to even increase node memory limit. In fact, my project is more complex than example provided.

[awood45]

This may need a deeper investigation - I am able to see that the nodejs10.x runtime DOES include the aws-sdk package, so the debugging issue may be something else.

[davistardif]

Looks to me like the NODE_PATH env variable isn't getting set when using debug mode. Normally, the entrypoint is ENTRYPOINT ["/var/rapid/init", "--bootstrap", "/var/runtime/bootstrap"] from the node 10 docker container and
/var/runtime/bootstrap runs

if [ -z "$NODE_PATH" ];
then
  nodejs_mods="/opt/nodejs/node_modules"
  nodejs10_mods="/opt/nodejs/node10/node_modules"
  runtime_mods="/var/runtime/node_modules"
  export NODE_PATH="$nodejs10_mods:$nodejs_mods:$runtime_mods"
fi

but in debug mode the entrypoint is modified and the above never gets run. This might be why the runtime can't find aws-sdk which (I believe) is located in /var/runtime/node_modules

@awood45 thoughts?

[davistardif]

I'm fairly certain that this is the issue. I tested this by adding the NODE_PATH env variable to the node 10.x Dockerfile, building it locally and invoking my lambda with --skip-pull-image. I'm curious if there's a way to fix this from within sam...

[jfuss]

@davistardif Have a look at this commit: https://github.com/lambci/docker-lambda/commit/b6b123d2247039fb7d848203dfe79d4eddb41811#diff-55d835ef6c80cb642181f7d2c362c861. This was changed for Python3.7 debugging but since Node10.x is also bootstrapped off of provided in docker-lambda.

[davistardif]

Yup, someone saw this bug coming...

For other programming languages (say Nodejs), we might need to create a separate bootstrap
file that starts the node interpretor with debug args. In this case I anticipate us to invoke
using a command similar to this:

/var/rapid/init \
    --bootstrap /var/runtime/bootstrap.debug \
    --bootstrap-args '["--inspect-brk", "0.0.0.0:9090"]'

[mhart]

If the issue is that NODE_PATH is already set by SAM CLI during debugging and that's why -z "$NODE_PATH" isn't running, then the modification should be made in SAM CLI – not in docker-lambda which is intended to match production as closely as possible.

[jfuss]

@mhart I am not aware of use setting NODE_PATH at all in SAM CLI. I haven't dug into this deep enough, just smelled of maybe needing another bootstrap file (there was a comment about it in this git commit.

This is how SAM CLI is currently doing 'debug' for Nodejs10.x: https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/local/docker/lambda_debug_entrypoint.py#L108. Since Node10.x is bootstrapped from the provided runtime, there might be special things we (SAM CLI or docker-lambda) need to do. Do you have any suggestions?

[mhart]

Ah I see – so the problem is that you're bypassing the bootstrap that's used on production Lambda (which is what sets up the environment variables).

Easiest thing is just to pass these env vars in when you invoke docker. I think you already do for a bunch of them – can just add NODE_ENV for nodejs10.x

[mhart]

You can see if there's anything else you might want to pass in if you're bypassing the bootstrap too:

$ docker run --rm --entrypoint cat lambci/lambda:nodejs10.x /var/runtime/bootstrap

#!/bin/sh

if [ -z "$NODE_PATH" ];
then
  nodejs_mods="/opt/nodejs/node_modules"
  nodejs10_mods="/opt/nodejs/node10/node_modules"
  runtime_mods="/var/runtime/node_modules"
  export NODE_PATH="$nodejs10_mods:$nodejs_mods:$runtime_mods"
fi

if [ -n "$AWS_LAMBDA_FUNCTION_MEMORY_SIZE" ];
then
  new_space=$(expr $AWS_LAMBDA_FUNCTION_MEMORY_SIZE / 10)
  semi_space=$(expr $new_space / 2)
  old_space=$(expr $AWS_LAMBDA_FUNCTION_MEMORY_SIZE - $new_space)
  MEMORY_ARGS=(
    "--max-semi-space-size=$semi_space"
    "--max-old-space-size=$old_space"
  )
fi

exec /var/lang/bin/node \
  --expose-gc \
  --max-http-header-size 81920 \
  "${MEMORY_ARGS[@]}" \
  /var/runtime/index.js

[mhart]

TBH the best solution would be to get the Lambda team to support extra args passed in to the bootstrap – that way you're not gonna get out of sync with any changes they make!

[davistardif]

@mhart Do you know who the point of contact would be for changing that bootstrap file?

[mhart]

@davistardif No idea unfortunately – I don't know who works on what runtime at AWS. A much better question for @jfuss

[singulli1]

any updates on this? I' having the same issue

[sanathkr]

This should be solved as soon as #1646 is fixed

[mark-hayward]

Is there a workaround for this? I had to downgrade to 8.10 but it's EOL in two weeks.

[singulli1]

Set node_path as an environmental variable: NODE_PATH: '/var/runtime/node_modules'  Sent from Mail for Windows 10 From: MarkHaywardSent: Wednesday, December 18, 2019 7:31 AMTo: awslabs/aws-sam-cliCc: singulli1; CommentSubject: Re: [awslabs/aws-sam-cli] NODE_PATH is not set in the Container when in debug mode (#1246) Is there a workaround for this? I had to downgrade to 8.10 but it's EOL in two weeks.—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe. 

[jfuss]

Patched was released in 0.38.0.

Closing