Aws-sam-cli: RC1 failing local invoke for Node12.

[c2tarun]

Description

sam local invoke fails for nodejs12 project with following error:

    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/containers/b750d7791ae2d2b1ab5b57a27a0b50f8379e170e06e9205c9ab150827f69fcea/start

Steps to reproduce

1. sam init a node12 project.
2. sam local invoke in that project.

Observed result

❯ sam local invoke --debug                                                                                                                                                               ─╯
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
Found one Lambda function with name 'HelloWorldFunction'
Invoking app.lambdaHandler (nodejs12.x)
No environment variables found for function 'HelloWorldFunction'
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=/Users/tarumall/workplace/byol-testbed/sam-app-n12/.aws-sam/build, CodeUri=HelloWorldFunction
Resolved absolute path to code is /Users/tarumall/workplace/byol-testbed/sam-app-n12/.aws-sam/build/HelloWorldFunction
Code /Users/tarumall/workplace/byol-testbed/sam-app-n12/.aws-sam/build/HelloWorldFunction is not a zip/jar file
Skipping building an image since no layers were defined

Fetching amazon/aws-sam-cli-emulation-image-nodejs12.x:latest Docker container image......
Mounting /Users/tarumall/workplace/byol-testbed/sam-app-n12/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 1565, 'exitReason': 'APIError', 'exitCode': 255, 'requestId': 'dc38f743-a36c-417f-a3a4-b10902e605b1', 'installationId': '2b7af8d7-cfbd-4d71-baa7-3dfc0f010bbe', 'sessionId': '72e9043d-9e17-4974-a387-431d2a80ad28', 'executionEnvironment': 'CLI', 'pyversion': '3.7.7', 'samcliVersion': '1.0.0rc1'}}]}
Telemetry response: 200
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/containers/8d47e54ddafcc6815d508126c6b260591a9cf2107a9fa7ae391b34d643288bfe/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sam", line 33, in <module>
    sys.exit(load_entry_point('aws-sam-cli==1.0.0rc1', 'console_scripts', 'sam')())
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 96, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/commands/local/invoke/cli.py", line 86, in cli
    parameter_overrides,
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/commands/local/invoke/cli.py", line 151, in do_cli
    context.function_name, event=event_data, stdout=context.stdout, stderr=context.stderr
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/commands/local/lib/local_lambda.py", line 100, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/local/lambdafn/runtime.py", line 83, in invoke
    self._container_manager.run(container)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/local/docker/manager.py", line 95, in run
    container.start(input_data=input_data)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/local/docker/container.py", line 188, in start
    real_container.start()
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/models/containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Mounts denied: \r\nThe path /usr/local/Cellar/aws-sam-cli-rc/1.0.0rc1/libexec/lib/python3.7/site-packages/samcli/local/rapid\r\nis not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'")

Expected result

Describe what you expected.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS Catalina
2. sam --version: SAM CLI, version 1.0.0rc1

Add --debug flag to command you are running

[awood45]

The issue here is that on OSX, Docker by default can mount the following folders:

/Users
/Volumes
/private
/tmp

The Homebrew install path is not covered in these permissions, hence the issue (and also how it skipped past all testing before release).

[drissamri]

I've added /usr and the homebrew path directly but this does not seem to help?
Then the error changes to urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

[awood45]

If you add the path to Docker's shared folders, that workaround should be valid and worked for all of us. That said, it's not a long term solution to ask everyone to open up their Homebrew paths, so this will be fixed before 1.0.0.

[awood45]

@drissamri I haven't seen that error before - I think the solutions being worked on will prevent that as well, but what did you change and what command did you run where you saw that?

[drissamri]

I upgraded (and now downgraded back to previous version) and ran sam local invoke again on my Java project. First I got the Mounts denied exception and after added extra shared paths in docker the timeout popped up.

I didn't dive in too deep since I was prepping for a presentation, so was easier to downgrade first :)

[awood45]

This is fixed in 1.0.0rc2, closing.

FWIW, glad you were able to move back to 0.53.0 successfully. We're working on avoiding this type of edge case getting through our tests again, but the possibility was why we made this opt-in/--pre and we appreciate you testing it and letting us know what you saw. Hopefully you're able to try 1.0.0rc2 and find the problems resolved for good.

[jcardus]

I'm using version 1.0.0 and I have exactly the same issue

[c2tarun]

I'm using version 1.0.0 and I have exactly the same issue

Hi @jcardus, I am not seeing this issue in nodejs12 anymore. Can you please share more details around the environment where you are seeing this issue?

Feel free to re-open this issue if problem persists for you.

Thanks
Tarun

[Stone2517]

Invoking index.handler (nodejs12.x) Failed to download a new amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.0.0 image. Invoking with the already downloaded image. Mounting /Users/naveen/Downloads/AWS/sam/newton/src/lambda/postSignUp as /var/task:ro,delegated inside runtime container

I pulled the amazon/aws-sam-cli-emulation-image-nodejs12.x: latest, still I get the same message.

[c2tarun]

Hi @Stone2517, this is a minor bug which we are tracking #2165.

[seongyeolpark]

i've got same message when i tried to "sam local invoke"

[sam cli version]
SAM CLI, version 1.0.0

[message]
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.0.0 image. Invoking with the already downloaded image.

[andrehigher]

I'm having the same issue when I run sam deploy --guided

➜  sam-app sam --version
SAM CLI, version 1.0.0

➜  sam-app sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]:
    AWS Region [us-east-1]:
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to samconfig.toml [Y/n]: y

    Looking for resources needed for deployment: Found!
Traceback (most recent call last):
  File "/usr/local/bin/sam", line 33, in <module>
    sys.exit(load_entry_point('aws-sam-cli==1.0.0', 'console_scripts', 'sam')())
  File "/usr/local/Cellar/aws-sam-cli/1.0.0/libexec/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.0.0/libexec/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)

[mg98]

Me too, I'm getting the message on function invocations..

Failed to download a new amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.0.0 image. Invoking with 
the already downloaded image.

Using various quick start templates for nodejs12.x

[LucianViana]

Also with problem:
Failed to download a new amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.1.0 image. Invoking with the already downloaded image.

[pfigueroax]

I had the same problem and I fixed it setting up Docker File Sharing.
Add the folder you are working on and restart Docker and that's all
https://docs.docker.com/docker-for-windows/#file-sharing

Note: I am getting the same error (_Failed to download a new amazon/aws-sam-cli-emulation-image-nodejs10.x:rapid-1.1.0 image_) but the process automatically takes an already downloaded image and ends successfuly.

> sam local invoke "HelloWorldFunction" -e events/event.json
Invoking app.lambdaHandler (nodejs10.x)
Failed to download a new amazon/aws-sam-cli-emulation-image-nodejs10.x:rapid-1.1.0 image. **Invoking with the already downloaded image.**
Mounting C:\sam-app\.aws-sam\build\HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 2ddb162f-2032-10a5-2654-923c47671941 Version: $LATEST
END RequestId: 2ddb152f-2032-10a5-2654-923c47671941
REPORT RequestId: 2ddb152f-2032-10a5-2654-923c47671941  Init Duration: 164.41 msDuration: 3.49 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 43 MB

{"statusCode":200,"body":"{\"message\":\"hello world\"}"}

[sburgos]

I had the same problem, but when I changed it to nodejs10, everything started working well

[mattvb91]

same issue SAM CLI, version 1.10.0

Error: Could not find amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.10.0 image locally and failed to pull it from docker.

[chrisbelyea]

Sam issue as https://github.com/aws/aws-sam-cli/issues/2095#issuecomment-726681715 but with a Python 3.8 project on Ubuntu 20.04.

SAM CLI, version 1.10.0

$ sam local invoke
Invoking app.lambda_handler (python3.8)
Image was not found.
Building image......
Error: Could not find amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.10.0 image locally and failed to pull it from docker.

Presuming that SAM CLI is trying to pull from Docker Hub, the only tagged image there is latest.

---

I tried to hack around the issue like so:

# Get the only available image from Docker Hub
docker pull amazon/aws-sam-cli-emulation-image-python3.8:latest

# Re-tag it as the image SAM is looking for
docker tag amazon/aws-sam-cli-emulation-image-python3.8:latest amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.10.0

# The image SAM wants is now present locally, so don't bother downloading it
sam local invoke

Unfortunately, it still blew up:

Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.10.0.

Mounting /home/chris/repo/fortune-slack-bot/.aws-sam/build/FortuneSlackBotFunction as /var/task:ro,delegated inside runtime container
Traceback (most recent call last):
  File "/home/chris/.local/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/home/chris/.local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.35/containers/c3b2da83aa3b6f11e04443d28393c422982f95b814db64fcd44e902a1a212de5/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chris/.local/bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/lib/telemetry/metrics.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/commands/local/invoke/cli.py", line 72, in cli
    do_cli(
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/commands/local/invoke/cli.py", line 152, in do_cli
    context.local_lambda_runner.invoke(
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/commands/local/lib/local_lambda.py", line 103, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/local/lambdafn/runtime.py", line 84, in invoke
    self._container_manager.run(container)
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 110, in run
    container.start(input_data=input_data)
  File "/home/chris/.local/lib/python3.8/site-packages/samcli/local/docker/container.py", line 219, in start
    real_container.start()
  File "/home/chris/.local/lib/python3.8/site-packages/docker/models/containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/chris/.local/lib/python3.8/site-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/home/chris/.local/lib/python3.8/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/chris/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/var/rapid/init\": stat /var/rapid/init: no such file or directory": unknown")

[mattvb91]

NodeJs12 image just started working for me again!