Aws-sam-cli: sam local invoke Error: OCI runtime create failed

[fbeauchemin]

Description:

When running sam local invoke SamCoreFunction --event event.json --debug I get this error:

500 Server Error: Internal Server Error ("OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"setenv: invalid argument\"": unknown")

Fun Fact: running sam local start-api works and can navigate to http://127.0.0.1:3000/ and see Hello AWS Serverless displayed.

Steps to reproduce the issue:

1. dotnet new serverless.EmptyServerless -n SamCore
2. cd SamCore
3. dotnet new sln -n SameCore
4. dotnet sln SamCore.sln add src/SamCore/SameCore.csproj
5. create yaml template for SAM (see below)
6. sam local generate-event apigateway aws-proxy > testApiRequest.json
7. dotnet publish -c Release
8. sam local invoke "DebuggingExampleFunction" --event testApiRequest.json

Observed result:
Error 500

Expected result:
"Hello AWS Serverless" printed on console/output

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
Windows 10 Pro
.Net Core 2.1

YAML Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample SAM Template for SamCore

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
    Function:
        Timeout: 10

Resources:

    SamCoreFunction:
        Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
        Properties:
            FunctionName: SamCore
            CodeUri: src/SamCore/bin/Release/netcoreapp2.1/publish
            Handler: SamCore::SamCore.Functions::Get
            Runtime: dotnetcore2.1
            Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
                Variables:
                    PARAM1: VALUE
            Events:
                SamCore:
                    Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
                    Properties:
                        Path: /
                        Method: get

Outputs:

    SamCoreApi:
      Description: "API Gateway endpoint URL for Prod stage for Debugging Example function"
      Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/SamCore/"

    SamCoreFunction:
      Description: "Debugging Example Lambda Function ARN"
      Value: !GetAtt SamCoreFunction.Arn

    SamCoreFunctionIamRole:
      Description: "Implicit IAM Role created for Debugging Example function"
      Value: !GetAtt SamCoreFunctionRole.Arn

Output of sam --version:
SAM CLI, version 0.6.0

Optional Debug logs:
2018-10-19 13:51:44 local invoke command is called
2018-10-19 13:51:44 2 resources found in the template
2018-10-19 13:51:44 Found Serverless function with name='SamCoreFunction' and CodeUri='src/SamCore/bin/Release/netcoreapp2.1/publish'
2018-10-19 13:51:44 Trying paths: ['C:\Users\fbeauchemin\.docker\config.json', 'C:\Users\fbeauchemin\.dockercfg']
2018-10-19 13:51:44 Found file at path: C:\Users\fbeauchemin.docker\config.json
2018-10-19 13:51:44 Found 'auths' section
2018-10-19 13:51:44 Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
2018-10-19 13:51:44 Found 'credsStore' section
2018-10-19 13:51:44 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2018-10-19 13:51:44 Trying paths: ['C:\Users\fbeauchemin\.docker\config.json', 'C:\Users\fbeauchemin\.dockercfg']
2018-10-19 13:51:44 Found file at path: C:\Users\fbeauchemin.docker\config.json
2018-10-19 13:51:44 Found 'auths' section
2018-10-19 13:51:44 Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
2018-10-19 13:51:44 Found 'credsStore' section
2018-10-19 13:51:44 Found one Lambda function with name 'SamCoreFunction'
2018-10-19 13:51:44 Invoking SamCore::SamCore.Functions::Get (dotnetcore2.1)
2018-10-19 13:51:44 Environment variables overrides data is standard format
2018-10-19 13:51:44 Loading AWS credentials from session with profile 'None'
2018-10-19 13:51:44 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2018-10-19 13:51:44 Changing event name from before-call.apigateway to before-call.api-gateway
2018-10-19 13:51:44 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2018-10-19 13:51:44 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-sca
ling.CreateLaunchConfiguration
2018-10-19 13:51:44 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2018-10-19 13:51:44 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2018-10-19 13:51:44 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaun
chConfiguration.complete-section
2018-10-19 13:51:44 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateEx
portTask
2018-10-19 13:51:44 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.comple
te-section
2018-10-19 13:51:44 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Se
arch
2018-10-19 13:51:44 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-s
ection
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable config_file from defaults.
2018-10-19 13:51:44 Loading variable credentials_file from defaults.
2018-10-19 13:51:44 Loading variable data_path from defaults.
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable credentials_file from defaults.
2018-10-19 13:51:44 Loading variable config_file from defaults.
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable metadata_service_timeout from defaults.
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable metadata_service_num_attempts from defaults.
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Looking for credentials via: env
2018-10-19 13:51:44 Looking for credentials via: assume-role
2018-10-19 13:51:44 Looking for credentials via: shared-credentials-file
2018-10-19 13:51:44 Found credentials in shared credentials file: ~/.aws/credentials
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable region from defaults.
2018-10-19 13:51:44 Loading variable profile from defaults.
2018-10-19 13:51:44 Loading variable region from defaults.
2018-10-19 13:51:44 Resolving code path. Cwd=C:\Users\fbeauchemindev\epic\spikes\sam-core\SamCore, CodeUri=src/SamCore/bin/Release/netcorea
pp2.1/publish
2018-10-19 13:51:44 Resolved absolute path to code is C:\Users\fbeauchemindev\epic\spikes\sam-core\SamCore\src\SamCore\bin\Release\netcorea
pp2.1\publish
2018-10-19 13:51:44 Code C:\Users\fbeauchemindev\epic\spikes\sam-core\SamCore\src\SamCore\bin\Release\netcoreapp2.1\publish is not a zip/ja
r file
2018-10-19 13:51:44 Trying paths: ['C:\Users\fbeauchemin\.docker\config.json', 'C:\Users\fbeauchemin\.dockercfg']
2018-10-19 13:51:44 Found file at path: C:\Users\fbeauchemin.docker\config.json
2018-10-19 13:51:44 Found 'auths' section
2018-10-19 13:51:44 Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
2018-10-19 13:51:44 Found 'credsStore' section
2018-10-19 13:51:44 http://localhost:None "GET /v1.35/images/lambci/lambda:dotnetcore2.1/json HTTP/1.1" 200 None
2018-10-19 13:51:44 Looking for auth config
2018-10-19 13:51:44 Using credentials store "wincred"
2018-10-19 13:51:44 Looking for auth entry for 'https://index.docker.io/v1/'
2018-10-19 13:51:44 Found auth config
2018-10-19 13:51:45 http://localhost:None "POST /v1.35/images/create?tag=dotnetcore2.1&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:dotnetcore2.1 Docker container image......
2018-10-19 13:51:45 Mounting C:\Users\fbeauchemindev\epic\spikes\sam-core\SamCore\src\SamCore\bin\Release\netcoreapp2.1\publish as /var/tas
k:ro inside runtime container
2018-10-19 13:51:45 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2018-10-19 13:51:46 http://localhost:None "GET /v1.35/containers/ac75f71ea9499c515cc84e69748259f879b46f1c2c07a33e68124477cf3d3c31/json HTTP/
1.1" 200 None
2018-10-19 13:51:46 http://localhost:None "GET /v1.35/containers/ac75f71ea9499c515cc84e69748259f879b46f1c2c07a33e68124477cf3d3c31/json HTTP/
1.1" 200 None
2018-10-19 13:51:46 http://localhost:None "POST /v1.35/containers/ac75f71ea9499c515cc84e69748259f879b46f1c2c07a33e68124477cf3d3c31/start HTT
P/1.1" 500 189
2018-10-19 13:51:46 http://localhost:None "GET /v1.35/containers/ac75f71ea9499c515cc84e69748259f879b46f1c2c07a33e68124477cf3d3c31/json HTTP/
1.1" 200 None
2018-10-19 13:51:46 http://localhost:None "DELETE /v1.35/containers/ac75f71ea9499c515cc84e69748259f879b46f1c2c07a33e68124477cf3d3c31?v=False
&link=False&force=True HTTP/1.1" 204 0
Traceback (most recent call last):
File "c:\python37\lib\site-packages\docker\api\client.py", line 229, in _raise_for_status
response.raise_for_status()
File "c:\python37\lib\site-packages\requests\models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localnpipe/v1.35/containers/ac75f71ea9499c515c
c84e69748259f879b46f1c2c07a33e68124477cf3d3c31/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python37\Scripts\sam-script.py", line 11, in
load_entry_point('aws-sam-cli==0.6.0', 'console_scripts', 'sam')()
File "c:\python37\lib\site-packages\click\core.py", line 722, in __call__
return self.main(args, *kwargs)
File "c:\python37\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "c:\python37\lib\site-packages\click\core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\python37\lib\site-packages\click\core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\python37\lib\site-packages\click\core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "c:\python37\lib\site-packages\click\core.py", line 535, in invoke
return callback(args, *kwargs)
File "c:\python37\lib\site-packages\click\decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], *kwargs)
File "c:\python37\lib\site-packages\click\core.py", line 535, in invoke
return callback(args, *kwargs)
File "c:\python37\lib\site-packages\samcli\commands\local\invoke\cli.py", line 49, in cli
docker_volume_basedir, docker_network, log_file, skip_pull_image, profile, region) # pragma: no cover
File "c:\python37\lib\site-packages\samcli\commands\local\invoke\cli.py", line 90, in do_cli
stderr=context.stderr)
File "c:\python37\lib\site-packages\samcli\commands\local\lib\local_lambda.py", line 80, in invoke
self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
File "c:\python37\lib\site-packages\samcli\local\lambdafn\runtime.py", line 79, in invoke
self._container_manager.run(container)
File "c:\python37\lib\site-packages\samcli\local\docker\manager.py", line 63, in run
container.start(input_data=input_data)
File "c:\python37\lib\site-packages\samcli\local\docker\container.py", line 176, in start
real_container.start()
File "c:\python37\lib\site-packages\docker\models\containers.py", line 374, in start
return self.client.api.start(self.id, *kwargs)
File "c:\python37\lib\site-packages\docker\utils\decorators.py", line 19, in wrapped
return f(self, resource_id, *args, *kwargs)
File "c:\python37\lib\site-packages\docker\api\container.py", line 1066, in start
self._raise_for_status(res)
File "c:\python37\lib\site-packages\docker\api\client.py", line 231, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "c:\python37\lib\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("OCI runtime create failed: container_linux.go:348: starting container proc
ess caused "process_linux.go:402: container init caused \"setenv: invalid argument\"": unknown")

[jfuss]

@fbeauchemin Thanks for the report. This seems to be coming from docker but unsure what is causing it. Will need to try and reproduce.

Updating labels.

[nickcox]

I have the same issue on Windows. I guess it may have to do with how the path to the event data is being passed? It works if I do this instead:

cat ./event.json | sam local invoke SamCoreFunction

[MuazOthman]

I'm facing the same issue here, the last workaround posted by @nickcox works. Any updates how this can be fixed?

[blbradley]

Can confirm, also on Windows.

[drettie]

i just had the same issue on windows and resolved by changing the example event json file encoding from utf-16-LE to utf-8 and changing CRLF to LF.

[Dimagithub]

i just had the same issue on windows and resolved by changing the example event json file encoding from utf-16-LE to utf-8 and changing CRLF to LF.

this works for me!

[reegz]

i just had the same issue on windows and resolved by changing the example event json file encoding from utf-16-LE to utf-8 and changing CRLF to LF.

This seems to do the trick. Thanks for the tip!

[sriram-mv]

This is something we employ on our integration tests: https://github.com/awslabs/aws-sam-cli/blob/develop/tests/integration/buildcmd/test_build_cmd.py#L452

Should potentially add a change to generate-event to always generate LF line endings irrespective of platform.