Aws-sam-cli: `start-lambda` does not catch runtime errors, so calls to it never fail

[mereghost]

Description

With sam local start-lambda the failure is only taken into account if the actual function returns the specified json with all 3 keys (even if just 1 is mandatory) (errorType, errorMessage and stackTrace).

When a function fails due to other issues, a wrong method call or some kind of runtime error for example, start-lambda will not catch it, therefore marking the task as succeeded.

This hampers any effort of testing step functions with retry mechanisms or branching based on failure.

Steps to reproduce

A simple app that should trigger failure (ruby in this case, but language is irrelevant):

def handle(context:, event:)
  not_defined_value.someMethod
end

Observed result

The response from the service is { "statusCode" : 200 }

Expected result

A response that indicates that an error occured: { "statusCode" : 200, "FunctionError" : "Unhandled" }

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux
2. sam --version: 0.21.0

[mereghost]

I did some investigation on this and it seems related on how lambda_service deals with responses.

Basically it grabs stdout from the container and tries to parse it, but in this case, there's nothing in there so it thinks that execution succeeded (LambdaOutputParser#is_user_error_response returns false) but there's no checks for runtime errors even if the actual container output
{ "errorType" : "NoMethodError", "errorMessage" : "some debugging message" }

The actual service, while running on AWS would fail in this case.

[anthonymq]

This behaviour implies that we can't develop step functions locally as every failing step is considered a Success.

[anthonymq]

Same issue in Linux with 0.21.0

Same in 0.22.0

[anton-pfc]

Same problem here on MacOS.
Please improve SAM so we can use it for local development.

Cmon Amazon you can do it better!

[zhulik]

Same here, this issue makes SAM absolutely useless for integration testing.

[adamrecsko]

+1
same issue here

[ashleyfrieze]

I had the exact same problem as the above in Java, even though Java was correctly throwing an Exception and the Lambda was finishing with the right JSON exception output.

Looking at the code, it seems that the valid output for an error is expected to have 2 or 3 properties. In Java, the cause field provides a fourth. I worked around this in my code so that the exception thrown by the handler did not provide a cause. This resolved the issue for me.

However, it seems to me that the correct logic should be 2, 3, 4 and when I get a moment, I'll create a PR for that.

[ashleyfrieze]

I have a fix for this issue in #1820 - any chance of someone reviewing it?

[jfuss]

The initially issue was previously handled in #1642. @ashleyfrieze Had a similar issue with another key we were missing. Since the initially issue was fixed here, I am going to close it but @ashleyfrieze additional fix will be applied in our next release (which impacts Java for sure and I believe .Netcore runtimes).