Aws-sam-cli: [Feature Request] Add support for X-Ray on SAM Local

[hugoprudente]

Hi Team,

When trying to use the AWS SAM Local with Node.js per example it fails with the following error:

 Error: Missing AWS Lambda trace data for X-Ray. Expected _X_AMZN_TRACE_ID to be set.

This happens as the X-Ray Lambda already have the traces as per documentation: http://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html

So when using SAM Local this Trace Header is a missing.

[PaulMaddox]

Thanks for opening this.

I believe the steps we need to take to add this to SAM Local are:

1. Check if Tracing: Active is set in the SAM template for the function being invoked
2. If so, run the AWS X-Ray daemon as a sidecar container in invoke.go
3. Set the correct environment variables (e.g. _X_AMZN_TRACE_ID) in env.go

[svenemtell]

It would be nice to use X-Ray from lambdas run using AWS SAM Local BUT more important to me would be a way to silently discard X-Ray calls in the code.
I want to be able to use the same code in production as when testing using AWS SAM Local and for that to work now I have written something like this (which I don't want to do):

if (process.env.AWS_SAM_LOCAL) {
AWS = require('aws-sdk');
} else {
// Only run AWS X-Ray when NOT running AWS SAM Local
AWSXRay = require('aws-xray-sdk-core');
AWS = AWSXRay.captureAWS(require('aws-sdk'));
}

[rvypandey]

+1

This will be a very valuable tool for us!

[Willis0826]

Could we using sam local with X-Ray now?

[metaskills]

It appears not yet. I've been using the process.env check above as mentioned till then. Not a big issue for me either way.

[jabalsad]

The primary use-case I have for enabling x-ray integration is really just so that I don't have to have different configuration in my local dev environment vs my production environment. I'd like to be able to have my code enable x-ray, and the SAM local simulation not dying or raising errors on me. I'm less opinionated about whether the requests actually get submitted to the x-ray service upstream, and care more about just being able to emulate it locally.

{"level":"info","ts":"2020-01-30T01:52:08.225Z","msg":"Starting MyService HTTP server..."}
START RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a Version: $LATEST
{"level":"debug","ts":"2020-01-30T01:52:08.237Z","msg":"Created new Resource: {\"id\":\"4KML2o7ZZXR\",\"field\":\"111122223333\",\"name\":\"integrationtestconfig\",\"destinationArn\":\"arn:aws:s3:::some-bucket\"}"}
2020-01-30T01:52:08Z [Error] Suppressing AWS X-Ray context missing panic: failed to begin subsegment named 'dynamodb': segment cannot be found.
END RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a
REPORT RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a  Init Duration: 159.52 ms    Duration: 6.39 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 27 MB
Invalid API Gateway Response Keys:
[snip snip]

[destinybonavita]

Any update on this?

[jfuss]

We are not looking to support X-Ray locally at this time.

Closing.

[hugoprudente]

Not planing to have the feature and closing it instead moving it to a backlog to see how the priority should grow it's not the best way to handle the issue.

You are simply saying here that the community feedback and customers using the aws-sam-cli doesn't matter.

[awsjeffg]

@hugoprudente thanks for the honest feedback, we appreciate it. I realize it is frustrating to have filed this issue 3 years ago and have it closed after all this time. We decided to close this issue as part of an effort on the team currently maintaining SAM cli to do a better job of communicating with community members about what our intentions are, even if the answer isn't the one you would prefer.

Having said that, let's talk about it. If sam cli were to "support" X-Ray, what would your ideal dev experience be? The original report indicates you just don't want the lambda to crash - is that it, or do you want sam cli to actually behave like X-Ray in the cloud?

[hugoprudente]

A first point is prevent the crash and prevent us dev to create unnecessary handling as the production will have X-Ray so the sam-cli that we use as dev/test environments also should handle it.

Supporting the feature is also necessary as what's the point to have something that I can't test my traces locally and only in production and if I need to test X-Ray locally I can't use SAM-CLI and do everything by hand so in the end it's better to do everything by hand as I have being doing instead wasting my time in the SAM-CLI as the effort of testing x-ray and adding and removing it to work with SAM-CLI doesn't make sense at all.

X-Ray already works locally publishing it to the X-RAY service or being intercepted locally by any UDP handlers so having the SDK on a similar fashion s the Lambda have so users can try simulate the best of the Lambda features locally and understand how do X-Ray will work for their use case it's just a extra plus.

From my use cases only if doesn't use X-Ray at all I have the SAM-CLI and if it has the probability of needing X-Ray in the feature I don't use it as the extra effort describing above of crating 2 lanes todo 1 thing I prefer to pay extra for the lambdas and test on the real environment than waste my developers time.

[qingchm]

@hugoprudente Hi there, first of all unfortunately the reason that we said that we're not looking to support it is 1) some components just cannot support x-ray locally, example: https://github.com/aws/aws-lambda-runtime-interface-emulator#level-of-support 2) local tracing with x-ray doesn't work if the x-ray resource is not created yet. We really want to support the idea of having local runs to capture more Lambda features but x-ray isn't one that we can support yet. However, we are more than willing to communicate with you to provide a workaround locally so that you don't have to waste your time modifying your templates. Here are 2 options that we will try to provide: 1) we can generate warnings instead of errors for tracing property so that the local run wouldn't fail 2) we can mock what x-ray tracing does by printing to stdout locally. Please give us a feedback on which workaround idea makes more sense to you so we can improve your experience using sam-cli!

[hugoprudente]

Hi @qingchm,

I didn't know which project was behind the creation of the AWS Lambda mock. The base support for this feature should be provided there it looks like.

Regarding the X-Ray Resource is not created, which resource exactly you are referring to? As AWS X-Ray only needs the AWS X-Ray SDK in the code that is provided by AWS Lambda and bootstrapped together with the code that executes the lambda_handler in the same context.

Regarding the options that you provided here are my observations:

1) Show the warning will continue to fail the code due to the missing embedded AWS X-Ray SDK in the lambda_handler, and if I add the SDK manually it will be different from the production, which will double the work of testing it locally.

2) Mock the Xray, it will only be possible if you embed the X-Ray SDK on the lambda_handler executer context before the executions, which will solve the issue.

---

Here is a little bit more context on how the AWS X-Ray works with AWS Lambda so we can trace the best approach to support minimal features on SAM CLI.

The AWS Lambda code that executes the lambda_handler when X-Ray is enabled somehow loads the AWS X-RAY SDK Core in python will be this:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

xray_recorder.configure(service='AWS SAM Local')
patch_all()

Then when I execute my lambda_handler during the AWS Lambda execution the xray_recoreder will be on my context creating my segment.

The aws_xray_sdk.core will automatically try to send the information to UDP2000 port, which may or not listen or do something with it.

With a couple of lines of doc, It's easy on how to pick that information and sent it to AWS X-RAY Api or for a mock socket server for troubleshooting.

The most important bit here is how can we bootstrap all the language supporting X-Ray to enable the X-Ray Recorder with the basic configuration and particularities like Segment not mutable.

I'm not sure if this will be resolved here, or if it will be resolved on the AWS Lambda Runtime Interface Emulator.

Either way, I will be glad to help add a wrapper to the executor to it but I'm sure that the AWS Lambda Team would be able to provide more insights on how they are wrapping it so this could be consistent, with production/edge and provide the same limitations to be as "real" as possible, as I know that some particularities that only happens on X-Ray SDK with AWS Lambda.

Looking forward to hearing from you on which project will be responsible for the lambda_handler execution so I can help you with a simple example and we can plan the roadmap on how to add this feature in the upcoming future, as I'm sure I'm not the only one that put SAM-CLI a side due to this limitation.

[qingchm]

@hugoprudente Hi sorry for the confusion in my expressions, since I am studying on this issue just recently. But as for the first workaround, what I mean is to generate warning while skipping the requirement for the x-ray sdk. And by the creation what I mean is the daemon that the x-ray sdk communicates with. We need to integrate that into all environments as well in order for the sdk to work. Hopefully my words make better sense now. And to further address this issue, we are involving more discussions and I should be in touch with you soon!