Aws-app-mesh-roadmap: Support Dynamic AWS X-Ray sampling configuration

[lavignes]

One feature that was missed when launching X-Ray integration was the ability to customize the sampling configuration.

Today our Envoy image is hard-coded to sample 5% of requests when the X-Ray tracer is enabled.

[efe-selcuk]

This will also allow more general configuration of the extension, beyond just the sampling configuration.

[Puneeth-n]

@efe-selcuk Will running X-Ray agent as a daemon (rather than a side car) be supported anytime soon? I believe, since the host is harcoded to 127.0.0.1 (https://github.com/aws/aws-app-mesh-roadmap/issues/17#issuecomment-533069992) All tasks in ECS should have awsvpc networking.

[lavignes]

Hi @Puneeth-n

Will running X-Ray agent as a daemon (rather than a side car) be supported anytime soon?

Yes. We are actively working on that right now. Just the other day we started adding an improved version of the X-Ray extension to upstream Envoy: envoyproxy/envoy#8526

Specifically, you can see that the new config will support changing the host:
https://github.com/envoyproxy/envoy/pull/8526/files#diff-59c1f81c2509c74bbba8f573f4902423R209

[dastbe]

Current status is that you can configure a local sampling rule via Envoy configuration. However, Envoy does not yet support retrieving dynamically defined sampling rules

[LarsFronius]

@dastbe Do you have an example for how I can overwrite the static sampling rules of envoy in the context of app mesh? I am hoping I won't need to build a custom container, but so far don't see how I could change the default sampling rate using e.g. environment variables.

[CarlosDomingues]

@LarsFronius Did you find an answer?

[arthuroz]

Current status is that you can configure a local sampling rule via Envoy configuration. However, Envoy does not yet support retrieving dynamically defined sampling rules

I am not sure that is the case, xray-daemon is not configuring the SamplingRule, here is the code

[lavignes]

@arthuroz The sampling decision for traces that Envoy emits are made in Envoy itself. The config referenced in the Envoy docs is read by Envoy and is used to configure its own rule.

What is missing today is the option to easily enable this in our official Envoy image.

[arthuroz]

@lavignes thank you Scott, now I get it.
Dumb question, is the Envoy repo an open source project? I cannot seem to find it...

[lavignes]

@arthuroz Are you referring to the App Mesh Envoy image? Envoy Proxy itself is here: https://github.com/envoyproxy/envoy

Unfortunately we haven't open-sourced everything in our custom image yet though. We're tracking the most relevant part to this issue here: https://github.com/aws/aws-app-mesh-roadmap/issues/264

[arthuroz]

@lavignes Thanks for the link, I meant a public repo for the custom Envoy image by App Mesh team, it would be much easier than digging into the image to figure things out, plus it allows the community to contribute.

This is getting frustrating :(

Though from Envoy 1.15 onward, most of the logic has been moved to a binary, so it is effectively impossible to work-around without examples of working bootstrap configuration to start from.

Voted for #264.

[CarlosDomingues]

My use case for for this feature is that the 5% sample rate makes App Mesh tracing useless for low traffic / internal applications.

[robert-csdisco]

@dastbe

Current status is that you can configure a local sampling rule via Envoy configuration. However, Envoy does not yet support retrieving dynamically defined sampling rules

How do you pass in the custom config for config.trace.v3.XRayConfig or the config.trace.v3.ZipkinConfig into the AWS provider envoy image? I don't see a lot of information on how to mix in our own envoy config into the one that AWS writes into the /tmp/envoycfg* file

Edit to add: I found ENVOY_STATS_CONFIG_FILE and ENVOY_STATS_SINKS_CFG_FILE referenced here: https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy-config.html, but nothing about how to inject our own tracing config