Aws-app-mesh-roadmap: [BUG] X-Ray extension causing seg-fault in app-mesh-envoy

[kiranmeduri]

_Important note on security disclosures: If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly._

Describe the bug
When using application with App Mesh Envoy sidecar (111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy:v1.11.1.0-prod) with x-ray enabled I am noticing segfaults. Seems like if existing headers have x-amzn-trace-id (ALB sets this), then other headers are parsed. However, I notice they are garbled in the sampling.cc and then a seg-fault. Result of this is that the Envoy dies and tasks churn.

Below is the stacktrace from Envoy (debug-mode)

[2019-08-29 17:42:53.055][36][debug][http] [source/common/http/conn_manager_impl.cc:606] [C17][S646880477383125921] request headers complete (end_stream=true):
':authority', 'howto-publi-[redacted].us-west-2.elb.amazonaws.com'
':path', '/color'
':method', 'GET'
'x-forwarded-for', '[redacted]'
'x-forwarded-proto', 'http'
'x-forwarded-port', '80'
'x-amzn-trace-id', 'Root=1-5d680e9d-6905f4ecaed375d0eee2d06c'
[2019-08-29 17:42:53.055][36][debug][http] [source/common/http/conn_manager_impl.cc:1098] [C17][S646880477383125921] request end stream
[2019-08-29 17:42:53.055][36][debug][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:161] start sampling strategy.
[2019-08-29 17:42:53.055][36][debug][tracing] [source/extensions/tracers/xray/sampling.cc:139] determining ShouldTrace decision for: host: howto-publi-[redacted].us-west-2.elb.amazonaws.comF4ErWjfJXir0xuKhXFSbplQAz/DxwceYMBo7Nhbbo27q/a2ywtrvAkcTisDxszGt@ path: /colorZpc2EgZUNvbW1l
cmNlIFJvb3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvV95WHm6h
2mCxlCfLF9sHP4CFT8icttD0b0/Pmdjh28JIXDqsOT method: GETpI/k4+oKsGGelT84ATB+0t
vz8KPFUgOSwsAGl0lUq8ILKpeeUYiZGo3BxN77t+Nwtd/jmliFKMAGzsGHxBvfaL
dXe6YJ2E5/4tAgMBAAGjQjBAMA8GA1UdEwEB/
[2019-08-29 17:42:53.055][36][debug][tracing] [source/extensions/tracers/xray/xray_tracer_impl.cc:172] starting generate a segment based on downstream xray header.
[2019-08-29 17:42:53.055][36][critical][assert] [source/common/http/header_map_impl.cc:210] assert failure: valid().
[2019-08-29 17:42:53.055][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation fault, suspect faulting address 0x0
[2019-08-29 17:42:53.055][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-08-29 17:42:53.230][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::SignalAction::sigHandler() [0x19aa49a]
[2019-08-29 17:42:53.230][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#1: __restore_rt [0x7f09991091f0]
[2019-08-29 17:42:53.239][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::SignalAction::sigHandler() [0x19aa49a]
[2019-08-29 17:42:53.239][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#1: __restore_rt [0x7f09991091f0]
[2019-08-29 17:42:53.341][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#2: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x1766d38]
[2019-08-29 17:42:53.528][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:69] starting async DNS resolution for internal-howto-[redacted].us-west-2.elb.amazonaws.com
[2019-08-29 17:42:53.529][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#2: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x1766d38]
[2019-08-29 17:42:53.530][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:76] async DNS resolution complete for internal-howto-[redacted].us-west-2.elb.amazonaws.com
[2019-08-29 17:42:53.630][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#3: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x1766d97]
[2019-08-29 17:42:53.644][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#3: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x1766d97]
[2019-08-29 17:42:53.836][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#4: Envoy::Extensions::Tracers::XRay::Driver::startSpan() [0xdfc03e]
[2019-08-29 17:42:53.930][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-examples#4: Envoy::Extensions::Tracers::XRay::Driver::startSpan() [0xdfc03e]
[2019-08-29 17:42:54.129][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-roadmap#17: Envoy::Tracing::HttpTracerImpl::startSpan() [0x13fad05]
[2019-08-29 17:42:54.139][36][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-roadmap#17: Envoy::Tracing::HttpTracerImpl::startSpan() [0x13fad05]
[2019-08-29 17:42:54.334][37][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] aws/aws-app-mesh-roadmap#18: Envoy::Http::ConnectionManagerImpl::ActiveStream::traceRequest() [0x139de2d]

Platform
ECS service behind public ALB

To Reproduce
Steps to reproduce the behavior:
I was able to use https://github.com/kiranmeduri/aws-app-mesh-examples/tree/alb/walkthroughs/howto-alb to repro this consistently.

Expected behavior
Envoy should not seg-fault

Config files, and API responses
If applicable config files and responses from our API.

Additional context
Add any other context about the problem here.

[lavignes]

I've identified the issue and we will be releasing a new Envoy image to fix the bug. I'll update when the image is ready.

[lavignes]

The new image is ready and available in all App Mesh regions:

111345817488.dkr.ecr.REGION.amazonaws.com/aws-appmesh-envoy:v1.11.1.1-prod

I'll be updating the documentation accordingly.

[lavignes]

The docs have been updated: https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy.html

[lavignes]

Closing. The rest of the docs changes are going out now. Thanks for catching this @kiranmeduri !