Envoy: Segfault using ext_authz request buffering with allow_partial_message set

Created on 2 May 2019 · 4Comments · Source: envoyproxy/envoy

Description:

Envoy can segfault when using ext_authz with a request body larger than the limit and partial messages allowed. It does not always segfault, but for me it segfaults within about 5 or so attempts on an otherwise idle envoy.

cc @gsagula

Example: Send a 16k file a few times with a 8k limit:

          http_filters:
          - name: envoy.ext_authz
            config:
              grpc_service:
                envoy_grpc:
                  cluster_name: auth-test-cluster
                timeout: 0.2s
              failure_mode_allow: true
              with_request_body:
                max_request_bytes: 8192
                allow_partial_message: true
          - name: envoy.router
            config: {}

Repro steps:

Create a 16k test data file (larger files seem to work faster - try 100k):

dd if=/dev/zero of=16k.data bs=1k count=16

Send with curl:

curl --data-binary '@16k.data' http://localhost:8000

Config:

Simple config with 8k limit:

static_resources:

  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: http-test-server
          http_filters:
          - name: envoy.ext_authz
            config:
              grpc_service:
                envoy_grpc:
                  cluster_name: auth-test-cluster
                timeout: 0.2s
              failure_mode_allow: true
              with_request_body:
                max_request_bytes: 8192
                allow_partial_message: true
          - name: envoy.router
            config: {}
  clusters:
  - name: http-test-server
    connect_timeout: 0.25s
    type: strict_dns
    load_assignment:
      cluster_name: http-test-server
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: http-test-server
                port_value: 8086
  - name: auth-test-cluster
    connect_timeout: 0.25s
    type: strict_dns
    http2_protocol_options: {}
    load_assignment:
      cluster_name: auth-test-cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: auth-test-service
                port_value: 8000
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Call Stack:

Stacktrace using envoyproxy/envoy-alpine-debug:latest with gdb added:

0x00000000007fe66a in Envoy::Extensions::Filters::Common::ExtAuthz::GrpcClientImpl::onSuccess (this=0x4242800, response=...,
    span=...) at source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:68
68  source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc: No such file or directory.
(gdb) bt
#0  0x00000000007fe66a in Envoy::Extensions::Filters::Common::ExtAuthz::GrpcClientImpl::onSuccess (this=0x4242800, response=...,
    span=...) at source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:68
#1  0x00000000007fe979 in Envoy::Grpc::TypedAsyncRequestCallbacks<envoy::service::auth::v2::CheckResponse>::onSuccessUntyped (
    this=0x4242808, response=..., span=...)
    at bazel-out/k8-opt/bin/include/envoy/grpc/_virtual_includes/async_client_interface/envoy/grpc/async_client.h:104
#2  0x0000000000bf9bbd in Envoy::Grpc::AsyncRequestImpl::onRemoteClose (this=0x4199ba0, status=Envoy::Grpc::Status::Ok,
    message=...) at source/common/grpc/async_client_impl.cc:260
#3  0x0000000000bf9da0 in Envoy::Grpc::AsyncStreamImpl::onTrailers (this=0x4199ba8, trailers=...)
    at source/common/grpc/async_client_impl.cc:160
#4  0x0000000000c00a15 in Envoy::Http::AsyncStreamImpl::encodeTrailers (this=0x4282c00, trailers=...)
    at source/common/http/async_client_impl.cc:111
#5  0x0000000000ba6129 in Envoy::Http::StreamDecoderWrapper::decodeTrailers (this=0x4242920, trailers=...)
    at bazel-out/k8-opt/bin/source/common/http/_virtual_includes/codec_wrappers_lib/common/http/codec_wrappers.h:44
#6  0x0000000000c5c411 in Envoy::Http::Http2::ConnectionImpl::onFrameReceived (this=0x424a878, frame=0x4065598)
    at source/common/http/http2/codec_impl.cc:470
#7  0x0000000000c6739c in session_call_on_frame_received (frame=0x4065598, session=0x4065400)
    at /build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:3295
#8  session_after_header_block_received (session=0x4065400)
    at /build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:3796
#9  nghttp2_session_mem_recv (session=0x4065400, in=0x4270148 "", inlen=<optimized out>)
    at /build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/execroot/ci/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:6254
#10 0x0000000000c5d6a7 in Envoy::Http::Http2::ConnectionImpl::dispatch (this=0x424a878, data=...)
    at source/common/http/http2/codec_impl.cc:360
#11 0x0000000000bf0dc1 in Envoy::Http::CodecClient::onData (this=0x424a7e0, data=...) at source/common/http/codec_client.cc:116
#12 0x0000000000bf0f9d in Envoy::Http::CodecClient::CodecReadFilter::onData (this=<optimized out>, data=...)
    at bazel-out/k8-opt/bin/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:167
#13 0x0000000000a9bfac in Envoy::Network::FilterManagerImpl::onContinueReading (this=this@entry=0x4283420,
    filter=filter@entry=0x0) at source/common/network/filter_manager_impl.cc:56
#14 0x0000000000a9c07c in Envoy::Network::FilterManagerImpl::onRead (this=this@entry=0x4283420)
    at source/common/network/filter_manager_impl.cc:66
#15 0x0000000000a9881a in Envoy::Network::ConnectionImpl::onRead (read_buffer_size=584, this=0x4283400)
    at source/common/network/connection_impl.cc:263
#16 Envoy::Network::ConnectionImpl::onReadReady (this=this@entry=0x4283400) at source/common/network/connection_impl.cc:497
--Type <RET> for more, q to quit, c to continue without paging--
#17 0x0000000000a98f2a in Envoy::Network::ConnectionImpl::onFileEvent (this=0x4283400, events=<optimized out>)
    at source/common/network/connection_impl.cc:473
#18 0x0000000000a92bea in std::function<void (unsigned int)>::operator()(unsigned int) const (__args#0=<optimized out>,
    this=<optimized out>) at /usr/include/c++/7/bits/std_function.h:706
#19 Envoy::Event::FileEventImpl::<lambda(int, short int, void*)>::operator() (__closure=0x0, arg=<optimized out>,
    what=<optimized out>) at source/common/event/file_event_impl.cc:64
#20 Envoy::Event::FileEventImpl::<lambda(int, short int, void*)>::_FUN(int, short, void *) ()
    at source/common/event/file_event_impl.cc:65
#21 0x0000000000dc8489 in event_process_active_single_queue.isra ()
#22 0x0000000000dc89bf in event_process_active ()
#23 0x0000000000dca5c8 in event_base_loop ()

bug

Source