Envoy: Segmentation fault when upstreaming large files

Created on 24 Jan 2019 · 7Comments · Source: envoyproxy/envoy

We have envoy as edge proxy
and it's config :

  GNU nano 2.5.3                                                                    File: config.yaml                                                                                                                                               

admin:
  access_log_path: /var/log/envoy/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 15000
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          use_remote_address: true
          xff_num_trusted_hops: 2
          access_log:
          - name: envoy.file_access_log
            config:
              path: /var/log/envoy/access-prod.log
          route_config:
            virtual_hosts:
            - name: stage
              domains:
              - "prod-aaa.bbb.cc"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: nginx1
                  timeout: 0s
          http_filters:
            name: envoy.router
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          use_remote_address: true
          xff_num_trusted_hops: 2
          access_log:
          - name: envoy.file_access_log
            config:
              path: /var/log/envoy/access-stage.log
          route_config:
            virtual_hosts:
            - name: stage
              domains:
              - "stage-aaa.bbb.cc"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: nginx2
                  timeout: 0s
          http_filters:
            name: envoy.router
  clusters:
  - name: nginx1
    type: Strict_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    connect_timeout: 1s
    load_assignment:
      cluster_name: nginx1
      endpoints:
      - priority: 0
        lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: "8080"
  - name: nginx2
    type: Strict_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    connect_timeout: 1s
    load_assignment:
      cluster_name: nginx2
      endpoints:
      - priority: 0
        lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: "8081"

nginx has file file5.txt with 5 MB size

when we trying to get it
curl http://prod-aaa.bbb.cc/file5.txt
curl: (18) transfer closed with 4913369 bytes remaining to read
vDSRPV%
And we have envoy logs:

[2019-01-23 08:00:31.126][1][info][config] [source/server/listener_manager_impl.cc:997] all dependencies initialized. starting workers
[2019-01-23 08:00:34.271][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<11> obj<envoy> (If unsymbolized, use tools/stack_decode.py):
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #0 0x982fc8 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #1 0x7dacf9 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #2 0x7dab00 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #3 0x998706 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #4 0x990bfc (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #5 0x995e44 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #6 0x99593d (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #7 0xae7919 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #8 0x9970e0 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #9 0x998f22 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #10 0x93af2e (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #11 0x93b0fc (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #12 0x7de046 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #13 0x7daf9e (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #14 0x7db6c5 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #15 0x7d5467 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #16 0xb28b14 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #17 0xb2925e (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #18 0xb2bfe7 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #19 0x7d45a2 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #20 0x7d08a8 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #21 0xdcaac4 (unknown)
[2019-01-23 08:00:34.272][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<11> obj</lib/x86_64-linux-gnu/libpthread.so.0>
[2019-01-23 08:00:34.273][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<11> #22 0x7f5f87ae26b9 start_thread
[2019-01-23 08:00:34.273][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<11> obj</lib/x86_64-linux-gnu/libc.so.6>
[2019-01-23 08:00:34.273][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<11> #23 0x7f5f8750f41c (unknown)
[2019-01-23 08:00:34.273][11][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 11

if secondary filter

filter_chains:

filters:

name: envoy.http_connection_manager

"stage-aaa.bbb.cc"

was been deleted - all works fine

Issue triggered o 1.8.0, 1.9.0 and 1.10.0-dev

Envoy was been started as a dockered systemd-service:

[Unit]
Description=Envoy-proxy-docker
After=docker.service
Requires=docker.service

[Service]
Type=simple
User=root
Restart=always
RestartSec=10
ExecStart=/usr/bin/docker run --restart=always --name=envoy_proxy  --network host \
        -v /etc/envoy/envoy.yaml:/etc/envoy/envoy.yaml \
         envoyproxy/envoy:v1.9.0
RemainAfterExit=true
ExecStop=/usr/bin/docker  stop envoy_proxy
ExecStopPost=/usr/bin/docker rm -f envoy_proxy
ExecReload=/usr/bin/docker restart envoy_proxy

[Install]
WantedBy=multi-user.target

bug stale

Source

turbotankist

All 7 comments

Can you provide a core dump w/ symbols or at least a usable backtrace with symbols? Thank you.

mattklein123 on 24 Jan 2019

How can I set to backtrace with symbols? Only this is in output

turbotankist on 25 Jan 2019

https://drive.google.com/file/d/1xxV3wZhQnbstWyhcBJrwnI-aGcYgR0Qk/view?usp=sharing
core_dump

turbotankist on 25 Jan 2019

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] on 24 Feb 2019

👎2

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

stale[bot] on 3 Mar 2019

Bump, bot!