Contour: Envoy segfaults and errors when ExternalName service is defined

[alexanderturner]

What steps did you take and what happened:
When executing the below code to define an externalName service all Envoy pods segfault and error.

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: externaltest
  namespace: default
spec:
  routes:
    - services:
      - name: externaltest
        port: 443
      requestHeadersPolicy:
        set:
        - name: Host
          value: projectcontour.io
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: externaltest
  name: externaltest
  namespace: default
  annotations:
    projectcontour.io/upstream-protocol.tls: 443,https
spec:
  externalName: projectcontour.io
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  type: ExternalName
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: externaltest
  namespace: projectcontour
spec:
  secretName: externaltest
  commonName: "externaltest.mydomain.com"
  dnsNames:
  - "externaltest.mydomain.com"
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: externaltest
  namespace: projectcontour
spec:
  virtualhost:
    fqdn: externaltest.mydomain.com
    tls:
      secretName: externaltest
  includes:
    - name: externaltest
      namespace: default

Logs from an Envoy pod:

[2021-01-10 01:55:23.463][1][critical][main] [source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 8fb3cb86082b17144a80402f5367ae65f06083bd/1.16.0/Clean/RELEASE/BoringSSL
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #0: [0x55608e88fe0b]
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #1: [0x55608e88fc79]
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x55608ee9ef93]
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x55608e748542]
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x55608e2dbcfd]
[2021-01-10 01:55:23.463][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x55608e0c31d5]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #6: [0x55608e0c6773]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #7: [0x55608e0c5b15]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #8: [0x55608e0c634b]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #9: [0x55608e0c4d50]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #10: [0x55608e2de8a5]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #11: [0x55608e2d8366]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #12: [0x55608e710fe8]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #13: [0x55608e70f9be]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #14: [0x55608e2bb79c]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #15: [0x55608d619008]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #16: [0x55608d619807]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #17: [0x55608d617bdc]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: __libc_start_main [0x7f1979649b97]
[2021-01-10 01:55:23.464][1][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x1

Pods:

kubectl get pods -n projectcontour                                    389ms  Sun 10 Jan 13:15:25 2021
NAME                            READY   STATUS      RESTARTS   AGE
contour-8556b76b45-h2g5k        1/1     Running     0          8m56s
contour-8556b76b45-xm5ht        1/1     Running     0          8m56s
contour-certgen-v1.11.0-f5zs9   0/1     Completed   0          8m56s
envoy-7vbvh                     1/2     Error       1          8m56s
envoy-9bdzh                     1/2     Error       1          8m56s
envoy-xmnf8                     1/2     Error       1          8m56s

What did you expect to happen:
Service is created successfully

Environment:

• Contour version:
  -1.11.0
• Kubernetes version: (use kubectl version):
  Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.14", GitCommit:"f238f5142728be4033c37aa0ad69bf806090beae", GitTreeState:"clean", BuildDate:"2020-11-11T13:03:54Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version:
  Rancher K8S latest stable
• Cloud provider or hardware configuration:
  self hosted, 3 node vmware cluster
• OS (e.g. from /etc/os-release):
  Ubuntu 20.04

[youngnick]

Thanks for logging this @alexanderturner. I'll ping @stevesloka as he has the most experience with ExternalName services. Any thoughts Steve? Something about TLS ExternalNames maybe?

I've marked this as "Needs investigation" because we haven't seen anything like this before.

[sunjayBhatia]

Noticed the Envoy version you're using is 1.16.0, might not be the issue but there should be some crashes fixed in 1.16.1 and more bug fixes in 1.16.2: https://www.envoyproxy.io/docs/envoy/v1.17.0/version_history/v1.16.1

[alexanderturner]

@sunjayBhatia interesting. This was just installed from the quickstart manifests. I'll patch it later today

[sunjayBhatia]

@alexanderturner did you happen to build Envoy yourself or is this from the upstream image? just wondering in case we can decode the stack trace

[alexanderturner]

Upstream image. I just ran the quick start manifest here:
https://projectcontour.io/getting-started/#option-1-quickstart

[sunjayBhatia]

Huh, not sure if I'm overlooking something, but the Envoy stack trace decode script is not working on the above trace, running addr2line directly is not either 🤔

[stevesloka]

I'm not sure why you're getting the segfault @alexanderturner. Testing on my local cluster with your configs I don't get them.

Could you take out the TLS termination on Contour just to remove the dependency on CertManage? I don't think that will change anything, but good to remove more pieces.

When I curl against an upstream of projectcontour.io, I get 404s from Netlify (where projectcontour.io is hosted):

$ curl -i local.projectcontour.io/                                                                                                                        [9:27:05]
HTTP/1.1 404 Not Found
cache-control: private, max-age=0
content-length: 69
content-type: text/plain; charset=utf-8
date: Tue, 12 Jan 2021 14:27:14 GMT
age: 0
server: envoy
x-nf-request-id: 962a236e-1df9-4f43-a12d-0006f48793f9-27277692
x-envoy-upstream-service-time: 150
vary: Accept-Encoding

Not found - Request ID: 962a236e-1df9-4f43-a12d-0006f48793f9-27277692%                                                                                                                               

If I change the domain to my own (stevesloka.com), I get the 421 redirect from Envoy which is programmed in Contour to solve a different problem so it seems something has changed in Envoy or Contour, but I don't think anything in Contour will be to blame.

In any case let me poke through this a bit more and we can dig down to the root cause of your issue.

[stevesloka]

So quick update, I think this might be a regression in Contour v1.11, testing with v1.10 it works fine. I'm still not able to repo your segfault unfortunately.

[stevesloka]

In testing I found the switch to xDS v3 chose the wrong host rewrite proto. I fixed that in #3252, but still doesn't explain how you got to the seg fault. I'd try reducing all the config to the bare minimum and then add bits in to find out what might be causing it, but I'm unable to reproduce.

[skriss]

@alexanderturner we just released Contour 1.12 (https://github.com/projectcontour/contour/releases/tag/v1.12.0) yesterday; would you be able to re-test with that version?

[skriss]

@alexanderturner checking in once more to see if you were able re-test with the latest Contour. We've added ExternalName integration tests recently as well which seem to be working fine.

[alexanderturner]

Hey @skriss looks like this is working now, at least no longer crashing. Thanks for the quick action. Just confirming that the ExternalName currently only supports external endpoints that present HTTP only? My external services are HTTPs and don't appear to be working.

(Understand this is not ExternalName, however I was seeing similar behaviour with ExternalName)

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: proxmox
  namespace: default
spec:
  routes:
    - services:
      - name: proxmox
        port: 443
---
apiVersion: v1
kind: Endpoints
metadata:
  name: proxmox
subsets:
- addresses:
  - ip: 192.168.2.254
  ports:
  - name: app
    port: 8006
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: proxmox
  name: proxmox
  namespace: default
  annotations:
    projectcontour.io/upstream-protocol.tls: 8006,https
spec:
  ports:
  - name: app
    port: 8006
    protocol: TCP
    targetPort: 8006
  type: ClusterIP
  clusterIP: None
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: proxmox
  namespace: projectcontour
spec:
  virtualhost:
    fqdn: proxmox.mydomain.com
    tls:
      secretName: wildcard
  includes:
    - name: proxmox
      namespace: default

[skriss]

@alexanderturner HTTPS ExternalName services should work fine too. Let me try out something similar to your setup to confirm.

[skriss]

Just looking at this again, looks like your HTTPProxy specifies a port of 443 but your Service is using 8006, those would need to match.

The other thing I see is that you have an annotation on the service of projectcontour.io/upstream-protocol.tls: 8006,https -- the value(s) of that annotation should be the port name or number, so in this case either 8006 or app (the name of your port). https is not the name of a port so is not valid. Also, you don't need to specify both the number and name; either one should be fine.

[alexanderturner]

Right you are; I missed that. I worked this out and put together an article about it so hopefully others don't get stuck. https://alexpturner.medium.com/exposing-external-services-in-contour-5a4adcd3dfbd

Thanks for the help