Contour: Run envoy pod as non-root

[mkilchhofer]

Please describe the problem you have
Referring to the blog post at kubernetes.io > 11 Ways (Not) to Get Hacked we should try running envoy as a non root user.

I already tried to specify a container level securityContext on the envoy container inside the Envoy Pod allowing the capability NET_BIND_SERVICE:

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]
  runAsUser: 1001
  allowPrivilegeEscalation: false

But I had no success:

[2020-09-14 10:51:35.921][1][warning][misc] [bazel-out/k8-opt/bin/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.router' for 'envoy.filters.http.router'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-09-14 10:51:35.922][1][info][upstream] [source/server/lds_api.cc:74] lds: add/update listener 'stats-health'
[2020-09-14 10:51:35.922][1][info][config] [source/server/listener_manager_impl.cc:844] all dependencies initialized. starting workers
[2020-09-14 10:51:35.923][1][warning][config] [source/common/config/grpc_subscription_impl.cc:100] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) ingress_http: cannot bind '0.0.0.0:80': Permission denied
ingress_https: cannot bind '0.0.0.0:443': Permission denied

[2020-09-14 10:51:35.925][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x10
[2020-09-14 10:51:35.925][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-09-14 10:51:35.925][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 50ef0945fa2c5da4bff7627c3abf41fdd3b7cffd/1.15.0/Modified/RELEASE/BoringSSL
[2020-09-14 10:51:35.927][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #0: [0x5652b1c8dd86]
[2020-09-14 10:51:35.927][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #1: [0x5652b1c915fa]
[2020-09-14 10:51:35.928][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x5652b2582f13]
[2020-09-14 10:51:35.929][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x5652b2582fec]
[2020-09-14 10:51:35.929][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x5652b2579a2d]
[2020-09-14 10:51:35.929][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x5652b2a66bc5]
[2020-09-14 10:51:35.930][13][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: start_thread [0x7fe1637e3fa3]

The other option in my opinion is to promote other listeners though the xDS API and use 8080/8443 for example. Since you use a service for exposing the envoys outside kubernetes (when using a loadbalancer) and hostPorts (when exposing directly on every host) we then are able to "translate" port 80/443 to the real ports used inside the container.

[youngnick]

Yes, you can definitely do that - it's a little confusing though because you have to tell Contour that Envoy should be configured on 8080 and 8443 as well as set up the Daemonset and Service wiring.

Contour will actually default to using those ports for Envoy's HTTP and HTTPS listener respectively, in our example YAMLs, we override that using --envoy-service-http-port and --envoy-service-https-port`.

That will finish up the network parts, and allow the requests to flow correctly. I'll be curious to know if everything works after that.

Once you're all done, please consider writing this up as a Guide in site/_guides, it would be great to have more documentation of this.

[mkilchhofer]

Hi @youngnick,
Thanks for explaining it. I knew I already saw these params while I was working on the bitnami helm chart improvements for contour.
I already have a working non-root setup with the bitnami chart: https://github.com/bitnami/charts/pull/3674.

Regarding writing this down in a guide: I will do it as far as I find some time to do this :+1:

[binnythomas-1989]

Guys I tried the same using @mkilchhofer helm chart changes that he made. Had to do some more stuff mentioned here https://github.com/bitnami/charts/pull/3674 to get it working. But The envoy container on 8443 says connection refused.

I have no name!@contour-ingress-envoy-csbdw:/$ curl localhost:8080
I have no name!@contour-ingress-envoy-csbdw:/$ curl localhost:8443
curl: (7) Failed to connect to localhost port 8443: Connection refused

[youngnick]

Contour won't open either the secure or insecure ports unless you have something that needs it. Do you have a Ingress or HTTPPRoxy object that requests TLS configured @binnythomas-1989?

[binnythomas-1989]

Yes this is sorted @youngnick It only opens one when we created an ingress resource. Can you check this https://github.com/projectcontour/contour/issues/2922

[jpeach]

xref https://github.com/kubernetes/kubernetes/issues/56374

[embano1]

Had a similar use case and created this patch:

diff --git a/examples/contour/03-envoy.yaml b/examples/contour/03-envoy.yaml
index 9265b74a..2e0283d9 100644
--- a/examples/contour/03-envoy.yaml
+++ b/examples/contour/03-envoy.yaml
@@ -23,6 +23,8 @@ spec:
       labels:
         app: envoy
     spec:
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       containers:
       - command:
         - /bin/contour
@@ -51,6 +53,9 @@ spec:
         - --service-cluster $(CONTOUR_NAMESPACE)
         - --service-node $(ENVOY_POD_NAME)
         - --log-level info
+        securityContext:
+          runAsUser: 101
+          allowPrivilegeEscalation: false
         command:
         - envoy
         image: docker.io/envoyproxy/envoy:v1.16.1
@@ -104,6 +109,9 @@ spec:
         - --envoy-cafile=/certs/ca.crt
         - --envoy-cert-file=/certs/tls.crt
         - --envoy-key-file=/certs/tls.key
+        securityContext:
+          runAsUser: 101
+          allowPrivilegeEscalation: false
         command:
         - contour
         image: docker.io/projectcontour/contour:main

Ignore the hostNetwork stuff as it's just an example for deploying in non-LB environments but still running as non-root as one workaround to avoid CVE-2020–15257.

I can file a PR for these changes if it makes sense.

[youngnick]

I'd love for Contour to be confirmed able to run as non-root, but I'm concerned about interactions with the standard installs. Does this interact with the currently-default hostPort setup? Or if we swap to NodePort?

Aside from that small concern, I'd love to see this in - running as root is not great, and remapping the userns by default seems much better. This would certainly need a changelog note though.