Envoy: Improve documentation around running Envoy in Docker container

[phlax]

description

a conversation came up in the slack channel around running Envoy in (non-root) docker containers.

i think most of the relevant information is there (https://www.envoyproxy.io/docs/envoy/latest/start/start#using-the-envoy-docker-image) but the section could do with being broken up a bit and possibly moved

[supasteev0]

Sharing my experience in migrating to 1.15 in a Kubernetes environment :

• Documentation should maybe stress out that the container has to be run as root, and that its'only the envoy process inside the container that will be run as non root.
  Because i actually thought the pod should be run as non root user, hence came up with a lot of issues due to missing docker capabilities and enforced security policies in Kubernetes.
• Also, it's not very clear that the port has to be above > 1024. I would make this information clearer, not just by showing an example, and specify that the port_value in envoy.yaml should match the container port (I think default value in example config is always 80, so this has to be changed as well)

Here is an example deployment that will work in Kubernetes with 1.15, running with default uid & gid = 101.
containerPort is the most important change

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy
  labels:
    app: envoy
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: envoy
  template:
    metadata:
      labels:
        app: envoy
    spec:
      containers:
      - name: envoy
        imagePullPolicy: IfNotPresent
        image: envoyproxy/envoy:v1.15.2
        args:
        - --config-path /etc/envoy/envoy.yaml
        - -l info
        ports:
        - name: ingress
          containerPort: 8080
          protocol: TCP
        - name: admin
          containerPort: 9901
          protocol: TCP
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 9901
            scheme: HTTP
          periodSeconds: 30
        volumeMounts:
          - name: envoy-config
            mountPath: /etc/envoy/
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

And Envoy config file should also reflect the port change:

static_resources:
  listeners:
  - name: ingress
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
...

If I may, I really think that some example Kubernetes manifests would be a very nice to have in the Envoy github repository.

[phlax]

thanks @supasteev0 im updating docs around this anyway - so ill take on board and incorporate your suggestions on how we can improve them.

feel free to PR any specific changes that you think would help

[phlax]

/assign phlax