Contour: No way to log the real client IP on Google Kubernetes Engine

[yob]

Applications behind contour/envoy currently have no way to know the IP address of the real client - they only see the IP of the GCP load balancer.

This manifest as logs with a GCP private IP as the client IP:

10.152.0.4 - - [12/Jul/2018:05:25:37 +0000] "POST /examples HTTP/1.1" 201 - 0.0049

GCP L4 Load Balancers support a PROXY mode that can provide these details to contour/envoy, and contour/envoy supports PROXY mode on AWS, however currently there's no way for contour to create the Load Balancer with PROXY mode enabled on GCP.

[davecheney]

Thanks for raising this issue. At the moment contour supports using the proxy protocol on listeners but the Kubernetes Service document lacks support for an annotation to enable this on GCP load balancers.

[yob]

I think we can close this, it's more of a @yob-is-still-learning-kube-issue than a GCP limitation.

The contour pods in a cluster are exposed to the Internet via a service with type: LoadBalancer. That creates a GCP Network Load Balancer, which doesn't terminate TCP connections and shouldn't mask the client IP.

However, by default a GKE service of type: LoadBalancer results in all nodes in the cluster accepting incoming connections on behalf of the service. Not all nodes have the contour pods, so there's a internal kubernetes process that proxies the request to a contour pod. that proxy is what's masking our client IPs.

Kube has some decent docs on accessing source ips, and it turns out the client IP should be accessible to contour if we add externalTrafficPolicy: Local to the Service. Something like:

apiVersion: v1
kind: Service
metadata:
 name: contour
spec:
 ports:
 - port: 80
   name: http
   protocol: TCP
   targetPort: 8080
 - port: 443
   name: https
   protocol: TCP
   targetPort: 8443
 selector:
   app: contour
 type: LoadBalancer
 externalTrafficPolicy: Local

That changes the GCP load balancer to only direct traffic to GKE nodes that have a contour pod running on them, which removes the need for an internal proxy.

I've tested this on an experimental cluster and confirmed that a ruby app behind contour can see (and log) the real client IP when externalTrafficPolicy is set to Local.