Calico: Pods not seeing external source IP

[holmesb]

External workstation --> Ingress controller pod --> application pod

In the above scenario, only the Ingress controller logs (nginx) show the true external IP address of the inbound connection from external workstation. The application pod's logs do not, and instead show the inbound connection as from the ingress controller IP. This is a problem because we use a single ingress-controller that serves our entire cluster, which means network policies can only source-ip-filter the whole cluster, not individual apps.

Above flow is simplified. Is a k8s Service (type = LoadBalancer) in front of the Ingress Controller that is in "Local" traffic mode (spec.externalTrafficPolicy = Local).

Expected Behavior

Pods can see incoming connections from true source IP, not snat IP of ingress controller. Or otherwise pls suggest how it is possible to use Network Policies to restrict access to an app in k8s, so only a whitelist of external IPs are allowed, when an Ingress controller is installed.

• Calico version: v3.7.3
• Orchestrator version: k8s 1.15.3 (bare-metal, on-prem)
• Operating System and version: Ubuntu 18.04

[edit] It seems the app pod does receive the true source IP address, but it is in the http_x_forwarded_for header field, not remote_addr that Network Policies use.

[edit2 - 14/10] Network Policies seem to obtain source IP from elsewhere than the HTTP header. I've set remote_addr to the real source-ip, but NPs still do not function (traffic is only allowed when the ingress-controller IP is whitelisted).

[rafaelvanoni]

@holmesb Could you clarify this bit ?

Is a k8s Service (type = LoadBalancer) in front of the Ingress Controller that is in "Local" traffic mode (spec.externalTrafficPolicy = Local).

Do you mean that the ingress controller the same thing as the load balancer? If not, the extra hop might explain the behavior you're seeing.

Wrt where the source IP comes from, Calico operates at level 3 (IP) and not level 7 (application).

[holmesb]

Hi @rafaelvanoni, it's a standard implementation of nginx ingress. Traffic arrives at the cluster to the ingress controller's Service ("ingress-nginx-nginx-ingress-controller") that is of type LoadBalancer. Behind that Service sits the ingress controller pod.

[koehn]

I’m in the same boat. I have a k8s cluster on a set of nodes with Calico, and nginx shows arriving traffic has the source IP of the node instead of the external IP address. I really like Calico, but I’d really like to resolve this.

[rafaelvanoni]

I don't think this is really possible with nginx as the ingress controller since that's where the connection is terminated. I'd suggest looking at other controllers that do preserve the source IP, if possible.

@spikecurtis also mentioned that if you're running on prem, you could use one of our new features for advertising the cluster IP over BGP with a router that supports ECMP to load balance.

[koehn]

FWIW, something changed in either nginx and/or calico, and I'm now able to get external IP addresses in nginx on k8s.