Ingress-nginx: Service upstream support with Istio

[Stono]

Hey folks,
Please see https://github.com/istio/istio/issues/9107. As you know I'm a big advocate of using ingress-nginx and istio in unison however I'm still having some problems making it truly transparent.

Historically we have used the following annotations to ensure that nginx doesn't try to send traffic to a pod directly by its IP, rather it sends it to the cluster service address instead and then istio-proxy will decide which node it goes to. This is because istio needs to know the service upstream that the request is targeting.

nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/upstream-vhost: your-app.namespace.svc.cluster.local

This works fine, however we've recently deployed an application which looks at the Host header. The issue with the above implementation is that the Host header gets rewritten to the internal cluster address, thus really confusing the upstream application.

"host": "your-app.namespace.svc.cluster.local",
"x-forwarded-host": "your-app.our-cluster.env.k8.atcloud.io",
"x-forwarded-port": "443",
"x-forwarded-proto": "https",
"x-original-uri": "/",

We could not not use nginx.ingress.kubernetes.io/service-upstream, and have the VirtualService in istio configured with the your-app.our-cluster.env.k8.atcloud.io host, however then nginx will constantly be watching and reloading config when pods change, and making routing decisions which are irrelevant (because istio-proxy will intercept them anyway and make another routing decision).

I did try doing:

nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/upstream-vhost: your-app.namespace.svc.cluster.local
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header Host "your-app.our-cluster.env.k8.atcloud.io";

But nginx 404's

Ideally the behaviour we need for a truly transparent proxy that is compatible with Istio is to have the above two headers used only to look up the service IP, but then still use the original Host header when making the request to that IP as part of the proxy_pass.

Perhaps nginx.ingress.kubernetes.io/upstream-vhost-preserve-host or something? What do you think?

I mean really I need that Host header to be your-app.our-cluster.env.k8.atcloud.io, regardless of if we're using nginx.ingress.kubernetes.io/service-upstream or not.

[aledbf]

@Stono if you remove nginx.ingress.kubernetes.io/upstream-vhost: your-app.namespace.svc.cluster.local you should see the behavior you mentioned. It will use the service IP and not touch the Host header

[Stono]

Shit, it does. You have no idea how long we've been suffering with this! I need to write a blog post for others! Thank you!

[aledbf]

@Stono can we close this issue?

[ermik]

@Stono is there an Istio doc we can update with this TIL?

[varungbt]

Shit, it does. You have no idea how long we've been suffering with this! I need to write a blog post for others! Thank you!

@Stono can you plz share how you setup the route istio ingressgateway --> nginx ingress controller --> k8s service ?

I tried to use the annotation nginx.ingress.kubernetes.io/service-upstream: "true" on the ingress object. I am able to route the traffic to the service without issues, however no telemetry data is recorded for the traffic between nginx ingress controller pod and the k8s service. Did you hit a similar note ?

[cbenjemaa]

@varungbt
If you want to keep Nginx as your ingress, the setup you're showing won't work ingressgateway --> nginx ingress controller --> k8s instead you should aim for something like this nginx ingress controller --> k8s.
Because istio by default changes the IPTABLES rules to force inbound and outbound traffic to go through the injected sidecar container. Knowing that the nginx ingress won't be in the front of your architecture. So we need to tweak how istio initialize pods.

To do so you add traffic.sidecar.istio.io/includeInboundPorts: "" annotation to Nginx ingress deamonset.yaml like so:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: my-nginx-ingress
spec:
  updateStrategy: 
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  minReadySeconds: 10
  template:
    metadata:
      labels:
        k8s-app: my-nginx-ingress
        name: my-nginx-ingress
      annotations:
        traffic.sidecar.istio.io/includeInboundPorts: ""
    spec:
.....

Hope that helps.