Kubernetes-ingress: Nginx server behind nginx ingress

[toddams]

We decided to move our apps from Service:LoadBalancer to Ingress, and I've chosen Nginx Ingress Controller, as I'm familiar with it, and because it's one of the most popular controllers in Kubernetes world

Previously we had Nginx => Uwsgi combination that stands behind ELB. We compile nginx from source, as we need some 3rd party modules and Lua support.

ELB => Nginx Server => UWSGI
ELB => Nginx Ingress (Load Balancer) => Nginx (Server) => UWSGI

My question is: is it okay to have 2 nginx in a proxy chain?

I understand that one plays the role of LoadBalancer, and another is a server itself. But for me it comes with a pain, like if I change some options in Server nginx.conf, like increase the size of client body to 8MB, I should do the same on Nginx-Ingress. Also I'm wondering how to set timeouts: as there is a timeout between ingress=>server and server=>uwsgi, and in general how to tune the performance while having 3 proxies before request hits the app?

Is it a good practice to remove Nginx Server, so Ingress Controller acts like a server and loadbalancer at the same time? What about 3rd party modules that we use?

[pleshakov]

The approach with having an NGINX load balancer in front of NGINX reverse proxies is very common.

Depending on the custom logic you had implemented with 3rd party modules, it makes sense to either move that logic to the Ingress controller or leave it in NGINX reverse proxies.

To improve performance, I suggest enabling keepalive between NGINX Ingress controller and NGINX proxies, see nginx.org/keepalive annotation here.

For timeouts, the timeouts between an NGINX reverse proxy and an UWSGI app should be less than the timeouts between an NGINX Ingress and NGINX reverse proxies.

[achandak123]

Hi @pleshakov, I am trying to use the ingress controller as the web-server as well. However, i am not able to figure out how to set up something like this:

server {
    location / {
        root /data/www;
    }
}

Basically, for default backend case, i want to serve some static files from /data/www. How do i achieve this?

[pleshakov]

@achandak123 the Ingress controller is not designed to serve static content. If you'd like to serve static content, consider having a separate Service for it:

 - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80
      - path: /
        backend:
          serviceName: static-content
          servicePort: 80

[RumataEstor]

The approach with having an NGINX load balancer in front of NGINX reverse proxies is very common.

@pleshakov
What benefits gives a cascade of reverse proxies where one is of "load balancer" kind and another is "static content web server" running exactly the same software and the second redoing the work first have already did?

To me this "common" way looks possible but not necessarily the best one from the configuration and performance points of view.

[CharlieReitzel]

I agree with @RumataEstor, it is a common and time proven optimization for web applications to push static content as close to the external edge of the network as possible. Indeed, with CDNs, outside of host network, and as close to the consumer as possible. Would this be possible with a custom Ingress template? An NGINX Ingress, is already a reverse proxy, so all that remains is to make the storage location with the static content available to the Ingress process. Again, the storage is easily configured for the cluster, so the glue necessary seems reasonable.

[pleshakov]

@CharlieReitzel

Would this be possible with a custom Ingress template?

you can try to use the snippets annotations to serve files from the disc. Consider the example below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress-with-annotations
  annotations:
    nginx.org/server-snippets: |
      location /static/ {
        root /data;
      }
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

In this case requests with a URL like /static/image.png will be served from the /data path on the disk.

See also
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
https://docs.nginx.com/nginx/admin-guide/web-server/web-server/