Kubernetes-ingress: Securing ingress with TransportServer

[RazaGR]

I recently setup a k8 cluster successfully and used TransportServer to open few TCP ports required for my custom apps to work, now I was at the point of securing the cluster and was unable to find any information if I could use ModSecurity in docs..

After reading docs I came up with solution by creating custom annotations custom.nginx.org/block-ips: "79.167...." and than tried to output below loop in main-templatevalue as I wanted to available this restriction globally via http {}block:

{{range $ip := split (index $.Ingress.Annotations "custom.nginx.org/blocked-ips") ","}}
     deny {{trim $ip}};
{{end}}

I encountered error that split isn't defined. I moved that range loop in ingress-template config value and it worked, I could see in /etc/nginx/conf.d/nginx-ingress-cafe-ingress.conf that it is listing my given IP in location block but it doesn't block the IP,

   location / { 
    deny 79.167....;

so I have few questions:

1 ) why it is not blocking HTTP traffic even though I have setup deny that IP address?
2 ) is split helper function not available in main-template config value?
3 ) is ModSecurity not available in Nginx ingress? ( I am using simple nginx, not plus)
4 ) how can I block some IP addresses to TransportServer? TCP/UDP? which could be any PORT

Thanks <3

[pleshakov]

Hi @RazaGR

1 ) why it is not blocking HTTP traffic even though I have setup deny that IP address?

That should have worked!

Could you possibly share the generated config?
Could you show the access log line that corresponds to the request that wasn't blocked?
Also, could you double check that the config was successfully applied by looking at the events https://docs.nginx.com/nginx-ingress-controller/troubleshooting/#checking-the-events-of-an-ingress-resource ?

2 ) is split helper function not available in main-template config value?

That function is only available in the Ingress templates. Moreover, from the main template, it is not possible to access Ingress annotations.

To add allow/deny rules to the http context, I suggest using the http-snippets configmap key.

3 ) is ModSecurity not available in Nginx ingress? ( I am using simple nginx, not plus)

It is possible. However, you need to build the modsecurity module and add it to the image. Some references:

• The instructions on how to use ModSecurity with NGINX OSS are here https://www.nginx.com/blog/compiling-and-installing-modsecurity-for-open-source-nginx/
• Here is an example of how to add a third-party module (Opentracing) to NGINX OSS image -- https://github.com/nginxinc/kubernetes-ingress/blob/master/build/DockerfileWithOpentracing
• To load module, you can use the main-snippet configmap key. For example, main-snippets: load_module modules/ngx_http_opentracing_module.so;

4 ) how can I block some IP addresses to TransportServer? TCP/UDP? which could be any PORT

You can use stream-snippets configmap key to insert something like below:

  stream-snippets: |
    deny 127.0.0.1;
    allow all;

That wlll deny connection from localhost IP address for all TransportServers. Unfortunately, it is not possible to define allow/deny rules per TransportServer.

[RazaGR]

Hi @pleshakov
A huge thanks for all those infos:

For 1) I will test and reply later, maybe it is because of using proxy_protocol
2) ok
4) ok

3) Thank you for those links, I just finished the build, it actually took me hours to compile ModSecurity and configure with owasp, Ingress is running successfully now, I will continue tomrrow

[RazaGR]

1) I changed set-real-ip-from to my private networking and its now blocking IP, I thought we could use only the LoadBalancer's IP, instead of private network?

set-real-ip-from: "10.114.0.0/20"

2)
p.s, I wanted to know is there any possibility to reload nginx? I am loading owasp rules remotely using modsecurity_rules_remote ,so when I have new rules I would like to reload nginx and download new owasp rules. For now I just see either updating ConfigMap.yaml file or by delete Nginx-ingress pod.

3)
also while I setup ModSecurity, I noticed I was receiving some suspicious activities from IP 10.114.0.4, even though I was seeing client IP of other requests in nginx logs on all my services, upon investigating further I found someone was scanning my laodbalancer IP address, so I had to use main-template and add these values hardcoded to server_name _; server { block

        server {
               .....
                set_real_ip_from 10.114.0.0/20;
                real_ip_header proxy_protocol;

                server_name _;
               .....
                {{end}}

        }

I think it should be a feature to support these values to get from ConfigMap? so I don't have to hardcode set-real-ip-from real_ip_header and server-snippets values in server_name _; of main configuration file?

4)
also is it possible to block any kind of traffic on the main loadbalncer IP? right now I am using:

server_name _;
 location / {
          deny all;
          return 444;
  }

[pleshakov]

@RazaGR

1. I changed set-real-ip-from to my private networking and its now blocking IP, I thought we could use only the LoadBalancer's IP, instead of private network?

Not sure I understood the question.
Using set-real-ip-from combined with proxy_protocol enables NGINX to see the real IP addresses of clients. The set-real-ip-from is required so that NGINX can trust the load balancer in front of it. For set-real-ip-from you can use either the private IP of the load balancer or the subnet.

1. p.s, I wanted to know is there any possibility to reload nginx? I am loading owasp rules remotely using modsecurity_rules_remote ,so when I have new rules I would like to reload nginx and download new owasp rules. For now I just see either updating ConfigMap.yaml file or by delete Nginx-ingress pod.

There is no direct way to reload NGINX. As you mentioned, a workaround way by changing the ConfigMap would work.

Note that using modsecurity_rules_remote instead storing the rules on the Ingress Controller pod can lead to problems if the server with rules is down or slow to respond. In that case, NGINX will fail to reload or a reload will take a long time. I recommend monitoring reload-related metrics

• controller_nginx_reloads_total. Number of successful NGINX reloads.
• controller_nginx_reload_errors_total. Number of unsuccessful NGINX reloads.
• controller_nginx_last_reload_status. Status of the last NGINX reload, 0 meaning down and 1 up.
• controller_nginx_last_reload_milliseconds. Duration in milliseconds of the last NGINX reload.

Note that if NGINX cannot be reloaded in 4 seconds, the Ingress Controller treat it as an error.

1. I think it should be a feature to support these values to get from ConfigMap? so I don't have to hardcode set-real-ip-from real_ip_header and server-snippets values in server_name _; of main configuration file?

That makes sense! I will discuss it with the team and let you if we implement it.

1. also is it possible to block any kind of traffic on the main loadbalncer IP? right now I am using:

Having both deny and return is redundant. If you have only deny all, NGINX will return with 403 Forbidden. If you have return 444, NGINX will close the connection without handling request. If you have both, return 444 will win.

[RazaGR]

Thanks @pleshakov

1) if I set set-real-ip-from to my LoadBalancers IP, in logs I am not seeing user's real IP(REMOTE_ADDR), but if I set set-real-ip-from to my private network range I am able to see user's real IP address

lets assume my LoadBalancer's external ip is 64-XXX-XX-X, than this doesn't show real user ip:

  real-ip-header: "proxy_protocol"
  set-real-ip-from: "64-XXX-XX-X"

but if I provide my private network IP range, I am able to see real user's IP in log etc:

  real-ip-header: "proxy_protocol"
  set-real-ip-from: "10.114.0.0/20"

2) ok

3) Thanks, I will leave the issue open

4) ok thanks,

5) Also I had another feature request, not sure if it is something already implemented or makes much sense, here is my current scenario, I am using Ingress like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  namespace: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    custom.nginx.org/my-custom-annotation: "on"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.org/proxy-connect-timeout: "30s"
    nginx.org/proxy-read-timeout: "20s"
    nginx.org/client-max-body-size: "4m"
spec:
  tls:
  - hosts:
    - domainA.com
    secretName: domainA.com-tls
  - hosts:
    - doainB.com
    secretName: domainB.com-tls
  rules:
  - host: domainA.com
    http:
      paths:
      - path: /
        backend:
          serviceName: domainA.comService
          servicePort: 8899
  - host: doainB.com
    http:
      paths:
      - path: /
        backend:
          serviceName: domainB.com Service
          servicePort: 8899

I am using my-custom-annotation , so issue is it is being applied to all the hosts, but I wanted to apply that only to domainA.com. To overcome this issue, again I had to use hardcode if/else statements iningress-template. I was thinking if it could be done by creating a custom operator, like we can say it NginxIngress and than provide annotations based on host, so each host could have their own annotations settings which we can use in ingress-template, i.e it could become something like this:

apiVersion: extensions/v1beta1
kind: NginxIngress
metadata:
  name: cafe-ingress
  namespace: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.org/proxy-connect-timeout: "30s"
    nginx.org/proxy-read-timeout: "20s"
    nginx.org/client-max-body-size: "4m"
spec:
  tls:
  - hosts:
    - domainA.com
    secretName: domainA.com-tls
  - hosts:
    - doainB.com
    secretName: domainB.com-tls
  rules:
  - host: domainA.com
    annotations:
       custom.nginx.org/my-custom-annotation: "on"
  - host: domainB.com
    annotations:
       custom.nginx.org/another-domain-annotation: "on"

So there is one global annotation and one for each host. and also if we can extend this idea to define custom server-snippets using this operator, ie:

  - host: domainA.com
    annotations:
       custom.nginx.org/my-custom-annotation: "on"
    server-snippet:
     location / {
      /// some location and other settings for domainA.com
    }
  - host: domainB.com
    annotations:
       custom.nginx.org/another-domain-annotation: "on"
    server-snippet:
     location / {
      /// some location and other settings for domainB.com
    }

[pleshakov]

Hi @RazaGR

1) if I set set-real-ip-from to my LoadBalancers IP, in logs I am not seeing user's real IP(REMOTE_ADDR), but if I set set-real-ip-from to my private network range I am able to see user's real IP address

The IP of the load balancer should be the internal one (belonging to your private network).

5) Also I had another feature request, not sure if it is something already implemented or makes much sense, here is my current scenario, I am using Ingress like this:

I wonder if it make sense for you to split your Ingress into multiple Ingresses. This way you would be able to apply annotations (including custom annotations and snippets) per Ingress. Will that work?

[RazaGR]

Hi @pleshakov , Thank you

The IP of the load balancer should be the internal one (belonging to your private network).

That's it, I was using external IP..

I wonder if it make sense for you to split your Ingress into multiple Ingresses. This way you would be able to apply annotations (including custom annotations and snippets) per Ingress. Will that work?

That's a good alternative.

I will leave issue open for :

I think it should be a feature to support these values to get from ConfigMap? so I don't have to hardcode set-real-ip-from real_ip_header and server-snippets values in server_name _; of main configuration file?

That makes sense! I will discuss it with the team and let you if we implement it.

[Dean-Coakley]

@pleshakov This was implemented in https://github.com/nginxinc/kubernetes-ingress/commit/905b69c8026157bac9ed20385dc725a404e360d4 and can be closed?

[pleshakov]

@Dean-Coakley 905b69c implements set-real-ip-from and real_ip_header. server-snippets for the default server are not implemented yet