Ingress-nginx: [nginx] Support SSL for TCP

[od0]

NGINX supports SSL for opened TCP ports – it would be great to be able to use this feature when specifying TCP ports in the --tcp-services-configmap ConfigMap.

Right now the template looks like this:

        listen {{ $tcpServer.Port }};

Perhaps a new optional flag in the ConfigMap could be added and the template modified to something like this:

        listen {{ $tcpServer.Port }}{{ if $tcpServer.SSLEnabled }} ssl{{ end }};

Happy to take a stab at implementing this and open a PR.

Thanks!

[aledbf]

@od0 what's the benefit? Sorry to ask but I never used this option in the stream section.

[od0]

@aledbf I hadn't used it previously but discovered when trying to set up a SSL service on a port other than 443 (Elasticsearch in this case). I was curious if I could get it to work and discovered that NGINX supports it and it's as simple as adding a ssl after the port declaration (as is done automatically for 443 in this ingress).

https://www.nginx.com/resources/admin-guide/nginx-tcp-ssl-termination/

[aledbf]

Closing. Using ssl in nginx requires the ssl certificates.

[naveensrinivasan]

@aledbf We are trying to use nginx ingress for our mqtt service. The present ingress does not support ssl for tcp services. This would be a great feature for services like this.

What are your thoughts? Do you see issues?

[aledbf]

@naveensrinivasan why not use the ssl passthrough option and terminate ssl in the backend?

[naveensrinivasan]

@aledbf doing this dev/local where there aren't certs would be an issue.
IMO nginx ingress is there to do this kind of loading like SSL termination.

[stela]

I have a similar use-case as @od0 and @naveensrinivasan, want to terminate a connection to an XMPP server listening on a TCP port for client connections. I have a certificate from letsencrypt stored in a k8s Secret, which is easy to configure for a bunch of REST endpoints, each application having a k8s Ingress where tls.secretName points to the letsencrypt secret. The non-encrypted XMPP port 5222 is easy to forward, however I'd like to have a TLS-version exposed instead, would be ideal to have the certificate and private key in a single Secret, which cert-manager can renew automatically.

Or is there an easy workaround where you can terminate SSL on the backend using an existing Secret for the certificate? Examples?

Update: In case anybody else needs this, got it working by using an stunnel docker image, https://hub.docker.com/r/dweomer/stunnel/, as a second container in my pod, configured similarly to https://github.com/PalmStoneGames/kube-cert-manager/blob/master/docs/consume-certificates.md to use the key + certificate. You can see which env variables are available to set here: https://github.com/dweomer/dockerfiles-stunnel/blob/master/stunnel.sh. Set STUNNEL_CONNECT=localhost:unencrypted-port-nr. Then use an nginx-ingress ConfigMap to set up TCP ingress, see https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/examples/tcp/tcp-configmap-example.yaml

[jrodonnell]

Wish I found this thread before I opened an identical feature request the other day but commenting here because there's more traction. I realize I am the beggar in this situation but I strongly agree with @od0 and @naveensrinivasan, NGINX is used all over the world today for this exact purpose and I don't see why that functionality should not be exposed when it's deployed as an ingress controller in a Kubernetes cluster. Not every app can terminate ssl nor can every app be a stateless 12-factor RESTful web app and there are apps in the public Helm chart repo today that are forced to rely on things like NodePorts and manual management of external load balancers because there is no kube-native way to manage tcp traffic via Ingress objects or ingress controllers. Elasticsearch, Postgres, RabbitMQ, and Redis are four examples off the top of my head so the need for this functionality is certainly there.

[dannyk81]

@aledbf just wanted to share a similar use case as @stela, we are planning to move our XMPP application over to K8s and we need an SSL/TLS-over-TCP termination on the Ingress Controller.

We looked at having the termination done on the backend (e.g. using an stunnel for example), but that is not a good fit for us for various reasons.

We are now looking into running an haproxy ingress-controller side-by-side just for this application, as it seems to support that option (https://github.com/jcmoraisjr/haproxy-ingress#tcp-services-configmap) however we would prefer to have this in the nginx Ingress Controller and not introduce another piece to the mix.

[aledbf]

@dannyk81 after https://github.com/kubernetes/ingress-nginx/pull/2929 we could see exactly what's required to implement this feature. Right now the tcp/udp configuration is stored in a configmap, which only handle strings, where adding additional information is too painful.

[dannyk81]

@aledbf given your previous statement that this enhancement might be considered, could we re-open the issue? to ensure we track this.

Thanks!

[arianitu]

+1 I want to SSL terminate a bunch of services that are not http

[enicdep]

Also from my side the same. I want ingress-nginx able to terminate TLS for tcp (non-http) traffic. Very frustrated about this missing fuctionality

[the-nicolas]

Any update? Also struggling here. Just realized thsatt it is not working when trying to set it up. That's so basic that I even forgot to assume that it is not possible.

[isavcic]

I'm seeing a bunch of people, including myself, wanting the same thing which NGINX supports. Yet, there is this weird pushback from the maintainers of this project. Why is that?

[aledbf]

Yet, there is this weird pushback from the maintainers of this project. Why is that?

There is no such thing. Someone needs to work in this feature. Pull requests are always welcome.

That said, in my previous comment.

Right now the tcp/udp configuration is stored in a configmap, which only handle strings, where adding additional information is too painful.

Is hard to add this or any other additional feature with such restriction. This is the limitation.

Also keep in mind Ingress is only about HTTP/HTTPS and the existence of TCP/UDP in this project is because there is no service type=LoadBalancer option in baremetal. I wish I could remove the TCP/UDP feature in a separate project, but sadly it's too late for that.

[xiangwangcheng]

Looking forward to seeing this feature, too. I think ingress-nginx should support this since Nginx does.

[evgmoskalenko]

What about an update? :-)

[sedflix]

Hey all,

I was able to achieve this via a small(and quite bad) hack.
It involved updating a part of nginx.tmpl. The diif for the same is

--- a/helm-charts/nginx-ingress/nginx/nginx.tmpl
+++ b/helm-charts/nginx-ingress/nginx/nginx.tmpl
@@ -679,15 +679,15 @@ stream {
         }

         {{ range $address := $all.Cfg.BindAddressIpv4 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ else }}
-        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ end }}
         {{ if $IsIPV6Enabled }}
         {{ range $address := $all.Cfg.BindAddressIpv6 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ else }}
-        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ end }}
         {{ end }}
         proxy_timeout           {{ $cfg.ProxyStreamTimeout }};
@@ -695,6 +695,21 @@ stream {
         {{ if $tcpServer.Backend.ProxyProtocol.Encode }}
         proxy_protocol          on;
         {{ end }}
+
+        proxy_protocol          on;
+
+        ssl_protocols {{ $cfg.SSLProtocols }};
+        ssl_ciphers '{{ $cfg.SSLCiphers }}';
+        ssl_prefer_server_ciphers on;
+
+        # PEM sha: {{ $cfg.DefaultSSLCertificate.PemSHA }}
+        ssl_certificate     {{ $cfg.DefaultSSLCertificate.PemFileName }};
+        ssl_certificate_key {{ $cfg.DefaultSSLCertificate.PemFileName }};
+
+
+        ssl_session_cache     shared:SSL2:60m;
+        ssl_session_timeout   10h;
+        ssl_handshake_timeout 30s;
     }
     {{ end }}

Reference:

• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/
• https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/

[rechandler12]

I've got another problem. I want SSL Passthrough to my mqtt broker but an error has occurred:

failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number.

I can connect from inside cluster but I have a problem to do it from the outside. So I assume that this is related with NGINX. My config:

controller:
  config:
    use-proxy-protocol: 'true'

  extraArgs:
    enable-ssl-passthrough: ""

  service:
    annotations:
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'

  metrics:
    enabled: true

    serviceMonitor:
      enabled: true
      additionalLabels:
        prometheus: doks-cluster-monitoring

tcp:
  8883: "mqtt/mosquitto:8883"

SOLUTION
It was caused by PROXY protocol. This config works:

controller:
  config:
    use-proxy-protocol: "true"

  service:
    annotations:
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

  metrics:
    enabled: true

    serviceMonitor:
      enabled: true
      additionalLabels:
        prometheus: doks-cluster-monitoring

tcp:
  8883: "mqtt/mosquitto:8883:PROXY"