Ingress-nginx: Custom Resource Definition or Annotation for TCP Services and UDP Services

[WilliamABradley]

It would be nice to be able to define and expose TCP Services and UDP Services as a Custom Resource Definition. This is especially useful in situations like Terraform, where it is difficult to manage multiple TCP Services across multiple namespaces.

After patching these to expose an External Port, it is very difficult to do inside Terraform, and would have to be scripted instead. This process is also not very simple for use cases outside of Terraform, as you need to know what to patch, in order to expose a port.

Current approach

Currently, to expose a TCP Service you have to:

1. Patch the TCP Services Config Map, adding a port to expose, then defining the namespace, service and port that it resolves to:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tcp-services
     namespace: ingress-nginx
   data:
     9000: "default/example-go:8080"
     5432: "default/example-go:5432"
   

2. Patch the Ingress Service config to expose a port to a service:

   apiVersion: v1
   kind: Service
   metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   spec:
     type: LoadBalancer
     ports:
       - name: http
         port: 80
         targetPort: 80
         protocol: TCP
       - name: https
         port: 443
         targetPort: 443
         protocol: TCP
       - name: go-service-postgres
         port: 5432
         targetPort: 5432
         protocol: TCP
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   

To remove an exposed service from above, using a script, it is much more difficult. You would have to find the related lines, remove it, and re-apply.

Custom Resource Definition

If we had the ability to create a Custom Resource Definition that configured the TCP Service, and then optionally updated the Ingress Resource to expose a port (If using TCP/UDP proxy).

This becomes very easy to add and remove for Terraform, with the new Kubernetes Alpha Provider, or using kubectl.

The benefit of a Custom Resource Definition is it would be easy to remove the resource, easy to add the resource across Namespaces, easy to add extra labels, annotations, etc. And configure how these services interact with the Ingress.

An example of how this might look:

apiVersion: nginx.ingress.kubernetes.io/v1
kind: TCPService
metadata:
  name: go-service-web
spec:
  listener:
    port: 9000
  upstream:
    name: example-go
    namespace: default
    port: 8080

apiVersion: nginx.ingress.kubernetes.io/v1
kind: TCPService
metadata:
  name: go-service-postgres
spec:
  listener:
    port: 5432
    proxy-expose: true
  upstream:
    name: example-go
    namespace: default
    port: 5432

Service Annotation

Alternatively, if this was an Annotation on the Service Resource:

tcp-service-external-port could support port mapping with [external port]:[internal port], or just one value if you want to keep the port the same.
tcp-service-proxy-expose could be true, false, 9000, 5432, or 9000,5432

apiVersion: v1
kind: Service
metadata:
  name: example-go
  annotations:
    nginx.ingress.kubernetes.io/tcp-service: true
    nginx.ingress.kubernetes.io/tcp-service-external-port: 9000:8080,5432
    nginx.ingress.kubernetes.io/tcp-service-proxy-expose: 5432
...

Does it require a particular kubernetes version?

Required Kubernetes 1.16+ for GA CRD support.

/kind feature

[aledbf]

@WilliamABradley thank you for the suggestion but I don't think this should be implemented.
Reasons:

• TCP/UDP support in ingress-nginx was added only as a last resort in bare-metal
• There is no official support for TCP/UDP in k8s Ingress
• The Ingress evolution has plans to add support https://github.com/kubernetes-sigs/service-apis
• The CRD approach requires permissions to create k8s objects (services in particular), something that we don't require right now. Also, it adds another variable for the sync of the model.

[WilliamABradley]

@aledbf Can you link me the part that discusses TCP support for Ingress in the future? I can't find it.

TCP support is valuable outside of bare-metal environments, such as exposing a Database, or other TCP Service, which might be used in production, e.g. Some game server, or for debugging or test environments to be able to view the Database.

This would provide a helpful workaround for before official support occurs, which isn't even Roadmapped yet, and even then will probably be years away.

If CRDs are too heavy, then annotations on the service are a simpler method, and are just as easy to add and remove.

[aledbf]

Can you link me the part that discusses TCP support for Ingress in the future? I can't find it.

Still a PR https://github.com/kubernetes-sigs/service-apis/pull/186

[aledbf]

If CRDs are too heavy, then annotations on the service are a simpler method, and are just as easy to add and remove.

The use of a configmap is on purpose, to keep these features simple as possible and avoid to deal with things like conflict resolution or overlaps in port definitions. I am aware of the downside you point out in the initial description of the issue.

[aledbf]

TCP support is valuable outside of bare-metal environments, such as exposing a Database, or other TCP Service, which might be used in production, e.g. Some game server, or for debugging or test environments to be able to view the Database.

One of the reasons why I always reject requests to add more things to the TCP/UDP feature is related to the NGINX reloads that we cannot avoid when the nginx.conf file changes. Not only for tcp/udp but because also breaks long-running HTTP requests.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.