Ambassador: Feature Request - Raw TCP support

[FuzzOli87]

We had to create a separate LoadBalancer type service for TCP services when we ran the NGINX ingress. Now that we switched to Ambassador and are comfortable with it's feature set, I would like to be able to expose TCP servers via Ambassador configurations so that I can route via hostname:port.

So HTTP services could go to api.example.com:80 and TCP server to api.example.com:1234. Or even by path api.example.com/tcp -> TCP server although it feels like that is a bit taboo.

[popaaaandrei]

Envoy seems to support TCP proxy
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/tcp_proxy

Is there any way to add configurations via Ambassador annotations?

[popaaaandrei]

It seems that it's possible through envoy_override annotation. Can someone provide an example for proxying one simple Kubernetes service?
Thank you

[sc85]

I was also looking for this feature. I was thinking about adding something like a bastion host to the cluster and use the same public IP but then e.g. port 22 which is forwarded to this port.
So far I wasn't able to find something in the documentation to make this happen.

Does anybody have some hints to make this work?

[cliveseldon]

I'm also interested in knowing how we could expose our custom TCP protocol via Ambassador so everything can go through Ambassador: we already use REST and gRPC endpoints.

[bencourliss]

Looks like Ambassador is only tackling the L7 area at the moment. Also adding an envoy_override as documented here https://www.getambassador.io/reference/override doesn't look like it will work as you'll have to violate one of their limitations, "It cannot change any element already synthesized in the mapping". I'm guessing that the route map is synthesized by Ambassador.
Further investigation on the Envoy side listed here https://www.envoyproxy.io/docs/envoy/latest/api-v1/network_filters/tcp_proxy_filter.html?highlight=tcp, seems to indicate that we'd have to add a route configuration for each service and new TCP port mapping combination. Without knowing how Ambassador generates that route configuration for Envoy to begin with, I don't know how you can do this. It's going to need to be done in the Ambassador code itself. Now where did I leave that Python For Dummies book...

[richarddli]

One way to start (that would help us, and help you) is to figure out a custom envoy j2 that would contain the necessary configuration. Once we have that, we could then update Ambassador to generate that custom Envoy configuration from annotations. (We usually don't start with hacking on the Python code; we have to figure out properly formed Envoy configuration first.)

[berlinsaint]

i m looking an best way to expose tcp service by the envoy tcp_proxy;

i make a POC based on the raw envoy;

my envoy.yaml config is

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 9292
    filter_chains:
    - filters:
      - name: envoy.tcp_proxy
        config:
          stat_prefix: ingress_tcp
          cluster: mysql_server
          access_log:
            - name: envoy.file_access_log
              config:
                path: /dev/stdout
  clusters:
  - name: xds_cluster
    connect_timeout: 0.25s
    type: strict_dns 
    lb_policy: ROUND_ROBIN
    #http2_protocol_options: {}
    load_assignment: 
      cluster_name: xds_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 172.31.10.8
                port_value: 7777 

  - name: mysql_server
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: 172.31.200.9
        port_value: 31323 
dynamic_resources:
  lds_config:
    api_config_source:
      api_type: REST
      cluster_names: [xds_cluster]
      refresh_delay: 10s
  cds_config:
    api_config_source:
      api_type: REST
      cluster_names: [xds_cluster]
      refresh_delay: 10s

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001
node:
  id: "1111"
  cluster: test-lds
  build_version: "22222"

i alsoe make an lds and cds service by flask:

 from flask import Flask, Response
import json
from flask import request

app = Flask(__name__)


@app.route("/v2/discovery:clusters", methods=['POST', 'GET'])
def discovery_cluster():
    print(request.data)
    r = '''{
            "version_info": "0",
            "resources": [
                {
                    "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                    "name": "hello-cluster",
                    "connect_timeout": "0.25s",
                    "lb_policy": "ROUND_ROBIN",
                    "type": "strict_dns",
                    "hosts": [
                        {
                            "socket_address": {
                                "address": "172.31.10.11",
                                "port_value": 4306
                            }
                        }
                    ]
                },
                {
                    "@type": "type.googleapis.com/envoy.api.v2.Cluster",
                    "name": "mysql-cluster",
                    "connect_timeout": "0.25s",
                    "lb_policy": "ROUND_ROBIN",
                    "type": "strict_dns",
                    "hosts": [
                        {
                            "socket_address": {
                                "address": "172.31.200.9",
                                "port_value": 31323
                            }
                        }
                    ]
                }
            ]
        }'''
    return Response(r, mimetype='application/json')


@app.route("/v2/discovery:listeners", methods=['POST', 'GET'])
def discovery():
    print(request.data)
    r = '''{
            "version_info": "0",
            "resources": [
                {
                    "@type": "type.googleapis.com/envoy.api.v2.Listener",
                    "name": "listener_0",
                    "address": {
                        "socket_address": {
                            "address": "0.0.0.0",
                            "port_value": 10000
                        }
                    },
                    "filter_chains": [
                        {
                            "filters": [
                                {
                                    "name": "envoy.tcp_proxy",
                                    "config": {
                                        "stat_prefix": "ingress_tcp",
                                        "cluster": "hello-cluster"
                                    }
                                }
                            ]
                        }
                    ]
                },
                {
                    "@type": "type.googleapis.com/envoy.api.v2.Listener",
                    "name": "listener_1",
                    "address": {
                        "socket_address": {
                            "address": "0.0.0.0",
                            "port_value": 9393
                        }
                    },
                    "filter_chains": [
                        {
                            "filters": [
                                {
                                    "name": "envoy.tcp_proxy",
                                    "config": {
                                        "stat_prefix": "mysql-tcp",
                                        "cluster": "mysql-cluster"
                                    }
                                }
                            ]
                        }
                    ]
                }
            ]
}'''

    return Response(r, mimetype='application/json')


if __name__ == '__main__':
    app.run(host='0.0.0.0', port="7777")

i make the tcp_proxy dynamically configured by the lds and cds succesfully, and envoy indeed works well when proxy the tcp traffic.

i want to that ambassador can implement this feature, because in many case , wo really to expose tcp service on kubernetes.

[grpatter]

@richarddli - is what @berlinsaint posted what you were looking for? This is a feature we could definitely want to leverage as well, and the only thing stopping us from moving forward with Ambassador. Sounds like the envoy override approach outlined above is a no-go, unless I'm mistaken?

[drshrey]

@richarddli any luck on this being in the pipeline soon? If this isn't being actively developed by the ambassador core team/ anybody else, I'm going to start working on this.

Also +1 to @grpatter on confirming that envoy_override would not work in this case. Thanks!

[richarddli]

If you're interested in working on this, that would be great! Just a note that any code should go against the 0.50 code base (which has changed substantially) in the release/0.50 branch. We had discussed using SNI to do the mapping, e.g.,

apiVersion: ambassador/v1
kind: TCPMapping
name: tcp_one
host: tcp.datawire.io
port: 8080
service: tcp-svc-one:80

---
apiVersion: ambassador/v1
kind: TCPMapping
name: tcp_two
sni:
- tcp2.datawire.io
- tcp-2.datawire.io
port: 8080
service: tcp-svc-one:80

[richarddli]

Done in 0.51.

[jaydipdave]

Is there a possibility to do something like this:

on port 443 i can use SSL termination, as well as TCP port forwarding, based on hostname.

--- apiVersion: ambassador/v1 kind: Mapping ambassador_id: default name: jenkins_mapping prefix: / # port is 443 service: jenkins.svc.cluster.local timeout_ms: 10000 host: jenkins.prod.com --- apiVersion: ambassador/v1 kind: TLSContext name: jenkins_tls_context hosts: - jenkins.prod.com secret: jenkins-https-certificates.production --- apiVersion: ambassador/v1 kind: TCPMapping name: ssl_inside_jenkins_mapping port: 443 (no SSL termination at ambassador) host: jenkins.auth.com service: jenkins:443 (jenkins has ssl certificate and needs a client certificate to validate the request)

[kongh]

Done in 0.51.

how can i config the ambassador?

ambassador-service

apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:

• name: http
  port: 80
  targetPort: 8080
• name: tcp
  port: 7000
  targetPort: 8080
  selector:
  service: ambassador

this is my upstream service

apiVersion: v1
kind: Service
metadata:
name: abyss2-tcp-gateway
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: TCPMapping
name: abyss2-tcp-gateway_mapping
port: 7000
host: api-tcp.abyss2.io
service: abyss2-tcp-gateway:7000
spec:
selector:
app: abyss2-tcp-gateway
ports:

• protocol: TCP
  port: 7000
  targetPort: 7000

It do not works for me, any ideas ? I am new user of ambasaddor . Help please!

[dotw]

@kongh did you try to have a different targetPort from 8080 in ambassador service. Port 8080 is already assigned for http.

[texascloud]

Is there a possibility to do something like this:

on port 443 i can use SSL termination, as well as TCP port forwarding, based on hostname.

   apiVersion: ambassador/v1
   kind: Mapping
   ambassador_id: default
   name: jenkins_mapping
   prefix: /
   # port is 443
   service: jenkins.svc.cluster.local
   timeout_ms: 10000
   host: jenkins.prod.com
   ---
   apiVersion: ambassador/v1
   kind: TLSContext
   name: jenkins_tls_context
   hosts:
   - jenkins.prod.com
   secret: jenkins-https-certificates.production
  ---
  apiVersion: ambassador/v1 
  kind: TCPMapping
  name: ssl_inside_jenkins_mapping
  port: 443 (no SSL termination at ambassador)
  host: jenkins.auth.com
  service: jenkins:443  (jenkins has ssl certificate and needs a client certificate to validate the request)

I'm trying to do exactly this too

[raghuP9]

I was trying to make this work with my RabbitMQ HA cluster. Since the communication doesn't use HTTP/1.1 or HTTP/2 protocol but AMQP 0.9.1, I was unable to make it work with existing ports 80 and 443 with TLSContext and TCPMapping.

In fact, whenever I added host detail in TLSContext, I was getting error

2020/06/19 08:23:19 Failed to connect to RabbitMQ: Exception (501) Reason: "EOF"
exit status 1

So, I modified the ambassador Loadbalancer service and added a new port(Make sure you add SG rules for the new port added based on the type of your cloud infrastructure and loadbalancer type. In my case, I added port 8443 to SG rules and added the same in ambassador Loadbalancer service):

 spec:
   clusterIP: 10.X.X.X
   externalTrafficPolicy: Cluster
   ports:
   - name: http
     nodePort: 32468
     port: 80
     protocol: TCP
     targetPort: 8080
   - name: https
     nodePort: 30583
     port: 443
     protocol: TCP
     targetPort: 8443
   - name: tcp
     nodePort: 31583
     port: 8443
     protocol: TCP
     targetPort: 8000

Now, the Loadbalancer was able to receive packets on port 8443 as well.

Then I added TCPMapping:

apiVersion: getambassador.io/v2
kind: TCPMapping
metadata:
  name: rabbitmq
  namespace: rmq
spec:
  host: raghu-poc-mq.mydomain.com
  port: 8000
  service: rmq-rabbitmq-ha:5671

And my RabbitMQ cluster was using TLS certificates provided as secret and managed by cert-manager and listening for secure connections on port 5671.

In order to verify this, I connected RabbitMQ with URL amqps://raghu-poc-mq.mydomain.com:8443/

[xlanor]

@raghuP9 , are you running this behind aws ELB by any chance?

[raghuP9]

@raghuP9 , are you running this behind aws ELB by any chance?

I am using MetalLB with bareOS cluster.