Fabio: Kubernetes - deployment

[ajohnstone]

Would be nice to have an example to get running within kubernetes.

[magiconair]

Indeed ...

[magiconair]

I could use some help here. I had a look a the K8s API and I can't find a way for the services to publish their prefixes like they do in tags in consul. I think deploying a consul cluster just for fabio seems a bit overkill since K8s already has a service registry. I think I could use labels but they are size restricted to 64 chars AFAIR. Services could offer their prefixes via an HTTP endpoint but that would require polling from fabio and it would not support 3rd party services. Anybody with Kubernetes experience has an idea on how to solve this?

[tanuck]

Was the OP more about writing an example k8s Pod or Deployment. Rather than using k8s as a registry for fabio?

[silentred]

using k8s as a registry would be the ideal way to do it.

[magiconair]

@silentred Got an idea where the meta-data can be stored (e.g. host/path and other options) Couldn't find something suitable in the k8s API last time I looked but granted I didn't dig very deep. Suggestions are welcome.

[Bregor]

Got an idea where the meta-data can be stored (e.g. host/path and other options) Couldn't find something suitable in the k8s AP

@magiconair, maybe you should look at the ingress?

Example for host/path:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: s1
          servicePort: 80
      - path: /bar
        backend:
          serviceName: s2
          servicePort: 80

[silentred]

@Bregor good idea. I did a little searching, and found this nginx reverse proxy, which just keeps querying Ingress resource to reload nginx for load balancing.

[Bregor]

Main problem (for now) with ingress is TLS. You can use tls section only once before any host sections, so you automagically lost tls-sni support.

[InAnimaTe]

I would absolutely love to see Fabio working on k8s. As I'm starting to build our own cluster, it seems the best course of action would be for Fabio to tap into the k8s API and work like other Ingress Controllers which load all Ingress registrations and serve them appropriately.

[taemon1337]

I am looking at doing an abbreviated integration with Kubernetes (specifically only the TCP+SNI) portion and letting Kubernetes handle the health checking. I am just trying to implement a custom Backend that will retrieve a list of all Kubernetes services from a REST url and inject those service routes into Fabio's routing table. I'm not very good with go.

It looks like Fabio stores it's configuration in Consul's KV store, is that right? Why does it do that, since isn't it building the config dynamically using Consul services?

(should we take this offline?)

[leprechau]

There are a few pieces of optional configuration stored in consul but the bulk of the route configuration comes from the specified registry backend.

The registry backends all conform to an interface defined in https://github.com/fabiolb/fabio/blob/master/registry/backend.go

Today there are three existing registry definitions: file, static and consul

Supporting kubernetes or rancher or any other provider of service information would involve contributing another registry backend.

[taemon1337]

Yes, I'm currently working on adding a Kubernetes backend, though because Kubernetes itself handles health checking and service discovery, the only func I think I need to implement is the WatchServices.

If I understand how Fabio works (and this is my question), can I just implement WatchServices such that each Fabio instance gets the list of services directly from the Kubernetes API and adds a route for each service? Or will it break Fabio to not implement the other functions?

My use case is limited to TCP+SNI only, so all my solution really needs to go is route
{service}.{namespace}.{external-domain-name} --> {service}.{namespace}.{kubernetes-internal-domain}
Kubernetes will then route each service to the actual pod(s).

[magiconair]

@taemon1337 fabio does not store routes in Consul. You can store routes in consul or some external program. fabio generates its primary routing table from the service information in Consul.

The reason it supports the KV store is that you'd want a way to manually patch a routing table in case of a broken deployment. Or you want to add some routes to services which are not in Consul.

To support k8s you wouldn't need to do this. As @leprechau suggested, you just need to create a new backend implementation and implement the WatchServices function which generates the routing table in the string format, e.g.

route add <svc name> host/path http://host:port

If we can do this with a simple REST call then I'd prefer that instead of pulling in the k8s go-client. If you send me the REST call then I can help.

[magiconair]

@taemon1337 Didn't see your earlier comment. Yes, WatchServices is sufficient. I should probably break that interface up. It is too big.

[taemon1337]

From the k8s api, '/api/v1/services'
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#list-all-namespaces-173

and here is the watch endpoint (i was going to start by just polling the above):
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#watch-list-all-namespaces-176

[taemon1337]

So I've got a very basic version working (https://github.com/taemon1337/fabio-k8). There is a readme in the registry/kubernetes which kinda explains it, it still allows k8 to do the service discovery and health checking as fabio just routes external traffic to k8 services.