Kube-router: ability to run kube-router (service-proxy only) outside of the cluster

[raffaelespazzoli]

I have a server outside of a kubernetes (Openshift to be more precise) cluster that can route IP packets to the cluster SDN.
I am looking for a solution to implement service proxies. Kube-router seems to fit the requirement.
It turns out though that it whants to know in which node it is running.
here is how I am launching it:

kube-router --run-service-proxy true --run-firewall false --run-router false --kubeconfig /etc/kubeconfigs/kubeconfig_raffa1 -v 6

and here is the error:

I1013 00:08:18.565145   18291 round_trippers.go:436] GET https://172.31.0.1:443/version 200 OK in 1 milliseconds
I1013 00:08:18.567447   18291 round_trippers.go:436] GET https://172.31.0.1:443/api/v1/nodes/sdn-tunnel-app-node-0-zxvgc 404 Not Found in 1 milliseconds
Failed to run kube-router: Failed to create network policy controller: Failed to identify the node by NODE_NAME, hostname or --hostname-override

what is weird is that the error seems to be related to the network policy feature, which I am setting to not be started.

Why does kube-router need to know which node it's running on when the only feature requested is service proxy?
Also is there a workaround?

[murali-reddy]

Please try running ./kube-router --run-service-proxy=true --run-firewall=false --run-router=false

[raffaelespazzoli]

I did it. I am still getting the same error, but something has changed: it looks like it's not trying anymore to create the network policy controller

kube-router --run-service-proxy=true --run-firewall=false --run-router=false --kubeconfig /etc/kubeconfigs/kubeconfig_raffa1 -v 6 --hostname-override none

and this is the last part of the output:

I1013 15:23:04.034819   17580 shared_informer.go:116] caches populated
I1013 15:23:04.039913   17580 round_trippers.go:436] GET https://172.31.0.1:443/api/v1/nodes/sdn-tunnel-app-node-0-bw6kd 404 Not Found in 2 milliseconds
I1013 15:23:04.042563   17580 round_trippers.go:436] GET https://172.31.0.1:443/api/v1/nodes/none 404 Not Found in 1 milliseconds
Failed to run kube-router: Failed to create network services controller: Failed to identify the node by NODE_NAME, hostname or --hostname-override

I get the same error without --hostname-override except it's not trying to lookup the overridden name.
I'd like to understand if being a cluster node is a prerequisite. It shouldn't be in mind for just doing the load balancing part.
If I am right maybe we could patch the code to support this configuration.

[raffaelespazzoli]

Digging a bit more it looks like the node api is called to get some info (host name and host ip) that could perhaps be obtained otherwise:
the code is here:
https://github.com/cloudnativelabs/kube-router/blob/b76d22f0d8f90e50bc794300550a4cd659264518/pkg/controllers/proxy/network_services_controller.go#L1989

Maybe is running outside of the cluster ip and hostname could be passed from the CLI or could be obtained with some standard libc calls.
Would that be a reasonable patch?

[murali-reddy]

I'd like to understand if being a cluster node is a prerequisite. It shouldn't be in mind for just doing the load balancing part.

Yes, right now its hard-coded assumption that node on which you are running kube-router is actually Kubernetes node. That's the reason for the above failures. For service proxy may be this restriction can be relaxed. But i am wondering what could be the use-case for this requirement?

What is the type service (ClusterIP, NodePort, ExternalIP etc) you are looking to expose?

[raffaelespazzoli]

@murali-reddy The use case is that I have created a network tunnel between multiple OpenShift SDNs. This tunnel can only route the Pod's IPs. Now I'd like to add discovery and service proxying/loadbalancing.
More info on what I am doing can be found here:
https://blog.openshift.com/connecting-multiple-openshift-sdns-with-a-network-tunnel/
and the git repo is here (this is the branch where I am trying to use kube-router:
https://github.com/raffaelespazzoli/openshift-sdn-encrypted-tunnel/tree/kube-router

I'd like to be able to proxy only the ClusterIP services. The others wouldn't work in my setup, but I don't think there is a way to disable them, it there?

So, If I patched kube-router to work in out-of-cluster mode, would you be interested in receiving the PR?

[uablrek]

Please consider using environment variables for host and host-ip, at least as an option. That will be compatible with the k8s downward-api (capabilities).

A kube-router pod may declare something like;

        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: spec.hostIP

[raffaelespazzoli]

I eventually did patch the kube-router code to support my use case.
You can find the PR here:
https://github.com/cloudnativelabs/kube-router/pull/559

[jerrykan]

I have a similar use case where I would like to run kube-router on the apiserver host, but I don't want to register the host as a node. All I really want is for kube-router to listen and update the routes for the node pod networks, but not advertise anything.

This would help to resolve some problems I had with using kubectl proxy and also with cert-manager which runs a webhook service in a pod to validate CustomResourceDefintions. Both of these use cases fail because the apiserver is unable to route traffic correctly.

Is there any potential for the PR to be merged? and if so would it be possible to extend the "standalone" functionality to include the --run-router option (which I believe handles setting the routes)

[elvizlai]

@murali-reddy +1 same issue.
I want to start a new vpc as nginx gateway to communicate with k8s internal.

Tring but failed:

[root@node02 ~]# ./kube-router --master=http://10.140.0.2:8080 --run-router=true --run-firewall=true --run-service-proxy=true
I0327 02:50:40.941630   23641 kube-router.go:207] Running ./kube-router version , built on , go1.11.4
Failed to run kube-router: Failed to create network policy controller: Failed to identify the node by NODE_NAME, hostname or --hostname-override

and when I tried using --kubeconfig

E0327 02:52:46.485895   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-router" cannot list resource "endpoints" in API group "" at the cluster scope
E0327 02:52:46.485932   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-router" cannot list resource "namespaces" in API group "" at the cluster scope
E0327 02:52:46.490498   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: pods is forbidden: User "system:kube-router" cannot list resource "pods" in API group "" at the cluster scope
E0327 02:52:46.490827   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User "system:kube-router" cannot list resource "nodes" in API group "" at the cluster scope
E0327 02:52:47.487366   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:kube-router" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E0327 02:52:47.489958   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Service: services is forbidden: User "system:kube-router" cannot list resource "services" in API group "" at the cluster scope
E0327 02:52:47.490037   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-router" cannot list resource "endpoints" in API group "" at the cluster scope
E0327 02:52:47.490104   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-router" cannot list resource "namespaces" in API group "" at the cluster scope
E0327 02:52:47.495602   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: pods is forbidden: User "system:kube-router" cannot list resource "pods" in API group "" at the cluster scope
E0327 02:52:47.495679   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User "system:kube-router" cannot list resource "nodes" in API group "" at the cluster scope

[wu0407]

@murali-reddy +1 same issue.
and when I tried using --kubeconfig

E0327 02:52:46.485895   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-router" cannot list resource "endpoints" in API group "" at the cluster scope
E0327 02:52:46.485932   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-router" cannot list resource "namespaces" in API group "" at the cluster scope
E0327 02:52:46.490498   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: pods is forbidden: User "system:kube-router" cannot list resource "pods" in API group "" at the cluster scope
E0327 02:52:46.490827   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User "system:kube-router" cannot list resource "nodes" in API group "" at the cluster scope
E0327 02:52:47.487366   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:kube-router" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E0327 02:52:47.489958   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Service: services is forbidden: User "system:kube-router" cannot list resource "services" in API group "" at the cluster scope
E0327 02:52:47.490037   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-router" cannot list resource "endpoints" in API group "" at the cluster scope
E0327 02:52:47.490104   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-router" cannot list resource "namespaces" in API group "" at the cluster scope
E0327 02:52:47.495602   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: pods is forbidden: User "system:kube-router" cannot list resource "pods" in API group "" at the cluster scope
E0327 02:52:47.495679   23687 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User "system:kube-router" cannot list resource "nodes" in API group "" at the cluster scope

this is the authorization failed

[wu0407]

need run kube-router on the master node that not running kubelet as route refeltor, use to advertise cluster ip and ExternalIP and loadbalance in the rr

[wu0407]

@raffaelespazzoli test branch https://github.com/raffaelespazzoli/kube-router/tree/standalone

docker run -it -e NODE_NAME=$HOSTNAME  --net=host   --privileged   --rm -v /etc/kubernetes:/etc/kubernetes cloudnativelabs/kube-router:standalone kube-router-cmd --enable-cni=false    --bgp-graceful-restart    --cluster-asn=64512    --enable-overlay=false    --enable-pod-egress=false    --kubeconfig=/etc/kubernetes/kube-route.kubeconfig    --run-firewall=false    --run-router    --run-service-proxy=false    --nodes-full-mesh=false --standalone --standalone-hostname link01 --standalone-ip 192.168.10.11 --standalone-iface bond0.10   

same result with the 0.3.1

Failed to run kube-router: Failed to create network routing controller: Failed getting node object from API server: Failed to identify the node by NODE_NAME, hostname or --hostname-override

[aauren]

Closing as this is a use-case that is not the core focus of this project for now.

[jerrykan]

@aauren If someone were to implement some sort of "listen-only" mode would that pull request be accepted? or is it a feature that is not wanted?

I curious because I would assume the use-case I mentioned about wanting to run kube-router on the api-server hosts without wanting to register them as a worker node would be a reasonably common on.

---

edit: changed "standalone" to "listen-only" as it is probably a more accurate description of what has been requested.

[aauren]

I would leave it up to @murali-reddy to say definitively one way or another, but my personal thought is that it adds a lot of logic handling for something that's not a core value of the project.

As far as I'm aware the most common use-case out there right now is to run your control plane on the actual kubelet nodes. This is the default for most cloud setups currently. It has benefits in being able to manage kubernetes the same way that you manage your software that you run on kubernetes and kube-router would fit perfect here in route-reflector mode.

I also believe that there are other ways to handle your use-case. For instance if you were to advertise your services via BGP then your kube-control nodes would be able to resolve the service endpoints for the cert-manager webhook.