K3os: EXTERNAL-IP <pending> forever

[frafra]

How to reproduce:

ip="your_server_ip"
rm -r ~/.kube
mkdir ~/.kube
scp rancher@$ip:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i "s/localhost/$ip/g" ~/.kube/config

ssh rancher@$ip mkdir /home/rancher/local-path-provisioner
curl -s https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml | sed 's;/opt;/home/rancher;g' | kubectl apply -f -
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

kubectl -n kube-system create sa tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller --history-max 200

helm install --name test-wordpress stable/wordpress

Result:

$ kubectl get svc --namespace default test-wordpress
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
test-wordpress   LoadBalancer   10.43.29.245   <pending>     80:30003/TCP,443:31258/TCP   10m

Additional details:

• I use /home/rancher instead of /opt for persistence because of #207
• kubectl get svc -n kube-system shows that Traefik got the right external ip
• I am running k3os 0.3.0 on Hetzner (VPS)
• I do not have much experience with Kubernetes
• I see many messages like that one coming from rancher/kipper-lb: Warning FailedScheduling 12s (x5 over 21s) default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

[dweomer]

@frafra do you run into the same issue when downloading and installing helm onto the k3os host and invoking from there?

[frafra]

@dweomer I actually get a different error:

k3os-0 [~]$ helm init --service-account tiller --history-max 200
$HELM_HOME has been configured at /home/rancher/.helm.
Error: error installing: Post http://localhost:8080/apis/extensions/v1beta1/namespaces/kube-system/deployments: dial tcp 127.0.0.1:8080: connect: connection refused

[dweomer]

@frafra did you make sure that k3s was fully up and running (it can take anywhere from 20 seconds to a few minutes) before attempting the helm invocation? Take a look at https://github.com/rancher/k3s/blob/master/scripts/sonobuoy for some inspiration on automating such. Specifically note that there are effectively three wait phases: kubeconfig, nodes, services and that you will wan to to include traefik along with coredns in with the services to wait on. If that's all a bit too much and/or you just need to wait manually I have found that invoking kubectl get -A pods a number of times until there are four pod lines, the helm line at "Completed" status with the others at "Running"

[frafra]

@dweomer thanks, but I waited > 20', and it still fails. Helm v2.14.3.

[zimme]

@dweomer I'm seeing pretty much the same problem when I try and run this on my hetzner instance.

➜  ~ kubectl get all -A
NAMESPACE            NAME                                           READY   STATUS             RESTARTS   AGE
kube-system          pod/coredns-66f496764-9qhkf                    1/1     Running            0          20h
kube-system          pod/helm-install-traefik-ct278                 0/1     Completed          0          20h
kube-system          pod/svclb-traefik-v86f5                        3/3     Running            0          20h
kube-system          pod/traefik-d869575c8-mkfwx                    1/1     Running            0          20h
kube-system          pod/svclb-traefik-8xhqq                        3/3     Running            0          20h
local-path-storage   pod/local-path-provisioner-ccbdd96dc-zq2f4     1/1     Running            0          18h
kube-system          pod/tiller-deploy-6b9c575bfc-kfmr4             1/1     Running            0          17h
default              pod/svclb-test-wordpress-wordpress-gtjl2       0/2     Pending            0          17h
default              pod/svclb-test-wordpress-wordpress-wqlwk       0/2     Pending            0          17h
default              pod/test-wordpress-mariadb-0                   1/1     Running            0          17h
default              pod/test-wordpress-wordpress-5b7848c78-gbjw5   0/1     CrashLoopBackOff   179        17h


NAMESPACE     NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP                  PORT(S)                                     AGE
kube-system   service/kube-dns                   ClusterIP      10.43.0.10      <none>                       53/UDP,53/TCP,9153/TCP                      20h
default       service/kubernetes                 ClusterIP      10.43.0.1       <none>                       443/TCP                                     20h
kube-system   service/traefik                    LoadBalancer   10.43.116.146   10.64.96.39,95.216.214.152   80:31212/TCP,443:31690/TCP,8080:31910/TCP   20h
kube-system   service/tiller-deploy              ClusterIP      10.43.172.201   <none>                       44134/TCP                                   17h
default       service/test-wordpress-mariadb     ClusterIP      10.43.127.174   <none>                       3306/TCP                                    17h
default       service/test-wordpress-wordpress   LoadBalancer   10.43.182.251   <pending>                    80:32594/TCP,443:32402/TCP                  17h

NAMESPACE     NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik                    2         2         2       2            2           <none>          20h
default       daemonset.apps/svclb-test-wordpress-wordpress   2         2         0       2            0           <none>          17h

NAMESPACE            NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
kube-system          deployment.apps/coredns                    1/1     1            1           20h
kube-system          deployment.apps/traefik                    1/1     1            1           20h
local-path-storage   deployment.apps/local-path-provisioner     1/1     1            1           18h
kube-system          deployment.apps/tiller-deploy              1/1     1            1           17h
default              deployment.apps/test-wordpress-wordpress   0/1     1            0           17h

NAMESPACE            NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system          replicaset.apps/coredns-66f496764                    1         1         1       20h
kube-system          replicaset.apps/traefik-d869575c8                    1         1         1       20h
local-path-storage   replicaset.apps/local-path-provisioner-ccbdd96dc     1         1         1       18h
kube-system          replicaset.apps/tiller-deploy-6b9c575bfc             1         1         1       17h
default              replicaset.apps/test-wordpress-wordpress-5b7848c78   1         1         0       17h

NAMESPACE   NAME                                      READY   AGE
default     statefulset.apps/test-wordpress-mariadb   1/1     17h


NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   1/1           26s        20h

➜  ~ kubectl logs pod/test-wordpress-wordpress-5b7848c78-gbjw5

Welcome to the Bitnami wordpress container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-wordpress
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-wordpress/issues

WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
nami    INFO  Initializing apache
nami    INFO  apache successfully initialized
nami    INFO  Initializing php
nami    INFO  php successfully initialized
nami    INFO  Initializing mysql-client
nami    INFO  mysql-client successfully initialized
nami    INFO  Initializing wordpress
mysql-c INFO  Trying to connect to MySQL server
Error executing 'postInstallation': Failed to connect to test-wordpress-mariadb:3306 after 36 tries
➜  ~

It's pending after like 17h but I also saw that the wordpress container crashed as it couldn't connect to the db.

[boelenz]

I have the same event happening. I cannot see what caused it.

[zimme]

~Pretty sure this has been resolved by #355 as the metsdata is properly fetched now when using hetzner as the datasource.~

Scratch that, just tested installing wordpress via a helmchart manifest file and it's pending.

[recklessop]

Im seeing the same thing on a LoadBalancer I just deployed....

[djpbessems]

Seeing the same on a freshly provisioned K3s cluster; I installed Longhorn (hoping to evaluate it/use it for NFS-mounts or just regular distributed block storage), but I'm seeing that the longhorn-frontend service stays on <pending>:

NAMESPACE         NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default           kubernetes                ClusterIP      10.43.0.1       <none>           443/TCP                      19m
kube-system       kube-dns                  ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       19m
kube-system       metrics-server            ClusterIP      10.43.198.102   <none>           443/TCP                      19m
default           traefik                   LoadBalancer   10.43.59.5      192.168.11.245   80:30249/TCP,443:32686/TCP   3m1s
longhorn-system   longhorn-backend          ClusterIP      10.43.129.62    <none>           9500/TCP                     93s
longhorn-system   longhorn-frontend         LoadBalancer   10.43.163.220   <pending>        80:30538/TCP

When I check the status of one of the pods, I see the following:

Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports.

[Rojikku]

I also have this issue.
I am running a one-node kubernetes cluster, barely understand kubernetes, and am really just learning and playing around.

The solution is as follows:
kubectl patch svc SERVICENAME -p '{"spec": {"type": "LoadBalancer", "externalIPs":["192.168.0.X"]}}'

Obviously replace the IP with the desired IP, and the SERVICENAME with the right name.
You can get the right name from kubectl get svc
I'm using this for my nginx ingress, because apparently I have to setup ingress in order to setup rancher.
There are probably issues associated with this solution. For example, if my node at that IP goes down, what would happen?

[djpbessems]

I solved it by changing the service from type LoadBalancer to ClusterIP and then create an IngressRoute for Traefik to publish the dashboard.

[Rojikku]

I solved it by changing the service from type LoadBalancer to ClusterIP and then create an IngressRoute for Traefik to publish the dashboard.

I've been working on a better solution, and was working on setting up Traefik. Could you give me some more information on how you set that up?

EDIT: Currently got stable/traefik working by doing the patch command on it, and that seems to work fine, as far as I know.

[djpbessems]

In case you're still interested (this is for Traefik 2.x):

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`traefik.example.com`)
      kind: Rule
      services:
        - name: api@internal
          kind: TraefikService
  tls:
    certResolver: default
    options:
      name: default
    domains:
    - main: 'traefik.example.com'

Ofcourse you will also have to define TLSOption and certResolver

[teknofile]

I also have this issue.
I am running a one-node kubernetes cluster, barely understand kubernetes, and am really just learning and playing around.

The solution is as follows:
kubectl patch svc SERVICENAME -p '{"spec": {"type": "LoadBalancer", "externalIPs":["192.168.0.X"]}}'

Obviously replace the IP with the desired IP, and the SERVICENAME with the right name.
You can get the right name from kubectl get svc
I'm using this for my nginx ingress, because apparently I have to setup ingress in order to setup rancher.
There are probably issues associated with this solution. For example, if my node at that IP goes down, what would happen?

How did you determine what IP address to give? I'm fairly certain that this error is because it is binding port 80 on the same IP address that traefik is already listening on. But I can't find out how k3s allocates EXTERNAL-IP's

[Rojikku]

I also have this issue.
I am running a one-node kubernetes cluster, barely understand kubernetes, and am really just learning and playing around.
The solution is as follows:
kubectl patch svc SERVICENAME -p '{"spec": {"type": "LoadBalancer", "externalIPs":["192.168.0.X"]}}'
Obviously replace the IP with the desired IP, and the SERVICENAME with the right name.
You can get the right name from kubectl get svc
I'm using this for my nginx ingress, because apparently I have to setup ingress in order to setup rancher.
There are probably issues associated with this solution. For example, if my node at that IP goes down, what would happen?

How did you determine what IP address to give? I'm fairly certain that this error is because it is binding port 80 on the same IP address that traefik is already listening on. But I can't find out how k3s allocates EXTERNAL-IP's

So the machine itself has an IP, that you probably already know. That's the one I'm talking about.

Actually, my above solution is horrible wrong though, I have found.
This is simply for the reason that K3os comes with Traefik installed under the kube-system namespace. As such, you shouldn't need to install any further services with external IPs. Just setup a service and an ingress route, and it should work out. At least, it has for me.

[zimme]

I believe this might have had something to do with that Hetzner didn't offer any load balancer as a service thing and k8s doesn't come with an implementation for LoadBalancer, it's something you need to provide using something like Metallb, etc. I do believe k3s comes with klipperlb that you can disable with --no-deploy servicelb.

Hetzner cloud might need one to use something like metallb for it to being able to resolve the ip.

Hetzner now seem to offer a load balancer as a service just like aws, gcloud, etc. do and that might be able to be used.

I might be completely off here as I'm pretty new to the k8s world.

[mysticaltech]

@zimme The Hetzner CCM is what offers that functionality.