Microk8s: nginx loadbalancer service EXTERNAL-IP is always in "pending" state

[khteh]

Please run microk8s.inspect and attach the generated tarball to this issue.
inspection-report-20181123_152413.tar.gz

We appreciate your feedback. Thank you for using microk8s.
nginx config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.config: |
    user nginx;
    worker_processes  3;
    error_log  /var/log/nginx/error.log;
    events {
      worker_connections  10240;
    }
    http {
      log_format  main
              'remote_addr:$remote_addr\t'
              'time_local:$time_local\t'
              'method:$request_method\t'
              'uri:$request_uri\t'
              'host:$host\t'
              'status:$status\t'
              'bytes_sent:$body_bytes_sent\t'
              'referer:$http_referer\t'
              'useragent:$http_user_agent\t'
              'forwardedfor:$http_x_forwarded_for\t'
              'request_time:$request_time';
      access_log        /var/log/nginx/access.log main;
      server {
          listen       80;
          server_name  _;
          location / {
              root   html;
              index  index.html index.htm;
          }
      }
      include /etc/nginx/virtualhost/virtualhost.conf;
    }
  virtualhost.config : |
    upstream app {
      server localhost:8080;
      keepalive 1024;
    }
    server {
      listen 80 default_server;
      root /usr/local/app;
      access_log /var/log/nginx/app.access_log main;
      error_log /var/log/nginx/app.error_log;
      location / {
        proxy_pass http://svc-web/;
        proxy_http_version 1.1;
      }
    }

nginx deployment and service:

apiVersion: v1
kind: Service
metadata:
  name: svc-frontend
spec:
  selector:
    app: nginx
    tier: frontend
  ports:
  - protocol: "TCP"
    port: 80
    targetPort: 80
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1 # Default is 1
  selector:
    matchLabels:
      app: nginx # Has to match .spec.template.metadata.labels
      tier: frontend
#      track: stable
  template:
    metadata:
      labels:
        app: nginx # Has to match .spec.selector.matchLabels
        tier: frontend
#        track: stable
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
#        image: k8s.gcr.io/nginx-slim:0.8
        image: nginx:latest
        lifecycle:
          preStop:
            exec:
              command: ["/usr/sbin/nginx","-s","quit"]
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /etc/nginx # mount nginx-config volume to /etc/nginx
          readOnly: true
          name: nginx-config-volume
        - mountPath: /var/log/nginx
          name: nginx-log-volume
      volumes:
      - name: nginx-config-volume
        configMap:
          name: nginx-config # place ConfigMap `nginx-conf` on /etc/nginx
          items:
            - key: nginx.config
              path: nginx.conf
            - key: virtualhost.config
              path: virtualhost/virtualhost.conf # dig directory
      - name: nginx-log-volume
        emptyDir: {}

$ k get all
NAME                                          READY   STATUS    RESTARTS   AGE
pod/blazegraph-0                              1/1     Running   0          2d1h
pod/default-http-backend-587b7d64b5-87kfk     1/1     Running   5          7d21h
pod/mysql-0                                   1/1     Running   0          2d1h
pod/nginx-595d6b7577-vmcng                    1/1     Running   0          5m45s
pod/nginx-ingress-microk8s-controller-slrtw   1/1     Running   408        7d21h
pod/web-0                                     1/1     Running   0          2d1h

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/default-http-backend   ClusterIP      10.152.183.78    <none>        80/TCP              7d21h
service/kubernetes             ClusterIP      10.152.183.1     <none>        443/TCP             8d
service/svc-db                 ClusterIP      None             <none>        3306/TCP,9999/TCP   2d1h
service/svc-frontend           LoadBalancer   10.152.183.176   <pending>     80:31219/TCP        5m45s
service/svc-web                ClusterIP      None             <none>        80/TCP              2d1h

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>          7d21h

NAME                                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend   1         1         1            1           7d21h
deployment.apps/nginx                  1         1         1            1           5m45s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/default-http-backend-587b7d64b5   1         1         1       7d21h
replicaset.apps/nginx-595d6b7577                  1         1         1       5m45s

NAME                          DESIRED   CURRENT   AGE
statefulset.apps/blazegraph   1         1         2d1h
statefulset.apps/mysql        1         1         2d1h
statefulset.apps/web          1         1         2d1h

NAME                                              REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/istio-pilot   Deployment/istio-pilot   <unknown>/55%   1         1         0          7d21h
$ k describe svc svc-frontend
Name:                     svc-frontend
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"svc-frontend","namespace":"default"},"spec":{"ports":[{"port":80,...
Selector:                 app=nginx,tier=frontend
Type:                     LoadBalancer
IP:                       10.152.183.176
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31219/TCP
Endpoints:                10.1.1.61:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
khteh@khteh-T580:/usr/src/kubernetes 2176 $ k get svc svc-frontend --watch
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
svc-frontend   LoadBalancer   10.152.183.176   <pending>     80:31219/TCP   5m23s

I expect the EXTERNAL-IP to be allocated so that I could curl / interact with the system. Please advise. Thanks.

[khteh]

One thing I found out from the logs:

$ k logs nginx --all-containers=true
Error from server (NotFound): pods "nginx" not found

This doesn't match pod/nginx-595d6b7577-vmcng from kubectl get all output. Why does the pod have the random name?

[ktsakalozos]

Hi @khteh ,

Kubernetes does not provide a loadbalancer. It is assumed that loadbalancers are an external component [1]. MicroK8s is not shipping any loadbalancer but even if it did there would not have been any nodes to balance load over. There is only one node so if you want to expose a service you should use the NodePort service type.

Why does the pod have the random name?

Pods get a random name because they are ephemeral. You are supposed to a access the virtual IP of a service.

[1] https://kubernetes.io/docs/concepts/services-networking/service/

[miraculixx]

@ktsakalozos IMO this remains unclear. Could you elaborate please?

According to the link you provide on services LoadBalancer _exposes the service externally using a cloud provider’s load balancer. NodePort and ClusterIP services, to which the external load balancer will route, are automatically created_

Indeed the NodePorts and the ClusterIP is created correctly for the LoadBalancer service, however no external IP is assigned. Thus in my understanding an external load balancer cannot route to the ClusterIP as it is not exposed.

In other words, the question is what is the canonical way in microk8s for the external LB to route to the ClusterIP?

[ktsakalozos]

Hi @miraculixx ,

There is no external LB shipping with MicroK8s, therefore there is no way to appoint an (LB provided) external IP to a service. What you can do is to expose a service to a host's port using NodePort.

Configuring a LB to provide external IPs is not trivial and it is LB specific. See for example the F5 LB installation instructions for Kubernetes (https://clouddocs.f5.com/containers/v1/kubernetes/).

what is the canonical way in microk8s for the external LB to route to the ClusterIP?

On your question, MicroK8s runs on a single host, the only available external IP is the hosts IP so use NodePort to appoint the host's IP to your service.

[khteh]

Hi, I have switched to NodePort and I am able to reach the Nginx service through the port. I am stuck at nginx refused connection to upstream service. Once I get passed this I would be able to deploy to AWS and use the LoadBalancer. I have opened another issue https://github.com/ubuntu/microk8s/issues/207. Any advice and insight is appreciated.

[ktsakalozos]

@khteh I am not familiar with configuring nginx, sorry.

[miraculixx]

There is no external LB shipping with MicroK8s,

@ktsakalozos Thanks. I understand MicroK8s comes without an LB but say I want to build such an LB, how can I do that?

I have been somewhat successful in using localhost to access the ports created by LoadBalancer with using microk8s.kubectl port-forward in that local ports are forwarded to the exposed service. However I'm not sure that's how it should work and how scalable that is (given that port-foward seems to start a proxy, which could be a bottleneck).

[miraculixx]

I am stuck at nginx refused connection to upstream service

I had the same problem. Did you enable DNS? Otherwise nginx cannot lookup the IP of the respective upstream service.

$ microk8s.enable DNS

[joaquin386]

If installing on bare metal like me (not in a cloud environment) had to install MetalLB after that an ExternalIP is assign to the service (after configure a configmaps)

[AntonOfTheWoods]

@joaquin386 , posting your example config would be much appreciated :-).

[trankchung]

@AntonOfTheWoods Follow this example here: https://metallb.universe.tf/tutorial/layer2/

I got it working in about 10 minutes using it with microk8s. The only thing is that I use a loopback address block for my LoadBalancer IP range. I use this CIDR: 127.0.0.64/27

[dvdblk]

@joaquin386 , posting your example config would be much appreciated :-).

EDIT: Nevermind. I got it working on a single (master) cluster. The externalIP is not <pending> anymore! Here is the config:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.64.140.43-10.64.140.49

[ktsakalozos]

It would be nice to have MetalLB as an addon we could enable with microk8s.enable metallb. This could be as easy as placing a metallb.yaml under https://github.com/ubuntu/microk8s/tree/master/microk8s-resources/actions

[VGerris]

Another option is to use NodePort instead of loadbalancer. That way a port will be assigned on the cluster IP that can be accessed directly.
A third option is to use the externalIPs option like so (where the IP is the cluster main ip for eth0 for example):

type: LoadBalancer
externalIPs:
- 10.12.11.30