kind not properly supporting ingress-nginx

Created on 7 Aug 2019 · 16Comments · Source: kubernetes-sigs/kind

Keep in mind this worked in kubeadm-dind

What happened:
I installed the ingress-nginx as indicated at https://kubernetes.github.io/ingress-nginx/deploy/ use the mandatory file and the "baremetal" file, but any requests to my app return it's home page, rather than proper sub-paths

What you expected to happen:
I expected to have sub-paths passed to the back-end app

How to reproduce it (as minimally and precisely as possible):

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

I then ran a sample app...

kubectl run kubia --image=trentonadams/kubia:0.0.5 --port=8080 --replicas=5
kubectl expose deployment.apps/kubia --port=8080 --type=ClusterIP --name kubia-http

I then applied this ingress rule...

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: kubia-ingress
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - backend:
          serviceName: kubia-http
          servicePort: 8080
        path: /kubia

Then I access the main path and a sub-path and they get the same response...
curl http://api.example.com:30125/kubia-http/
You've hit kubia-7d6dbdb646-dsw6x

curl http://api.example.com:30125/kubia-http/subpath
You've hit kubia-7d6dbdb646-dsw6x

Subpath returns the following when ran in just docker (not kind)

$ curl http://localhost:8080/
You've hit 9e3e347ed5f3
$ curl http://localhost:8080/subpath
yep, you reached the sub-path

Anything else we need to know?:
Nope, It should just work.

Environment:

kind version: v0.5.0-alpha
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-25T23:41:27Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Docker version: Docker version 18.09.6, build 481bc77
OS (e.g. from /etc/os-release):

NAME="Linux Mint"
VERSION="18.2 (Sonya)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 18.2"
VERSION_ID="18.2"
HOME_URL="http://www.linuxmint.com/"
SUPPORT_URL="http://forums.linuxmint.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/linuxmint/"
VERSION_CODENAME=sonya
UBUNTU_CODENAME=xenial

kinbug

Source

TrentonAdams

Most helpful comment

@aledbf thanks, that works.

For posterity, I'll post the whole working example (in Powershell):

# Setup
kind create cluster --config kube-config.yaml
$env:KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

# https://github.com/helm/charts/tree/master/stable/nginx-ingress
helm install my-ingress stable/nginx-ingress --set controller.service.type=NodePort --set controller.service.nodePorts.http=30080  --set controller.service.nodePorts.https=30443

kubectl create -f service.yaml

# Wait for nginx to come online (120 seconds might not be enough)
timeout.exe 120
curl.exe http://localhost:30080/test
Pause

# Teardown
$env:KUBECONFIG=""
kind delete cluster --name kind

kube-config.yaml

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080

service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: srvc-nginx
spec:
  ports:
  - port: 80
  selector:
    srvc: nginx-selector
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: deployment-nginx
spec:
  template:
    metadata:
      labels:
        srvc: nginx-selector
    spec:
      containers:
      - name: working-nginx
        image: nginxdemos/hello # https://github.com/nginxinc/NGINX-Demos/tree/master/nginx-hello
        ports:
          - containerPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: example
spec:
  rules:
    - host: localhost
      http:
        paths:
          - backend:
              serviceName: srvc-nginx
              servicePort: 80
            path: /test
  # This section is only required if TLS is to be enabled for the Ingress
  # tls:
  #     - hosts:
  #         - www.example.com
  #       secretName: example-tls

# If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
# ---
# apiVersion: v1
# kind: Secret
# metadata:
#   name: example-tls
#   namespace: foo
# data:
#   tls.crt: <base64 encoded cert>
#   tls.key: <base64 encoded key>
# type: kubernetes.io/tls

Ciantic on 13 Sep 2019

❤5 🚀2

All 16 comments

I apologize, this is not true, it does not work in kubeadm-dind either. I don't know how I had it working.

TrentonAdams on 7 Aug 2019

FWIW ingress-nginx actually uses kind for CI, I'll dig into this a bit before reaching out to them if need be ...

BenTheElder on 7 Aug 2019

I apologize, this is not true, it does not work in kubeadm-dind either. I don't know how I had it working.

Ah, ok. Good luck with that!

BenTheElder on 7 Aug 2019

I decided to install and use helm to get it running. It then gave me instructions, and the proper ingress yaml to use.

For my application, this now means a proper load balancing of back-ends...

$ for i in {1..10}; do curl http://api.example.com:30076/health/node; echo; done
inotes-f759d4b66-2rvhx
inotes-f759d4b66-fmzdb
inotes-f759d4b66-qbqng
inotes-f759d4b66-qbqng
inotes-f759d4b66-2rvhx
inotes-f759d4b66-q5c8h
inotes-f759d4b66-q5c8h
inotes-f759d4b66-qbqng
inotes-f759d4b66-wvwfc
inotes-f759d4b66-fmzdb

TrentonAdams on 7 Aug 2019

So I got the ingress going on kind, but doing it via helm doesn't work at all. it claims about not having the permissions.

Is this a misunderstanding on my part for not properly configuring kind, or should I file a bug regarding this?

helm install stable/nginx-ingress --name my-ingress --namespace ingress

Error: release my-ingress failed: namespaces "ingress" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "ingress"

TrentonAdams on 7 Aug 2019

Maybe you need read https://kubernetes.io/docs/reference/access-authn-authz/authentication/ and https://helm.sh/docs/using_helm/#role-based-access-control

Your helm looks incorrectly configured.

tao12345666333 on 7 Aug 2019

Oh, I didn't realize kind had authorization stuff enabled. I've been using kubeadm-dind, so I never had an issue with it before. I suppose I need to investigate that a little. Thanks!

TrentonAdams on 7 Aug 2019

👍1

Ah yeah that'll do it. FWIW we're using kubeadm defaults for auth. I would expect recent clusters to have RBAC by default, kubeadm or otherwise, but this didn't even exist in earlier versions of Kubernetes 😅

BenTheElder on 8 Aug 2019

Odd, I never had a problem using helm with kubeadm-dind, perhaps they didn't include it by default.

TrentonAdams on 8 Aug 2019

You can skip it by using Helm v3, or using Helm v2 with tiller run local (one hack way).

tao12345666333 on 9 Aug 2019

I'm using helm3 but I think it's not possible to use stable/nginx-ingress chart because it uses LoadBalancer which is not supported by the kind:

PS C:\> helm install my-ingress stable/nginx-ingress
...
PS C:\> kubectl get all
NAME                                                            READY   STATUS    RESTARTS   AGE
pod/my-ingress-nginx-ingress-controller-58bf5b95-8s8js          0/1     Running   0          51s
pod/my-ingress-nginx-ingress-default-backend-7f649fbcb6-xkgc4   1/1     Running   0          51s

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/kubernetes                                 ClusterIP      10.96.0.1       <none>        443/TCP                      112s
service/my-ingress-nginx-ingress-controller        LoadBalancer   10.97.191.140   <pending>     80:30298/TCP,443:31142/TCP   51s
service/my-ingress-nginx-ingress-default-backend   ClusterIP      10.96.128.194   <none>        80/TCP                       51s

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-ingress-nginx-ingress-controller        0/1     1            0           51s
deployment.apps/my-ingress-nginx-ingress-default-backend   1/1     1            1           51s

NAME                                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-ingress-nginx-ingress-controller-58bf5b95          1         1         0       51s
replicaset.apps/my-ingress-nginx-ingress-default-backend-7f649fbcb6   1         1         1       51s

Notice that LoadBalancer is just pending, because it's not supported by kind.

Did someone get this working? I'm trying to make a proof of concept script running nginx ingress on kind.

Ciantic on 13 Sep 2019

👍1

@Ciantic That's normal for LoadBalancer to not be supported, it's a cloud only feature. Ingress controllers can do load balancing too, but LoadBalancer is reserved for cloud use.

https://kubernetes.io/docs/concepts/services-networking/

LoadBalancer: Exposes the Service externally using a cloud provider’s load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.

TrentonAdams on 13 Sep 2019

@TrentonAdams Okay I know that. But how are we supposed to use

helm install my-ingress stable/nginx-ingress

If it creates a LoadBalancer?

I tried to do switcheroo:

kubectl patch svc my-ingress-nginx-ingress-controller --type=json -p '[{"op": "replace", "path": "/spec/type","value":"NodePort"}, {"op": "replace", "path": "/spec/ports/0/nodePort", "value": 30080}, {"op": "replace", "path": "/spec/ports/1/nodePort", "value": 30443}]'

But it gives error:

The Service "my-ingress-nginx-ingress-controller" is invalid:
* spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
* spec.ports[1].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

I can however patch it to ClusterIP if I remove the ports, but isn't the only option to convert that LoadBalancer to a NodePort in order to use nginx ingress with kind?

Ciantic on 13 Sep 2019

@Ciantic using --set controller.service.type=NodePort (default is LoadBalancer)

aledbf on 13 Sep 2019

👍2

@aledbf thanks, that works.

For posterity, I'll post the whole working example (in Powershell):

# Setup
kind create cluster --config kube-config.yaml
$env:KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

# https://github.com/helm/charts/tree/master/stable/nginx-ingress
helm install my-ingress stable/nginx-ingress --set controller.service.type=NodePort --set controller.service.nodePorts.http=30080  --set controller.service.nodePorts.https=30443

kubectl create -f service.yaml

# Wait for nginx to come online (120 seconds might not be enough)
timeout.exe 120
curl.exe http://localhost:30080/test
Pause

# Teardown
$env:KUBECONFIG=""
kind delete cluster --name kind

kube-config.yaml

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080

service.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: srvc-nginx
spec:
  ports:
  - port: 80
  selector:
    srvc: nginx-selector
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: deployment-nginx
spec:
  template:
    metadata:
      labels:
        srvc: nginx-selector
    spec:
      containers:
      - name: working-nginx
        image: nginxdemos/hello # https://github.com/nginxinc/NGINX-Demos/tree/master/nginx-hello
        ports:
          - containerPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: example
spec:
  rules:
    - host: localhost
      http:
        paths:
          - backend:
              serviceName: srvc-nginx
              servicePort: 80
            path: /test
  # This section is only required if TLS is to be enabled for the Ingress
  # tls:
  #     - hosts:
  #         - www.example.com
  #       secretName: example-tls

# If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
# ---
# apiVersion: v1
# kind: Secret
# metadata:
#   name: example-tls
#   namespace: foo
# data:
#   tls.crt: <base64 encoded cert>
#   tls.key: <base64 encoded key>
# type: kubernetes.io/tls

Ciantic on 13 Sep 2019

❤5 🚀2

@Ciantic Using a single node as per you example it works just fine.
I added additional worker nodes and it works only if the pod has been deployed onto the node where I configured the extraPortMappings.
I tried basic examples (pod+service) using nodePort: they work regardless the node where the pod has been deployed (as expected according to the nodePort concept).
What am I missing?
I'm using kind on MacOs