Rke: ingress with extra args: http-port and https-port contributes error for rke-ingress-controller-deploy-job

Created on 10 Oct 2018 · 5Comments · Source: rancher/rke

RKE version: v0.1.10

Docker version: 17.03.2-ce

Operating system and kernel: 16.04.3 LTS (Xenial Xerus) 4.4.0-103-generic

Bare-metal
Here is part of my ingress config:

ingress:
  provider: nginx
  node_selector:
    app: ingressrancher
  extra_args:
    http-port: 8181
    https-port: 442

rke with version v0.1.9 it deploys without error but with v0.1.10 it can't create rke-ingress-controller-deploy-job when extra_args http-port and https-port are present. When I delete the mentioned part of extra_args it deploys without error.

INFO[0051] [ingress] Setting up nginx ingress controller 
INFO[0051] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0051] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 
INFO[0051] [addons] Executing deploy job..              
WARN[0086] Failed to deploy addon execute job [rke-ingress-controller]: Failed to get job complete status: <nil> 
INFO[0086] [addons] Setting up user addons

kubectl get po --all-namespaces

NAMESPACE       NAME                                        READY   STATUS      RESTARTS   AGE
cattle-system   cattle-6767d6756-g494v                      1/1     Running     1          2d
cattle-system   cattle-cluster-agent-79686467cf-f4g6p       1/1     Running     0          2d
cattle-system   cattle-node-agent-76v7t                     1/1     Running     0          2d
cattle-system   cattle-node-agent-8nz7s                     1/1     Running     0          2d
cattle-system   cattle-node-agent-bhszz                     1/1     Running     0          11h
cattle-system   cattle-node-agent-dqd72                     1/1     Running     0          2d
cattle-system   cattle-node-agent-kdsvq                     1/1     Running     0          2d
cattle-system   cattle-node-agent-pd8kf                     1/1     Running     0          2d
cattle-system   cattle-node-agent-ttm97                     1/1     Running     0          2d
cattle-system   cattle-node-agent-vmp6j                     1/1     Running     0          2d
cattle-system   cattle-node-agent-xnbl5                     1/1     Running     0          2d
default         mongos-548d59c9b7-ftrxj                     1/1     Running     0          1d
default         mongoshades-5ffc964465-4ssnk                1/1     Running     0          1d
default         ubuntumy-8fc64f899-shpjq                    1/1     Running     0          2d
ingress-nginx   default-http-backend-797c5bc547-9ff7k       1/1     Running     0          2d
ingress-nginx   nginx-ingress-controller-8qc4n              1/1     Running     0          2d
ingress-nginx   nginx-ingress-controller-llwmv              1/1     Running     0          2d
kube-system     canal-5zf9b                                 3/3     Running     0          11h
kube-system     canal-9d2dg                                 3/3     Running     0          11h
kube-system     canal-9hqkl                                 3/3     Running     0          11h
kube-system     canal-gbwv8                                 3/3     Running     0          11h
kube-system     canal-ngrzf                                 3/3     Running     0          11h
kube-system     canal-pdfsx                                 3/3     Running     0          11h
kube-system     canal-sp67h                                 3/3     Running     0          11h
kube-system     canal-w6rw6                                 3/3     Running     0          11h
kube-system     canal-wfrss                                 3/3     Running     0          11h
kube-system     kube-dns-7588d5b5f5-65p2p                   3/3     Running     0          11h
kube-system     kube-dns-7588d5b5f5-dc8hc                   3/3     Running     0          2d
kube-system     kube-dns-7588d5b5f5-p2zms                   3/3     Running     1          2d
kube-system     kube-dns-autoscaler-5db9bbb766-vrf85        1/1     Running     0          2d
kube-system     kubernetes-dashboard-767dc7d4d-zwmf2        1/1     Running     1          2d
kube-system     metrics-server-97bc649d5-rrql7              1/1     Running     0          2d
kube-system     rke-ingress-controller-deploy-job-dnxfk     0/1     Error       0          13m
kube-system     rke-ingress-controller-deploy-job-f8dxf     0/1     Error       0          7m
kube-system     rke-ingress-controller-deploy-job-l6mjw     0/1     Error       0          1m
kube-system     rke-ingress-controller-deploy-job-pznrp     0/1     Error       0          23m
kube-system     rke-ingress-controller-deploy-job-r8lsp     0/1     Error       0          23m
kube-system     rke-ingress-controller-deploy-job-rt95m     0/1     Error       0          18m
kube-system     rke-ingress-controller-deploy-job-szt6z     0/1     Error       0          22m
kube-system     rke-ingress-controller-deploy-job-v28ft     0/1     Error       0          23m
kube-system     rke-ingress-controller-deploy-job-wn5k6     0/1     Error       0          21m
kube-system     rke-kubedns-addon-deploy-job-2h4s4          0/1     Completed   0          11h
kube-system     rke-metrics-addon-deploy-job-w62q9          0/1     Completed   0          11h
kube-system     rke-network-plugin-deploy-job-n4kg2         0/1     Completed   0          11h
kube-system     rke-user-addon-deploy-job-m5wk8             0/1     Completed   0          11h
kube-system     rke-user-includes-addons-deploy-job-kv58k   0/1     Completed   0          11h

kubectl logs rke-ingress-controller-deploy-job-pznrp -n kube-system

namespace/ingress-nginx configured
configmap/nginx-configuration configured
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/nginx-ingress-serviceaccount unchanged
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole configured
role.rbac.authorization.k8s.io/nginx-ingress-role unchanged
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding configured
error: error parsing /etc/config/rke-ingress-controller.yaml: error converting YAML to JSON: yaml: line 52: did not find expected key

kinbug

Source

harutyundermenjyan

👍1

Most helpful comment

I got an similiar issue trying to use the default-ssl-certificate attributes in the extraArgs for the nginx-ingress-controller.

Using kubectl logs for the rke-ingress-controller-deploy-jobs I get the the same message:

error: error parsing /etc/config/rke-ingress-controller.yaml: error converting YAML to JSON: yaml: line 50: did not find expected key

When I look at https://github.com/rancher/rke/blob/v0.1.10/templates/nginx-ingress.go#L213 there seems to be a issue with the placement of extraArgs in the template. I would have expected it after line 205 (the normal args) and not after the SecurityContext stuff.

That seems to be a regression from https://github.com/rancher/rke/commit/67448c38c672c81d20004d5532fc0db464189d30

patst on 12 Oct 2018

👍2

All 5 comments

I got an similiar issue trying to use the default-ssl-certificate attributes in the extraArgs for the nginx-ingress-controller.

Using kubectl logs for the rke-ingress-controller-deploy-jobs I get the the same message:

error: error parsing /etc/config/rke-ingress-controller.yaml: error converting YAML to JSON: yaml: line 50: did not find expected key

That seems to be a regression from https://github.com/rancher/rke/commit/67448c38c672c81d20004d5532fc0db464189d30

patst on 12 Oct 2018

👍2

I got an similiar issue with default-ssl-certificate, workaround ?

corentin59 on 14 Oct 2018

I can confirm @patst's findings. We were trying to start ingress with a custom http/https ports and it was failing. The generated ConfigMap for the rke-ingress-controller is broken:

        - name: nginx-ingress-controller
          image: rancher/nginx-ingress-controller:0.16.2-rancher1
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            runAsUser: 33
            - --http-port=18088 <- this should be appended to the args section?
            - --https-port=18443 <- this should be appended to the args section?

The only workaround for us right now was after the initialization to edit the rke-ingress-controller and set the proper configuration. Afterwards just remove the containers from rke-ingress-controller-deploy-job workload and remove any dead containers from nginx-ingress-controller as well. Things are working just fine.