Kubespray: Could not get node information: Get https://10.233.0.1:443/api/v1/nodes/node2: dial tcp 10.233.0.1:443: getsockopt: connection refused
*Is this a BUG REPORT (choose one):
Environment:
- hardware configuration:
- OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
Linux 3.10.0-957.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Version of Ansible (
ansible --version):
ansible 2.7.9
config file = None
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
Kubespray version (commit) (git rev-parse --short HEAD):
ea41fc5 or 2.8.3
Network plugin used:
default kubespray
Copy of your inventory file:
[all]
node1 ansible_host=192.168.56.3 ip=192.168.56.3
node2 ansible_host=192.168.56.6 ip=192.168.56.6
node3 ansible_host=192.168.56.4 ip=192.168.56.4
[kube-master]
node1
node2
[etcd]
node1
node2
node3
[kube-node]
node2
node3
[k8s-cluster:children]
kube-master
kube-node
Logs https://github.com/patsevanton/collect-info-kubespray-2.8.3
kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-5sh4b 0/1 CrashLoopBackOff 17 42m
kube-system calico-kube-controllers-6f7b5ccd5c-zk7gq 1/1 Running 0 43m
kube-system calico-node-dd7wk 1/1 Running 0 43m
kube-system calico-node-npkwq 1/1 Running 0 43m
kube-system calico-node-zg9cl 1/1 Running 0 43m
kube-system coredns-788d98cc7b-j5jmk 1/1 Running 0 40m
kube-system coredns-788d98cc7b-n5qq6 1/1 Running 12 40m
kube-system dns-autoscaler-66b95c57d9-mcvv9 1/1 Running 0 40m
kube-system kube-apiserver-node1 1/1 Running 0 47m
kube-system kube-apiserver-node2 1/1 Running 0 45m
kube-system kube-controller-manager-node1 1/1 Running 0 46m
kube-system kube-controller-manager-node2 1/1 Running 0 45m
kube-system kube-proxy-527tf 1/1 Running 0 43m
kube-system kube-proxy-pvw6c 1/1 Running 0 43m
kube-system kube-proxy-s4f9q 1/1 Running 0 43m
kube-system kube-scheduler-node1 1/1 Running 0 47m
kube-system kube-scheduler-node2 1/1 Running 0 45m
kube-system kubernetes-dashboard-5db4d9f45f-7x54l 1/1 Running 0 40m
kube-system local-volume-provisioner-rs8f5 1/1 Running 0 42m
kube-system local-volume-provisioner-vbhwg 0/1 CrashLoopBackOff 13 42m
kube-system local-volume-provisioner-w8prj 1/1 Running 0 42m
kube-system nginx-proxy-node3 1/1 Running 0 45m
kube-system tiller-deploy-bf6884cdb-th2mq 1/1 Running 0 39m
kubectl logs local-volume-provisioner-vbhwg -n kube-system
I0318 08:27:48.728040 1 common.go:259] StorageClass "local-storage" configured with MountDir "/mnt/disks", HostDir "/mnt/disks", BlockCleanerCommand ["/scripts/quick_reset.sh"]
I0318 08:27:48.728170 1 main.go:42] Configuration parsing has been completed, ready to run...
I0318 08:27:48.728464 1 common.go:315] Creating client using in-cluster config
F0318 08:27:48.765779 1 main.go:64] Could not get node information: Get https://10.233.0.1:443/api/v1/nodes/node2: dial tcp 10.233.0.1:443: getsockopt: connection refused
kubectl logs ingress-nginx-controller-5sh4b -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.21.0
Build: git-b65b85cd9
Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------
nginx version: nginx/1.15.6
W0318 08:26:25.175351 7 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0318 08:26:25.178393 7 main.go:196] Creating API client for https://10.233.0.1:443
kubectl logs coredns-788d98cc7b-n5qq6 -n kube-system
E0318 08:28:22.909051 1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:313: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0318 08:28:22.909190 1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:311: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0318 08:28:22.909263 1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:318: Failed to list *v1.Namespace: Get https://10.233.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
patsevanton
All 24 comments
Hi!
I'm having the same issue, as the output say, we are missing the kubeconfig nor master, this instruct ingress-nginx how to connect with the apiserver, at least for me it's not running in https://10.233.0.1:443.
I've rise the debug level adding the arg:
- --v=10
at /etc/kubernetes/addons/ingress_nginx/ds-ingress-nginx-controller.yml.
I'm also looking at ingress-nginx troubleshooting but nothing yet. Did you solve it?
Thanks.
krahser
on 12 May 2019
Hello!
We had a very similar issue and we found a way to solve it.
Problematic Kubespray version & inventory settings
Kubespray version 2.10.0 (but we also had the exact same problem with 2.8.3).
Relevant addons.yml contents:
# Nginx ingress controller deployment
ingress_nginx_enabled: true
# ingress_nginx_host_network: false
# ingress_nginx_nodeselector:
# beta.kubernetes.io/os: "linux": ""
# ingress_nginx_tolerations:
# - key: "node-role.kubernetes.io/master"
# operator: "Equal"
# value: ""
# effect: "NoSchedule"
# ingress_nginx_namespace: "ingress-nginx"
# ingress_nginx_insecure_port: 80
# ingress_nginx_secure_port: 443
# ingress_nginx_configmap:
# map-hash-bucket-size: "128"
# ssl-protocols: "SSLv2"
# ingress_nginx_configmap_tcp_services:
# 9000: "default/example-go:8080"
# ingress_nginx_configmap_udp_services:
# 53: "kube-system/kube-dns:53"
We used 3 nodes where each node is both a master and a worker.
See the attached file (hosts.yml) if interested, but it probably isn't too important.
Issue symptoms
root@kubernetes-01:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-6qgdd 0/1 Running 4 3m47s
ingress-nginx ingress-nginx-controller-6xrtc 0/1 Running 3 3m47s
ingress-nginx ingress-nginx-controller-lvh4s 0/1 Running 3 3m47s
kube-system calico-kube-controllers-5fb47fb8d6-9ww7f 1/1 Running 0 4m10s
kube-system calico-node-s5gmp 1/1 Running 0 4m19s
kube-system calico-node-ssrg9 1/1 Running 0 4m19s
kube-system calico-node-zdmpb 1/1 Running 0 4m19s
kube-system coredns-5d5bd8c6-h8n7h 1/1 Running 0 5m35s
kube-system coredns-5d5bd8c6-wssl9 1/1 Running 0 5m35s
kube-system coredns-7646874c97-vffp8 0/1 ContainerCreating 0 3m13s
kube-system dns-autoscaler-56c969bdb8-wtgkl 0/1 ContainerCreating 0 3m6s
kube-system kube-apiserver-kubernetes-01 1/1 Running 0 5m19s
kube-system kube-apiserver-kubernetes-02 1/1 Running 0 5m8s
kube-system kube-apiserver-kubernetes-03 1/1 Running 0 5m9s
kube-system kube-controller-manager-kubernetes-01 1/1 Running 0 5m20s
kube-system kube-controller-manager-kubernetes-02 1/1 Running 0 5m8s
kube-system kube-controller-manager-kubernetes-03 1/1 Running 0 5m9s
kube-system kube-proxy-6cbf6 1/1 Running 0 4m19s
kube-system kube-proxy-f4s58 1/1 Running 0 4m4s
kube-system kube-proxy-mzjvp 1/1 Running 0 4m8s
kube-system kube-scheduler-kubernetes-01 1/1 Running 0 5m9s
kube-system kube-scheduler-kubernetes-02 1/1 Running 0 5m8s
kube-system kube-scheduler-kubernetes-03 1/1 Running 0 5m9s
kube-system kubernetes-dashboard-6c7466966c-ljd96 0/1 ContainerCreating 0 2m59s
kube-system local-volume-provisioner-bqxdz 0/1 ContainerCreating 0 3m35s
kube-system local-volume-provisioner-wm2tx 0/1 ContainerCreating 0 3m35s
kube-system local-volume-provisioner-z8bfh 0/1 ContainerCreating 0 3m35s
kube-system nodelocaldns-82nxf 1/1 Running 0 3m2s
kube-system nodelocaldns-8z4p4 1/1 Running 0 3m2s
kube-system nodelocaldns-tx2f2 1/1 Running 0 3m2s
kube-system tiller-deploy-86c8b7897c-bvqz5 0/1 ContainerCreating 0 2m45s
root@kubernetes-01:~# kubectl logs ingress-nginx-controller-6xrtc -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.21.0
Build: git-b65b85cd9
Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------
nginx version: nginx/1.15.6
W0513 11:14:48.670870 8 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0513 11:14:48.671080 8 main.go:196] Creating API client for https://10.233.0.1:443
Solution to the issue
Use ingress_nginx_host_network: true within addons.yml.
Now all pods should be running as intended.
root@kubernetes-01:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-2jh48 1/1 Running 0 2m49s
ingress-nginx ingress-nginx-controller-c6mbw 1/1 Running 0 2m49s
ingress-nginx ingress-nginx-controller-tvglx 1/1 Running 0 2m49s
kube-system calico-kube-controllers-5fb47fb8d6-cpllv 1/1 Running 0 3m8s
kube-system calico-node-7wbwr 1/1 Running 0 3m17s
kube-system calico-node-fftb8 1/1 Running 0 3m17s
kube-system calico-node-l7dpj 1/1 Running 0 3m17s
kube-system coredns-7646874c97-62kq7 1/1 Running 0 2m9s
kube-system coredns-7646874c97-nk9mj 1/1 Running 0 2m20s
kube-system dns-autoscaler-56c969bdb8-4lgcq 1/1 Running 0 2m16s
kube-system kube-apiserver-kubernetes-01 1/1 Running 0 4m28s
kube-system kube-apiserver-kubernetes-02 1/1 Running 0 4m14s
kube-system kube-apiserver-kubernetes-03 1/1 Running 0 4m15s
kube-system kube-controller-manager-kubernetes-01 1/1 Running 0 4m15s
kube-system kube-controller-manager-kubernetes-02 1/1 Running 0 4m14s
kube-system kube-controller-manager-kubernetes-03 1/1 Running 0 4m15s
kube-system kube-proxy-kfq4g 1/1 Running 0 2m45s
kube-system kube-proxy-nwdzm 1/1 Running 0 2m55s
kube-system kube-proxy-pwtp8 1/1 Running 0 3m5s
kube-system kube-scheduler-kubernetes-01 1/1 Running 0 4m15s
kube-system kube-scheduler-kubernetes-02 1/1 Running 0 4m14s
kube-system kube-scheduler-kubernetes-03 1/1 Running 0 4m15s
kube-system kubernetes-dashboard-6c7466966c-9thp7 1/1 Running 0 2m12s
kube-system local-volume-provisioner-5lgbd 1/1 Running 0 2m39s
kube-system local-volume-provisioner-ql955 1/1 Running 0 2m39s
kube-system local-volume-provisioner-s656l 1/1 Running 0 2m39s
kube-system nodelocaldns-4f5kf 1/1 Running 0 2m14s
kube-system nodelocaldns-js6lw 1/1 Running 0 2m14s
kube-system nodelocaldns-vdkpx 1/1 Running 0 2m14s
kube-system tiller-deploy-86c8b7897c-gwqgb 1/1 Running 0 2m1s
MarioUhrik
on 13 May 2019
Thank you very much!
That solved it.
krahser
on 14 May 2019
Do you understand and can explain a cause why changing ingress_nginx_host_network to true helps to solve the problem. We had similar issue and this is also resolved just by changing it but I don't understand why exactly it helps.
danielszot
on 5 Jun 2019
When not running in hostNetwork: true mode, the --kubeconfig or --master must be specified.
woopstar
on 5 Jun 2019
In roles/kubernetes-apps/ingress_controller/ingress_nginx/templates/ds-ingress-nginx-controller.yml.j2 under ingress-nginx-controller args, right?
Should this be considered as kubespray issue or this is missing by purpose? If yes then I can create PR when I make sure it is solved.
danielszot
on 5 Jun 2019
When not running in
hostNetwork: truemode, the--kubeconfigor--mastermust be specified.
Could you please be more specific? Where should these flags be specified?
bernardhalas
on 6 Jun 2019
When not running in
hostNetwork: truemode, the--kubeconfigor--mastermust be specified.
i cannot find --master in docs https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/cli-arguments.md
also i think using --kubeconfig would be better, but then there should be conditional in that PR about if host_network is true then do no add this setting. Also this means kubeconfig should be mounted inside nginx container.
Atoms
on 12 Jun 2019
Hi,
I have the same problem. Using ingress_nginx_host_network: true option the issue is solved. But I don't understand why this happens.
If you run the playbook cluster.yml again or if you change ipvs to iptables in kube_proxy_mode, it also works.
jilgue
on 26 Jun 2019
fresh installed cluster with ipvs, (previously iptables, worked perfectly) installed over OpenStack
pods on nodes do not start:
....847fc7dc84-96f8x 0/1 ContainerCreating 0 25m
##Describe...
Get https://[10.233.0.1]:443/api/v1/namespaces/default: dial tcp 10.233.0.1:443: connect: connection refused
ssh into node
root@k8s-node-1:/home/ubuntu# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.17.0.1:30974 rr persistent 10800
TCP 192.168.16.215:30974 rr persistent 10800
TCP 192.168.16.215:32512 rr persistent 10800
TCP 10.233.0.1:443 rr
-> 192.168.16.211:6443 Masq 1 0 0
-> 192.168.16.212:6443 Masq 1 0 0
-> 192.168.16.213:6443 Masq 1 0 0
TCP 10.233.0.3:53 rr
-> 10.233.80.3:53 Masq 1 0 0
-> 10.233.96.3:53 Masq 1 0 0
TCP 10.233.0.3:9153 rr
-> 10.233.80.3:9153 Masq 1 0 0
-> 10.233.96.3:9153 Masq 1 0 0
TCP 10.233.8.222:443 rr
-> 10.233.96.2:8443 Masq 1 0 0
TCP 10.233.19.7:44134 rr
TCP 10.233.38.201:80 rr
TCP 10.233.39.49:80 rr persistent 10800
TCP 10.233.46.158:443 rr
TCP 10.233.117.0:30974 rr persistent 10800
TCP 127.0.0.1:30974 rr persistent 10800
TCP 127.0.0.1:32512 rr persistent 10800
TCP 169.254.25.10:30974 rr persistent 10800
TCP 169.254.25.10:32512 rr persistent 10800
TCP 172.17.0.1:32512 rr persistent 10800
TCP 10.233.32.63:80 rr persistent 10800
TCP 10.233.117.0:32512 rr persistent 10800
UDP 10.233.0.3:53 rr
-> 10.233.80.3:53 Masq 1 0 0
-> 10.233.96.3:53 Masq 1 0 0
testing connectivity
root@k8s-node-1:/home/ubuntu# curl -k https://[10.233.0.1]:443/api/v1/namespaces/default
curl: (7) Failed to connect to 10.233.0.1 port 443: Connection refused
root@k8s-node-1:/home/ubuntu# curl -k https://192.168.16.211:6443/api/v1/namespaces/default
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
Something is wrong in ipvs setup.. :-?
Note: From master nodes, curl -k https://[10.233.0.1]:443/api/v1/namespaces/default works..
radut
on 3 Jul 2019
I've experienced the same. At my side problem was also visible on ipvs. I didn't test iptables, yet. Setting hostNetwork to true helps. However I consider this as workaround and I would like to find a root cause too.
danielszot
on 3 Jul 2019
Can you try iptables ? You have your cluster BM or over cloud ?
radut
on 3 Jul 2019
I tried BM and AWS the problem was repeatable on both. On AWS I set:
cloud_provider: "aws"
I will try to test iptables. Can't promise I'll be able to do it today.
danielszot
on 3 Jul 2019
I found one more (alternative) workaround that uses ingress_nginx_host_network: false.
Note that Kubespray uses hostPort by default through these settings:
(/inventory/mycluster/group_vars/k8s-cluster/addons.yml)
# ingress_nginx_insecure_port: 80
# ingress_nginx_secure_port: 443
(roles/kubernetes-apps/ingress_controller/ingress_nginx/defaults/main.yml)
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
When hostPort is not used, all the pods suddenly start working despite ingress_nginx_host_network: false.
You can achieve this by either launching Kubespray with the default settings and then removing hostPort from the ingress-nginx-controller DaemonSet object using kubectl edit:
```
name: ingress-nginx-controller
ports:
- containerPort: 80
hostPort: 80
name: http
protocol: TCP
- containerPort: 443
hostPort: 443
name: https
protocol: TCP
->
```
name: ingress-nginx-controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
or just use ingress_nginx_insecure_port: null and ingress_nginx_secure_port: null in Kubespray addons.yml:
# Nginx ingress controller deployment
ingress_nginx_enabled: true
ingress_nginx_host_network: false
# ingress_nginx_nodeselector:
# beta.kubernetes.io/os: "linux": ""
# ingress_nginx_tolerations:
# - key: "node-role.kubernetes.io/master"
# operator: "Equal"
# value: ""
# effect: "NoSchedule"
# ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: null
ingress_nginx_secure_port: null
# ingress_nginx_configmap:
# map-hash-bucket-size: "128"
# ssl-protocols: "SSLv2"
# ingress_nginx_configmap_tcp_services:
# 9000: "default/example-go:8080"
# ingress_nginx_configmap_udp_services:
# 53: "kube-system/kube-dns:53"
Summary
The problem symptoms disappear when either
ingress_nginx_host_network: true, or
ingress_nginx_insecure_port: null and ingress_nginx_secure_port: null.
Could someone please explain this ? What's going on here ?
I read here that using hostPort blocks other applications trying to use the same port, which may be what's happening here.
Also, interestingly enough, this source seems to suggest that hostNetwork and hostPort have some things in common, as in both are ways to expose an application outside the cluster.
MarioUhrik
on 3 Jul 2019
I do not see/ understand why ingress-nginx-controller is related to kubernetes api server..
Is apiserver exposed through ingress.. no in my case
tcp6 0 0 :::6443 :::* LISTEN 24896/kube-apiserve
I found one more (alternative) workaround that uses
ingress_nginx_host_network: false.Note that Kubespray uses
hostPortby default through these settings:
_(/inventory/mycluster/group_vars/k8s-cluster/addons.yml)_# ingress_nginx_insecure_port: 80 # ingress_nginx_secure_port: 443_(roles/kubernetes-apps/ingress_controller/ingress_nginx/defaults/main.yml)_
ingress_nginx_insecure_port: 80 ingress_nginx_secure_port: 443When
hostPortis not used, all the pods suddenly start working despiteingress_nginx_host_network: false.
You can achieve this by either launching Kubespray with the default settings and then removinghostPortfrom the ingress-nginx-controller DaemonSet object usingkubectl edit:name: ingress-nginx-controller ports: - containerPort: 80 hostPort: 80 name: http protocol: TCP - containerPort: 443 hostPort: 443 name: https protocol: TCP->
name: ingress-nginx-controller ports: - containerPort: 80 name: http protocol: TCP - containerPort: 443 name: https protocol: TCPor just use
ingress_nginx_insecure_port: nullandingress_nginx_secure_port: nullin Kubesprayaddons.yml:# Nginx ingress controller deployment ingress_nginx_enabled: true ingress_nginx_host_network: false # ingress_nginx_nodeselector: # beta.kubernetes.io/os: "linux": "" # ingress_nginx_tolerations: # - key: "node-role.kubernetes.io/master" # operator: "Equal" # value: "" # effect: "NoSchedule" # ingress_nginx_namespace: "ingress-nginx" ingress_nginx_insecure_port: null ingress_nginx_secure_port: null # ingress_nginx_configmap: # map-hash-bucket-size: "128" # ssl-protocols: "SSLv2" # ingress_nginx_configmap_tcp_services: # 9000: "default/example-go:8080" # ingress_nginx_configmap_udp_services: # 53: "kube-system/kube-dns:53"Summary
The problem symptoms disappear when either
ingress_nginx_host_network: true, oringress_nginx_insecure_port: nullandingress_nginx_secure_port: null.Could someone please explain this ? What's going on here ?
I read here that usinghostPortblocks other applications trying to use the same port, which may be what's happening here.
Also, interestingly enough, this source seems to suggest thathostNetworkandhostPorthave some things in common, as in both are ways to expose an application outside the cluster.
I can sort of explain what is going on. It begins with kube-proxy in ipvs-mode setting up a virtual service on
This results in something like the following happening in the nodes firewall:
Chain PREROUTING (policy ACCEPT 1 packets, 54 bytes)
pkts bytes target prot opt in out source destination
150 9283 cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */
91 5480 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
112 7329 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
9 565 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain CNI-HOSTPORT-DNAT (2 references)
pkts bytes target prot opt in out source destination
25 1500 CNI-DN-648204fb76460a8bfecc6 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "cni0" id: "e27b3dea6bf8b765d7b83fa60bfb1eb587bf012db4b96b3a2f8dd5375ad96bd3" */ multiport dports 80,443
Chain CNI-DN-648204fb76460a8bfecc6 (1 references)
pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.233.97.7 0.0.0.0/0 tcp dpt:80
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:80
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.233.97.7:80
25 1500 CNI-HOSTPORT-SETMARK tcp -- * * 10.233.97.7 0.0.0.0/0 tcp dpt:443
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:443
25 1500 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:10.233.97.7:443
Effectively "stealing" all packages destined for the kubernetes service and instead sending them to a pod that is unable to start because it needs to talk to said service. Instead the pod ends up talking to an unborn instance of itself.
Telling iptables to ignore packages bound for kube-ipvs0 might be a solution but I don't really know if that would lead to other issues or where to implement it.
Found this: https://github.com/kubernetes/kubernetes/issues/66103
qvicksilver
on 3 Jul 2019
so.. how do you suggest I change my following config (keeping the same functionality for ingress) ?
ingress_nginx_host_network: false
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
into :
ingress_nginx_host_network: true
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
so that port 80 and port 443 will still work and be bound to the host
or is better to set
ingress_nginx_host_network: false
ingress_nginx_insecure_port: null
ingress_nginx_secure_port: null
and this way... :80 and :443 will still respond to external calls. The only thing is that now, they are routed trough ipvs ?
radut
on 3 Jul 2019
i have tested :
ingress_nginx_host_network: true
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
everything seems to work perfectly pods + nginx ingress
radut
on 3 Jul 2019
Do you have an opportunity to test below also?
ingress_nginx_host_network: false
ingress_nginx_insecure_port: null
ingress_nginx_secure_port: null
I tested :
ingress_nginx_host_network: false
ingress_nginx_insecure_port: null
ingress_nginx_secure_port: null
EDITED
Pods work but ingress does not work the way I want to.
.....
http://192.168.16.215
curl: (7) Failed to connect to 192.168.16.215 port 80: Connection refused
but which one to chose(better/advantages) ? and why ?
best for upgrading later the nginx controller.. :-??
radut
on 4 Jul 2019
I would prefer not to set hostNetwork to true.
One reason is described here in Services section: https://kubernetes.io/docs/concepts/configuration/overview/
The other is that when you use hostNetwork, containers form this Pod has direct access to Node network interfaces. For example an application that listens on all network interfaces will be accessible on all network interfaces of the Node the Pod is scheduled on.
danielszot
on 5 Jul 2019
Update:
ingress_nginx_host_network: false
ingress_nginx_insecure_port: null
ingress_nginx_secure_port: null
ingress does not work
Event though I tested earlier.. now ingress does not work, I guess I was desploying different config last time I tried this. I will try to double check it.
EDITED
node on port is not open/routed.. and i get conn refused
http://192.168.16.215
curl: (7) Failed to connect to 192.168.16.215 port 80: Connection refused
So I will stick with the
ingress_nginx_host_network: true
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
which makes ingress work the way I want.
curl http://192.168.16.215
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.15.6</center>
</body>
</html>
Issue https://github.com/kubernetes/kubernetes/issues/66103 is fixed. Kubespray master uses cni version 0.8.1 currently, which has the fix in place.
I can now run nginx-ingress with 'hostNetwork: false' without problems.
Kubespray does not update the cni plugins in /opt/cni/bin unfortunately. I had to run '/opt/cni/bin# tar xvzf /tmp/releases/cni-plugins-linux-amd64-v0.8.1.tgz' on all nodes manually after the kubespray run. I will create a separate issue. Seems to be due to the ansible module unarchive that does not update existing files.
msteenhu
on 1 Aug 2019
Solution to the issue
Use
ingress_nginx_host_network: truewithinaddons.yml.
Now all pods should be running as intended.
it works for me, thks!
goooogs
on 21 Jun 2020
Related issues
butuzov
路
4Comments
supabibz
路
3Comments
njdevils9
路
3Comments
nghiepvo
路
3Comments
bosatsu
路
3Comments
Most helpful comment
Hello!
We had a very similar issue and we found a way to solve it.
Problematic Kubespray version & inventory settings
Kubespray version 2.10.0 (but we also had the exact same problem with 2.8.3).
Relevant
addons.ymlcontents:We used 3 nodes where each node is both a master and a worker.
See the attached file (hosts.yml) if interested, but it probably isn't too important.
Issue symptoms
Solution to the issue
Use
ingress_nginx_host_network: truewithinaddons.yml.Now all pods should be running as intended.