Kind: āNodePortā-Service connection connection refused whether outside or inside the cluster
What happened:
Yesterday,i just created my HA-cluster with kind successfully, and today i just wanna test a few functions;So i create a service(NodePort) and a nginx Pod. Afer creating them,i tried to connect the nginx by NodeIP:NodePort ,but i got refused. Then i tried to connect the nginx inside the cluster, still got confused.
What you expected to happen: The connection could be done successfully
How to reproduce it (as minimally and precisely as possible):
the deployment.yaml :
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- name: http
containerPort: 80
the service.yaml:
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
spec:
selector:
app: nginx
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30001
after applying them,the status is below:
[root@localhost nginx-test]# kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-86c57db685-8cjqs 1/1 Running 0 17m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 23h
service/nginx NodePort 10.0.144.179 <none> 80:30001/TCP 16m
then i tried
[root@localhost nginx-test]# curl 10.0.144.179:80
curl: (7) Failed connect to 10.0.144.179:80;
[root@localhost nginx-test]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-86c57db685-8cjqs 1/1 Running 0 19m 10.244.5.2 test1-worker2 <none> <none>
[root@localhost nginx-test]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
98f0ba896de2 kindest/haproxy:2.1.1-alpine "/docker-entrypoint.ā¦" 24 hours ago Up 24 hours 127.0.0.1:32768->6443/tcp test1-external-load-balancer
cf5781bbaff1 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours test1-worker3
2d01ab56cb58 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours 127.0.0.1:32770->6443/tcp test1-control-plane
6166877552e9 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours test1-worker
9c4eaa853175 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours test1-worker2
e1168f4946b5 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours 127.0.0.1:32769->6443/tcp test1-control-plane2
635646d0fea0 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 24 hours ago Up 24 hours 127.0.0.1:32771->6443/tcp test1-control-plane3
[root@localhost nginx-test]# docker inspect --format='{{.NetworkSettings.IPAddress}}' 9c4eaa853175
172.17.0.8
then i tried 172.17.0.8:30001 on my host machine,still cannot connect it
Anything else we need to know?:
1ćping any website is ok
2ćping my host computer is ok
3ćthe kind cluster's creating config file isļ¼am i misssing something in the file?ļ¼:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
networking:
serviceSubnet: 10.0.0.0/16
imageRepository: registry.aliyuncs.com/google_containers
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
Environment:
- kind version: (use
kind version):kind v0.7.0 go1.13.6 linux/amd64 - Kubernetes version: (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
- Docker version: (use
docker info):
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:51 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:28:17 2019
OS/Arch: linux/amd64
Experimental: false
- OS (e.g. from
/etc/os-release):
CPE_NAME="cpe:/o:centos:centos:7"
Wenshiqi222
All 9 comments
please don't use these patches. (
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
networking:
serviceSubnet: 10.0.0.0/16
imageRepository: registry.aliyuncs.com/google_containers
)
kind has higher level config for service subnet and the kubeadm images should NOT be overridden, they're already present in the image and shouldn't be pulled! overriding this forces them to be pulled which unnecessarily slows everything down
https://kind.sigs.k8s.io/docs/user/configuration/#service-subnet
cc @aojea for nodeport
BenTheElder
on 18 Mar 2020
please don't use these patches. (
- | apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration metadata: name: config networking: serviceSubnet: 10.0.0.0/16 imageRepository: registry.aliyuncs.com/google_containers)
kind has higher level config for service subnet and the kubeadm images should NOT be overridden, they're already present in the image and shouldn't be pulled! overriding this forces them to be pulled which unnecessarily slows everything down
https://kind.sigs.k8s.io/docs/user/configuration/#service-subnetcc @aojea for nodeport
thanks for helpingļ¼i am confusing about the overriding: you mean
imageRepository: registry.aliyuncs.com/google_containers ļ¼ this command helps me to pull the necessary images which kubeadm requires, cause where i live cannot link the google resources~
Wenshiqi222
on 18 Mar 2020
kind has higher level config for service subnet
@Wenshiqi222 he meant something like this:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
serviceSubnet: 10.0.0.0/16
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
imageRepository: registry.aliyuncs.com/google_containers
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
In order to do curl 10.0.144.179:80
you have to install a route in your host (assuming is a Linux OS) to that IP through ANY of the KIND nodes that belong to the cluster (the Load Balancer does not belong to the cluster)
ip route add 10.0.144.179 via 172.17.0.8
However, something is wrong if you can't access the service in the port 172.17.0.8:30001
Can you ping 172.17.0.8?
Another useful tip for troubleshooting is connect directly from the nodes:
docker exec -it kind-worker curl 10.0.144.179
aojea
on 18 Mar 2020
kind has higher level config for service subnet
@Wenshiqi222 he meant something like this:
kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: serviceSubnet: 10.0.0.0/16 kubeadmConfigPatches: - | apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration metadata: name: config imageRepository: registry.aliyuncs.com/google_containers nodes: - role: control-plane - role: control-plane - role: control-plane - role: worker - role: worker - role: workerIn order to do
curl 10.0.144.179:80you have to install a route in your host (assuming is a Linux OS) to that IP through ANY of the KIND nodes that belong to the cluster (the Load Balancer does not belong to the cluster)
ip route add 10.0.144.179 via 172.17.0.8However, something is wrong if you can't access the service in the port
172.17.0.8:30001Can you ping 172.17.0.8?
Another useful tip for troubleshooting is connect directly from the nodes:
docker exec -it kind-worker curl 10.0.144.179
i recreate the cluster as @BenTheElder said, and here is my new config.yaml:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
metadata:
name: config
imageRepository: registry.aliyuncs.com/google_containers
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 8080
hostPort: 8080
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
this time, i got
[root@localhost ha]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
23e40193c37d kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes 0.0.0.0:8080->8080/tcp, 127.0.0.1:32770->6443/tcp kind-control-plane
c484dc6b58ca kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes kind-worker2
60ba6383310a kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes kind-worker
5a5af8c457e4 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes 127.0.0.1:32769->6443/tcp kind-control-plane2
7f9957e48e60 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes 127.0.0.1:32771->6443/tcp kind-control-plane3
d220b69f1d61 kindest/node:v1.17.0 "/usr/local/bin/entrā¦" 38 minutes ago Up 33 minutes kind-worker3
a2a3d8c52b7d kindest/haproxy:2.1.1-alpine "/docker-entrypoint.ā¦" 38 minutes ago Up 38 minutes 127.0.0.1:32768->6443/tcp kind-external-load-balancer
[root@localhost ha]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 31m
nginx NodePort 10.96.98.162 <none> 80:30001/TCP 20m
[root@localhost ha]# docker inspect --format='{{.NetworkSettings.IPAddress}}' 23e40193c37d
172.17.0.7
[root@localhost ha]# curl 172.17.0.7:30001
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
it works in the linux terminal,ļ¼ļ¼ļ¼
However! i still cannot get it on my host machine (windows browser ; Chrome),why is this happening?
Wenshiqi222
on 18 Mar 2020
However! i still cannot get it on my host machine (windows browser ; Chrome),why is this happening?
you have to provide more details about your setup, where is running KIND? a VM inside the windows host?
You have to provide connectivity from the Windows Host to your Kind Cluster ...
aojea
on 18 Mar 2020
However! i still cannot get it on my host machine (windows browser ; Chrome),why is this happening?
you have to provide more details about your setup, where is running KIND? a VM inside the windows host?
You have to provide connectivity from the Windows Host to your Kind Cluster ...
thanks for your helpļ¼now i can figure it out! just as you said, i didin't install a route from my Windows Host to the contianer,thank u !
@BenTheElder @aojea
Wenshiqi222
on 18 Mar 2020
That'll do it!
Thanks for working this out @aojea
BenTheElder
on 18 Mar 2020
thanks for helpingļ¼i am confusing about the overriding: you mean
imageRepository: registry.aliyuncs.com/google_containers ļ¼ this command helps me to pull the necessary images which kubeadm requires, cause where i live cannot link the google resources
Yes that. No images should be pulled. ALL images kubeadm requires are installed inside the kind node image. Them being pulled is a bug. But by setting this you force it to pull actually since they now don't match the already loaded ones, which is going to make kind perform worse and not work offline.
If you pull the kindest/node image and DONT supply any config like this you can disable your internet and still create working clusters. Nothing is pulled at runtime, it's all in the node image.
BenTheElder
on 18 Mar 2020
We're already specifically avoiding the problem of the infra not being available, you just have to obtain the kind node image which contains everything used at runtime.
BenTheElder
on 18 Mar 2020
Related issues
fgimenez
Ā·
4Comments
tommyknows
Ā·
3Comments
figo
Ā·
3Comments
cjwagner
Ā·
3Comments
leelavg
Ā·
3Comments