K3d: [BUG] pod can't find its own service name

[minhnguyenvato]

What did you do?

• How was the cluster created?

  • k3d create -n "d1" --workers 7 -x --tls-san="192.168.1.200"

• What did you do afterwards?

  • deploy helm a chart helm install --name vato-presto stable/presto

What did you expect to happen?

Chart deploy failed , Pod can't find its own service name ( hostname ) .

I tried to attach to coordinate pod and curl/ ping to it own service name

curl -v http://vato-presto:8080/ - ping vato-presto

but it is not working .

I tried deploy to another k8s it worked fine ( GKE , Minikube or a cluster created by kubespray )

I think pod can't find route to its service name

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture?

• Centos 6

Which version of k3d?

• k3d version v1.1.0

Which version of docker?

`
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.centos.x86_64
Go version: go1.10.3
Git commit: b2f74b2/1.13.1
Built: Wed May 1 14:55:20 2019
OS/Arch: linux/amd64
Experimental: false
`

[zeerorg]

Can you post the output for kubectl get pods --all-namespaces ? If the k3s' coredns pod is not online the pod to service mapping will not work.

[minhnguyenvato]

default       vato-presto-coordinator-68ddb58c57-dvp75   1/1     Running     0          84m
default       vato-presto-coordinator-68ddb58c57-jxfwk   1/1     Running     0          84m
default       vato-presto-worker-74f77979b-8stsm         1/1     Running     3          84m
default       vato-presto-worker-74f77979b-b2z6g         1/1     Running     11         84m
default       vato-presto-worker-74f77979b-b87b4         1/1     Running     5          84m
default       vato-presto-worker-74f77979b-qv47t         1/1     Running     8          84m
default       vato-presto-worker-74f77979b-rhlbb         1/1     Running     7          84m
kube-system   coredns-857cdbd8b4-qh6c4                   1/1     Running     0          16h
kube-system   helm-install-traefik-sqr2f                 0/1     Completed   0          16h
kube-system   svclb-traefik-667b974688-5rg55             2/2     Running     0          16h
kube-system   svclb-traefik-667b974688-sz492             2/2     Running     0          16h
kube-system   tiller-deploy-8458f6c667-q45hn             1/1     Running     0          16h
kube-system   traefik-55bd9646fc-c9v7h                   1/1     Running     0          16h

[minhnguyenvato]

kgs

vato-presto   ClusterIP    10.43.246.96   <none>        8080:32403/TCP   85m

I tried to run 2 instance of vato-presto-coordinator ( Shouldn't do this Presto not support this )
And It can find way to its service name
I think when vato-presto-coordinator-68ddb58c57-dvp75 curl to vato-presto , service will redirect to vato-presto-coordinator-68ddb58c57-jxfwk and revived.
When I reduce the deploy of vato-presto-coordinator to 1 , I can't curl to vato-presto service anymore from vato-presto-coordinator pod but from another pod is ok.
Seem like it's not support to find route from its own pod to its self service

[kkimdev]

@minhnguyenvato What's the output of docker logs k3d-d1-server ?

[iwilltry42]

Hi @minhnguyenvato, do you have any updates on this issue, so we can follow up? :+1:

[ravirdv]

I was able to see the behaviour using the step provided by @minhnguyenvato. It looks like this is due to https://github.com/rancher/k3s/issues/582

edit: confirmed, echoing "1" to cat /sys/devices/virtual/net/cni0/brif/veth*/hairpin_mode on node makes the service reachable from the pod hosting it.

[minhnguyenvan95]

I tried with helm chart redis , after install redis chart , I tried to execute these command in redis-master . Seem like It didn't find its own name service but another

I have no name!@redis-master-0:/$ redis-cli -h redis-master -p 6379 -a 'JAnz6QAKiz'
^C
I have no name!@redis-master-0:/$ redis-cli -h redis-slave -p 6379 -a 'JAnz6QAKiz'
redis-slave:6379> 

[minhnguyenvan95]

temporary solution to fix this : use docker to exec these below command in worker nodes
for veth in $(ls /sys/devices/virtual/net/cni0/brif/); do echo 1 > /sys/devices/virtual/net/cni0/brif/$veth/hairpin_mode; done
thank to @ravirdv

[iwilltry42]

Seems like this got resolved with k3s v0.10.0 which was just released: https://github.com/rancher/k3s/releases/tag/v0.10.0
Can someone confirm this?

[iwilltry42]

Fixed in recent versions of k3s