Serving: Unable to access services from a service in the same custom namespace

[xpepermint]

/area networking

I'm unable to access services from a service running in the same namespace. I'm running Docker for Mac with Kubernetes enabled and Knative 0.10 installed based on this guide.

I can access public internet domains (e.g. curl google.com) but requesting {name}.{namespace}.svc.cluster.local or {name}.{namespace}.customdomain.com returns curl: (6) Could not resolve host.

My service Docker image is super simple:

FROM node:current-alpine

WORKDIR /app
COPY package.json ./package.json
COPY . .

RUN npm install --only=production

CMD ["npm", "start"]

I enter the pod kubectl exec -it -n mynamespace {podname} /bin/sh and try accessing a function as curl {name}.mynamespace.customdomain.com, curl {name}.mynamespace.svc.cluster.local and curl -H 'Host: {name}.mynamespace.customdomain.com' http://localhost and similar.

Any idea on how to debug this issue?

[xpepermint]

Is this a bug in the latest Knative configuration? I can curl http://{ip} where the {ip} is a service name {name}-{revision}-private but other service-related IPs can not be reached. As already explained above, curl http://{name} and related does not work. /etc/resolv.conf looks good:

nameserver 10.12.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ... google.internal
options ndots:5

/etc/hosts shows the usual:

# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.8.2.3    echosrv5-nfnxn-deployment-6558f6486d-sddp7

Anyone?

[vagababov]

so you cannot curl echosrv5-nfnxn but can curl echosrv5-nfnxn-private ?

[xpepermint]

Yes. I have echosrv5 and echosrv6. I exec -it into echosrv6 and I get these results:

# curl http://echosrv5
curl: (6) Could not resolve host: echosrv5
# curl http://echosrv5-nfnxn
Error getting active endpoint: revision.serving.knative.dev "" not found
# curl http://echosrv5-nfnxn-private
Hello Go Sample v1 - echosrv5!

Here my list of services on the default namespace:

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP                                            PORT(S)             AGE
echosrv5                   ExternalName   <none>         cluster-local-gateway.istio-system.svc.cluster.local   <none>              150m
echosrv5-nfnxn             ClusterIP      10.12.13.90    <none>                                                 80/TCP              150m
echosrv5-nfnxn-metrics     ClusterIP      10.12.3.18     <none>                                                 9090/TCP,9091/TCP   150m
echosrv5-nfnxn-private     ClusterIP      10.12.1.191    <none>                                                 80/TCP,8022/TCP     150m
echosrv6                   ExternalName   <none>         cluster-local-gateway.istio-system.svc.cluster.local   <none>              4m55s
echosrv6-d5p5l             ClusterIP      10.12.8.166    <none>                                                 80/TCP              4m57s
echosrv6-d5p5l-metrics     ClusterIP      10.12.14.56    <none>                                                 9090/TCP,9091/TCP   4m58s
echosrv6-d5p5l-private     ClusterIP      10.12.12.55    <none>                                                 80/TCP,8022/TCP     4m57s

[xpepermint]

Maybe related https://github.com/kubernetes/kubernetes/issues/85604

[xpepermint]

I downgraded K8S docker for mac to the latest stable version (2.1.0.5) but it doesn't work. Resolving works for standard K8S services but not for Knative services.

If I read DNS logs like this:

$ kubectl logs coredns-6dcc67dcbc-gg4l6 --namespace=kube-system -f

I see a few lines when the curl request is performed:

2019-12-06T11:38:44.354Z [INFO] 10.1.0.35:50350 - 3836 "AAAA IN echosrv6.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.00013606s
2019-12-06T11:38:44.354Z [INFO] 10.1.0.35:50350 - 42735 "A IN echosrv6.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000088028s
2019-12-06T11:38:44.355Z [INFO] 10.1.0.35:58296 - 57516 "A IN echosrv6.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.00010032s
2019-12-06T11:38:44.355Z [INFO] 10.1.0.35:58296 - 48230 "AAAA IN echosrv6.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.000311491s

[xpepermint]

OK, I got it. You should follow Installing Istio for Knative and also install cluster local gateway. Don't follow tutorials on Istio website. This is what has been changed.