Serving: Cannot use service.default.svc.cluster.local DNS

[gadelkareem]

In what area(s)?

Remove the '> ' to select:

/area API

Other classifications:

/kind good-first-issue

Ask your question here:

I have a simple service

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello-world-knative-example
  namespace: default
spec:
  template:
    metadata:
      annotations:
        # Knative concurrency-based autoscaling (default).
        autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
        autoscaling.knative.dev/metric: concurrency
        # Target 10 requests in-flight per pod.
        autoscaling.knative.dev/target: "10"
        # Disable scale to zero with a minScale of 1.
        #autoscaling.knative.dev/minScale: "1"
        # Limit scaling to 20 pods.
        autoscaling.knative.dev/maxScale: "3"
      labels:
        knative-app: hello-world-knative-example
    spec:
      serviceAccountName: hello-world-knative-example-secrets-account
      containers:
        - image: {{image}}
          resources:
            requests:
              cpu: "10m"
              memory: "10Mi"
            limits:
              cpu: "20m"
              memory: "20Mi"

I am trying to call the service

nslookup hello-world-knative-example
Server:     169.254.25.10
Address:    169.254.25.10#53

*** Can't find hello-world-knative-example.default.svc.cluster.local: No answer

so if I check the service I can see this

$ k get svc | grep hello-world
hello-world-knative-example                 ExternalName   <none>          cluster-local-gateway.istio-system.svc.cluster.local   <none>                                                                                                                                       46h
hello-world-knative-example-q2grb           ClusterIP      10.233.60.213   <none>                                                 80/TCP                                                                                                                                       46h
hello-world-knative-example-q2grb-private   ClusterIP      10.233.33.112   <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP

why can't I use hello-world-knative-example.default.svc.cluster.local?

[Uvindu96]

I faced the same issue few days ago when I try to call my backend knative service via ...default.svc.cluster.local it gives me an error.

[vagababov]

Where are you trying to call it from and how?

[Uvindu96]

I have my backend knative service in the default namespace, and I have my frontend service in another namespace. When I try to access my backend service cluster locally it gives me an error 500 can't connect to backend. I face this problem in Knative version 0.12 previously I tried the same use case with version 0.8 and it worked fine.

[Uvindu96]

I think It's looking for a service called cluster-local-gateway in the namespace istio-system.

`apiVersion: v1
kind: Service
metadata:
name: cluster-local-gateway
namespace: istio-system
resourceVersion: "1579"
spec:
ports:
- name: status-port
port: 15020
protocol: TCP
targetPort: 15020
- name: http2
port: 80
protocol: TCP
targetPort: 80
- name: https
port: 443
protocol: TCP
targetPort: 443
- name: tcp
port: 31400
protocol: TCP
targetPort: 31400
- name: https-kiali
port: 15029
protocol: TCP
targetPort: 15029
- name: https-prometheus
port: 15030
protocol: TCP
targetPort: 15030
- name: https-grafana
port: 15031
protocol: TCP
targetPort: 15031
- name: https-tracing
port: 15032
protocol: TCP
targetPort: 15032
- name: tls
port: 15443
protocol: TCP
targetPort: 15443
selector:
app: istio-ingressgateway
istio: ingressgateway
release: istio

`
If I add this service, I can route with cluster.local DNS

[Uvindu96]

@vagababov can you please look into this..?

[vagababov]

cc @tcnghia

[gadelkareem]

@Uvindu96 I tried adding the service as you described but it did not help.
The problem is that I am using my own namespace for istio and knative and it seems that some service updates the external-ip of the main kservice after it is being created to link to a non-existing service cluster-local-gateway.istio-system.svc.cluster.local

$ kubectl get svc hello-world-knative-example --watch
NAME                            TYPE           CLUSTER-IP   EXTERNAL-IP                                            PORT(S)   AGE
hello-world-knative-example   ExternalName   <none>       hello-world-knative-example.mysystemnamespace.world.local   <none>    0s
hello-world-knative-example   ExternalName   <none>       cluster-local-gateway.istio-system.svc.cluster.local   <none>    0s

[Uvindu96]

@tcnghia any idea..?

[gadelkareem]

I fixed it by modifying the config-istio, config-domain configmaps and switching the selector for the local gateway in the https://github.com/knative/serving/releases/download/v0.12.1/serving.yaml as following

---
apiVersion: v1
data:
  example.com: ''
  svc.cluster.local: ''
kind: ConfigMap
metadata:
  labels:
    serving.knative.dev/release: v0.12.1
  name: config-domain
  namespace: mysystemnamespace
---
apiVersion: v1
data:
  gateway.knative-ingress-gateway: istio-ingressgateway.mysystemnamespace.svc.cluster.local
  local-gateway.cluster-local-gateway: istio-ingressgateway.mysystemnamespace.svc.cluster.local
kind: ConfigMap
metadata:
  labels:
    networking.knative.dev/ingress-provider: istio
    serving.knative.dev/release: v0.12.1
  name: config-istio
  namespace: mysystemnamespace
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    networking.knative.dev/ingress-provider: istio
    serving.knative.dev/release: v0.12.1
  name: cluster-local-gateway
  namespace: mysystemnamespace
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

Not 100% sure but the local gateway is kinda removed from Istio chart so the docs were not clear to me

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[knative-housekeeping-robot]

Stale issues rot after 30 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.
Rotten issues close after an additional 30 days of inactivity.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle rotten

[mattmoor]

Sounds like this was resolved.

In my experience 9 times out of 10 when foo.bar.svc.cluster.local doesn't work, the cluster-local gateway with Istio is not installed (or busted).