Serving: Can not access Knative service via ".svc.cluster.local"

[Uvindu96]

In what area(s)?

/area API
/area networking

What version of Knative?

0.13.x

Expected Behavior

I have my Knative service which is deployed in the default namespace (istio-enabled) and
I have my second service called foo in another namespace called test (istio not enabled), I need to access my Knative service cluster locally through my foo service. In this use case my frontend service is foo service and backend service is Knative servive, I tried the same use case in Knative
version 0.8 and it worked fine. but recent version does not support this

Actual Behavior

When I try this It gives me 500 internal server error, I found that cluster-local-gateway service
not there in the istio-system namespace, if I add it manually it works fine.

*   Trying 35.193.12.181...
* TCP_NODELAY set
* Connected to 35.193.12.181 (35.193.12.181) port 9090 (#0)
> GET /node/1.0.0/hello/node HTTP/1.1
> Host: 35.193.12.181:9090
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< content-type: application/json
< content-length: 102
< server: ballerina/0.990.5
< date: Fri, 6 Mar 2020 06:50:37 GMT
< 
* Connection #0 to host 35.193.12.181 left intact
{"fault":{"code":101503, "message":"Runtime Error", "description":"Error connecting to the back end"}}

[MIBc]

I think host should be your ksvc hostname.

[Uvindu96]

Actually I'm not talking Knative service directly, It's my backend service.

[MIBc]

You can try this example command.
curl -H"HOST:helloworld-go-tmp-v1.default.svc.cluster.local" -H"Knative-Serving-Revision:helloworld-go-tmp-v1" -H"Knative-Serving-Namespace:default" http://10.233.23.22

The ip in the above example is a clusterip for service.

[tshak]

I appear to be having the same issue. I'm running on Kubernetes v1.15.4. This worked fine on Istio 1.3 and Knative 1.11. When I build the stack with Istio 1.5 (via the Istio Operator) and Knative 0.13.0 this error appears from one of my apps:

lookup riser-server.riser-system.svc.cluster.local on 10.96.0.10:53: lame referral

The cluster local gateway is present and configured the same way in each environment. When looking at the ksvc that I'm trying to reach, there's a notable difference with the service that KNative generates:

Knative v1.11: EXTERNAL-IP is set to my custom domain (e.g. service-name.namespace.custom-domain)
Knative v1.12.1 and v1.13.0: EXTERNAL-IP is set to cluster-local-gateway.istio-system.svc.cluster.local

Note: I had issues getting >=1.12 of Knative to work with my Istio 1.3 stack (received IngressNotConfigured errors on all routes), hence why I don't have test results. I need to move to Istio 1.5 soon anyway.

[Uvindu96]

@MIBc I figured out the problem, I think when I invoked my ksvc via cluster locally it's looking for a service called "cluster-local-gateway" in Istio-system namespace which is not there when we install Knative, but this worked fine in Knative v8.0.0,

[tshak]

Update: The cluster-local-gateway crd in the knative-serving namespace on all clusters and is functionally identical. I have a Istio1.3-Knative0.12.1 cluster which has the cluster-local-gateway service running in istio-system. However, the istio1.5-Knative0.13.0 cluster is missing the cluster-local-gateway service in istio-system. There are no errors in the istiod or istio-ingressgateway logs.

[Uvindu96]

@tshak yes exactly, I think when we install Knative it should install cluster-local-gateway on
Istio-system.

[tshak]

This issue is resolved for me. After ensuring that the Istio Operator is installing the cluster-local-gateway with the istio: cluster-local-gateway label. The label is critical as Knative uses this label selector to associate the gateway.

Kubernetes: v1.15.11
Knative: v0.14.0
Istio: v1.5.1

I recommend closing this issue.