Serving: curl: (7) Failed to connect to helloworld-go.default.mydomain.com port 80: Connection refused
Knative on Google Kubernetes Engine - Running this:
https://knative.dev/docs/install/any-kubernetes-cluster/
Applying this hello world go:
https://knative.dev/docs/serving/samples/hello-world/helloworld-go/
I keep getting:
curl: (7) Failed to connect to helloworld-go.default.mydomain.com port 80: Connection refused
kubectl get ksvc helloworld-go
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.mydomain.com helloworld-go-ttgdz helloworld-go-ttgdz True
I have added this for my domain:
apiVersion: v1
kind: ConfigMap
metadata:
name: config-domain
namespace: knative-serving
data:
# example.org will be used for routes having app=prod.
# example.org: |
# selector:
# app: prod
# Default value for domain, for routes that does not have app=prod labels.
# Although it will match all routes, it is the least-specific rule so it
# will only be used if no other domain matches.
mydomain.com: |
Tried changing port: 8080 to 80 same result
JCzz
All 33 comments
what's you kubect get ksvc helloworld-go output?
vagababov
on 31 Mar 2020
@vagababov
kubectl get ksvc helloworld-go
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.mydomain.com helloworld-go-ttgdz helloworld-go-ttgdz True
So it seems that knative itself has started correctly.
Is your istio healthy?
e.g. are pods in istio-system all running? Pilot and ingressgateway?
Is kubectl get service -n istio-system istio-ingressgateway return a valid external IP?
Finally what does nslookup mydomain.com return? Does it match the IP above?
vagababov
on 31 Mar 2020
hmm, you have a point:
nslookup mydomain.com
returns:
Server: x.20.10.1
Address: x.20.10.1#53
Non-authoritative answer:
*** Can't find mydomain.com: No answer
I presume this is resolved.
/close
vagababov
on 1 Apr 2020
@vagababov: Closing this issue.
In response to this:
I presume this is resolved.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
knative-prow-robot
on 1 Apr 2020
@vagababov
Sorry, no it is not resolved - I am still trying.
I have tried changing my Google cloud DNS. But same result.
I can
dig http://helloworld-go.default.mydomain.com +short
returning the right - external ingress ip address
But:
curl http://helloworld-go.default.mydomain.com
Is still:
curl: (7) Failed to connect to helloworld-go.default.mydomain.com port 80: Connection refused
Any ideas are very welcome
JCzz
on 1 Apr 2020
I am just trying to deploy:
Note: helloworld-go is listening on port 8080, so I have tried to change containerPort back and forth from 80 <> 8080. Same result.
In this regards I have read that knative is defaulting to port 8080, it that true?
From here: https://cloud.ibm.com/docs/containers?topic=containers-serverless-apps-knative#knative-container-port
apiVersion: serving.knative.dev/v1 # Current version of Knative
kind: Service
metadata:
name: helloworld-go # The name of the app
namespace: default # The namespace the app will use
spec:
template:
spec:
containers:
- image: gcr.io/knative-samples/helloworld-go # The URL to the image of the app
env:
- name: TARGET # The environment variable printed out by the sample app
value: "Go Sample v1"
# ports:
# - name: http1
# containerPort: 80
Logs:
eyboard_arrow_right | 2020-04-01 20:07:48.936 | {"level":"info","ts":"2020-04-01T20:07:48.934Z","logger":"queueproxy","caller":"metrics/exporter.go:134","msg":"Successfully updated the metrics exporter; old config:Â ; new config &{knative.dev/internal/serving revision prometheus 5000000000Â false 9091 false { false}}","commit":"804b3bb","knative.dev/key":"default/helloworld-go-877sp","knative.dev/pod":"helloworld-go-877sp-deployment-6f495dc5d7-fgjqn"}
-- | -- | --
keyboard_arrow_right | 2020-04-01 20:07:48.936 | {"level":"info","ts":"2020-04-01T20:07:48.934Z","logger":"queueproxy","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/internal/serving revision prometheus 5000000000Â false 9091 false { false}}. Start the server for Prometheus exporter.","commit":"804b3bb","knative.dev/key":"default/helloworld-go-877sp","knative.dev/pod":"helloworld-go-877sp-deployment-6f495dc5d7-fgjqn"}
keyboard_arrow_right | 2020-04-01 20:07:48.936 | {"level":"info","ts":"2020-04-01T20:07:48.934Z","logger":"queueproxy","caller":"metrics/exporter.go:125","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"804b3bb","knative.dev/key":"default/helloworld-go-877sp","knative.dev/pod":"helloworld-go-877sp-deployment-6f495dc5d7-fgjqn"}
keyboard_arrow_right | 2020-04-01 20:07:48.933 | {"level":"info","ts":"2020-04-01T20:07:48.933Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
keyboard_arrow_right | 2020-04-01 20:07:48.933 | {"level":"info","ts":"2020-04-01T20:07:48.933Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
keyboard_arrow_right | 2020-04-01 20:07:48.665 | 2020/04/01 20:07:48 helloworld: listening on port 8080
keyboard_arrow_right | 2020-04-01 20:07:48.665 | 2020/04/01 20:07:48 helloworld: starting server...
Are the ksvs/route/revision/ingress all ready/succeeded?
vagababov
on 1 Apr 2020
@vagababov
kubectl get ksvc 1 ↵
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.mydomain.com helloworld-go-877sp helloworld-go-877sp True
kubectl get route
NAME URL READY REASON
helloworld-go http://helloworld-go.default.mydomain.com True
kubectl get revision
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
helloworld-go-877sp helloworld-go helloworld-go-877sp 1 True
kubectl get ingress
<nothing>
kubectl --namespace istio-system get service istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.124.4.210 <x.x.x.x> 15020:30582/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31663/TCP,15030:30481/TCP,15031:32355/TCP,15032:31407/TCP,15443:32648/TCP 32h
Sorry, I should have been more clear (kubectl get kingress)
But it seems that the service is ready.
Can you verify the following:
curl http://<istio-external-ip/ -H "Host: helloworld-go.default.<yourdomainhere>" -v
vagababov
on 2 Apr 2020
@vagababov Thanks, I am happy for any insights
kubectl get kingress
NAME READY REASON
helloworld-go True
curl http://x.x.x.x -H "Host: helloworld-go.default.mydomain.com" -v
* Trying x.x.x.x...
* TCP_NODELAY set
* Connection failed
* connect to x.x.x.x port 80 failed: Connection refused
* Failed to connect to x.x.x.x port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to x.x.x.x port 80: Connection refused
Info: I am on GKE, and I just tried to create a new cluster - to compare firewall rules - but looks the same.
Also I am on MacOs and tried to port scan
JCzz
on 2 Apr 2020
Info: the Istio installation is v0.13 "Installing Istio with SDS to secure the ingress gateway"
From:
https://knative.dev/docs/install/installing-istio/#installing-istio-with-sds-to-secure-the-ingress-gateway
Knative is also v0.13
From:
https://knative.dev/docs/install/any-kubernetes-cluster/
And Kubernetes:
kubectl version 7 ↵
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:33:14Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.9-gke.22", GitCommit:"9bae7fcacb520dfee658b26cc1a9643bf787dfc3", GitTreeState:"clean", BuildDate:"2020-02-27T18:43:37Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
And now 8 hours sleep :-)
JCzz
on 2 Apr 2020
cc @tcnghia @ZhiminXiang @nak3
my expertise ends here :-) But given you're using SDS perhaps you need to connect to port 443?
vagababov
on 2 Apr 2020
@vagababov Thanks so fare, appreciate it!
I will give it a go and let you know.
JCzz
on 2 Apr 2020
curl http://x.x.x.x:443 -H "Host: helloworld-go.default.mydomain.com" -v
* Trying x.x.x.x...
* TCP_NODELAY set
* Connection failed
* connect to x.x.x.x port 443 failed: Connection refused
* Failed to connect to x.x.x.x port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to x.x.x.x port 443: Connection refused
@JCzz what happens if you do the same curl, through a pod _inside_ your cluster and use the ClusterIP address instead of the external IP?
# Create a pod inside the cluster with curl
kubectl run --generator=run-pod/v1 curl-$RANDOM --image=radial/busyboxplus:curl -i --tty --rm
# Now you're inside the cluster
$ curl http://x.x.x.x -H "Host: helloworld-go.default.mydomain.com" -v
tcnghia
on 2 Apr 2020
@tcnghia
Cool, good idea:
Same result for :443
And :8080 gets "Connection timed out"
root@curl-11624:/ ]$ curl http://10.124.4.210 -H "Host: helloworld-go.default.mydomain.dev" -v
* Failed to connect to 10.124.4.210 port 80: Connection refused
curl: (7) Failed to connect to 10.124.4.210 port 80: Connection refused
Do you have network rules perhaps?
vagababov
on 2 Apr 2020
@vagababov on Google Cloud Platform?
JCzz
on 2 Apr 2020
My firewall rules - created by Knative/Istio is:
Description {"kubernetes.io/service-name":"istio-system/istio-ingressgateway", "kubernetes.io/service-ip":"x.x.x.x"}
Logs
Off
view
Network default
Priority 1000
Direction Ingress
Action on match Allow
Targets
Target tags
gke-mini-94da1bd4-node
Source filters
IP ranges
0.0.0.0/0
Protocols and ports
tcp:15020
tcp:80
tcp:443
tcp:31400
tcp:15029
tcp:15030
tcp:15031
tcp:15032
tcp:15443
Enforcement Enabled
In Istio?
Here's something you can try as well, to minimize the path.
In your kservice, add the following to the revision spec:
...
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/targetBurstCapacity: "0"
...
And redeploy.
See if that helps somehow?
vagababov
on 2 Apr 2020
Does the Ingress have running pod? Can you please try looking at the IPs listed in
kubectl get endpoints -n istio-system -o yaml
and hit one of those address:port with the same curl command (inside the cluster?) thanks.
tcnghia
on 2 Apr 2020
@tcnghia
From busyboxplus I tried:
curl http://10.60.0.51
It just returned nothing, which is good right?
as it has a port 80 open
kubectl get endpoints -n istio-system -o yaml
...
subsets:
- notReadyAddresses:
- ip: 10.60.0.51
nodeName: gke-mini-default-pool-15a06374-dftr
targetRef:
kind: Pod
name: istio-ingressgateway-66c8bbb77b-j99j6
namespace: istio-system
resourceVersion: "52041820"
uid: 93d64752-be43-443e-8c45-579fc2ba4a3b
ports:
- name: https-grafana
port: 15031
protocol: TCP
- name: https-kiali
port: 15029
protocol: TCP
- name: http2
port: 80
...
@vagababov
Also applied:
autoscaling.knative.dev/targetBurstCapacity: "0"
Even I dont know quite know what that means :-) but maybe something like autoscaling.knative.dev/maxScale: "1" but only doing spike in loads?
JCzz
on 2 Apr 2020
It makes sure activator is not in the request path. In general the value depends on your load, but it removes one variable from consideration.
You might want minScale=1 as well, to make sure activator is _never_ in the path.
If we eliminate that variable, you can remove those :)
vagababov
on 2 Apr 2020
@JCzz what does this show
kubectl get pods -n istio-system -listio=ingressgateway -o wide
Since your YAML only shows nonReadyAddresses I suspects your Ingress pods aren't ready, and as a result not hooked up to your ClusterIP or ExternalIP.
tcnghia
on 2 Apr 2020
Hi
kubectl get pods -n istio-system -listio=ingressgateway -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istio-ingressgateway-66c8bbb77b-j99j6 1/2 Running 0 45h **10.60.0.51** gke-mini-default-pool-15a06374-dftr <none> <none>
In busyboxplus, I tried:
curl 10.60.0.51 -v
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 10.60.0.51
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Thu, 02 Apr 2020 11:24:33 GMT
< server: istio-envoy
< content-length: 0
And:
curl 10.60.0.51:8080 -v
* Failed to connect to 10.60.0.51 port 8080: Connection refused
curl: (7) Failed to connect to 10.60.0.51 port 8080: Connection refused
Just to confirm access to kubernetes I did:
kubectl create deployment my-deployment --image=gcr.io/<mydomain>/samples/helloworld-go
kubectl expose deployment my-deployment --type=LoadBalancer --name=my-service --port=8080 --target-port=8080
And it works for standard Kubernetes deployment, but not for Istio & Knative:
curl x.xx.x:8080 130 ↵
Hello World!
I will try to create a new cluster with Istio(Installing Istio without sidecar injection) v0.13 & Knative v0.13
JCzz
on 2 Apr 2020
istio-ingressgateway-66c8bbb77b-j99j6 1/2 Running 0 45h
This is not good. Can you describe that pod to see what's not running?
vagababov
on 2 Apr 2020
@vagababov
Right
First, istio-system pods:
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-citadel-79894b8677-74mld 1/1 Running 0 2d4h
istio-galley-887c8c9fd-nc884 1/1 Running 0 2d4h
istio-ingressgateway-66c8bbb77b-j99j6 1/2 Running 0 2d4h
istio-pilot-5667f6589c-4q472 2/2 Running 1 2d4h
istio-pilot-5667f6589c-dchd6 2/2 Running 0 2d4h
istio-policy-54df456bd-k9ksd 2/2 Running 1 2d4h
istio-security-post-install-1.5.1-gldz9 0/1 Completed 0 2d4h
istio-sidecar-injector-6c767cdc56-pr8jt 1/1 Running 0 2d4h
istio-telemetry-5c75d9758b-xn6n2 2/2 Running 1 2d4h
istio-ingressgateway-66c8bbb77b-j99j6
kubectl describe pods istio-ingressgateway-66c8bbb77b-j99j6
Name: istio-ingressgateway-66c8bbb77b-j99j6
Namespace: istio-system
Priority: 0
Node: gke-awear-mini-default-pool-15a06374-dftr/10.132.0.20
Start Time: Tue, 31 Mar 2020 15:28:15 +0200
Labels: app=istio-ingressgateway
chart=gateways
heritage=Helm
istio=ingressgateway
pod-template-hash=66c8bbb77b
release=RELEASE-NAME
Annotations: sidecar.istio.io/inject: false
Status: Running
IP: 10.60.0.51
Controlled By: ReplicaSet/istio-ingressgateway-66c8bbb77b
Containers:
ingress-sds:
Container ID: docker://67d8d651525785d4b778920357c9e093f977ede5904aff54d3982b6442fad004
Image: docker.io/istio/node-agent-k8s:1.5.1
Image ID: docker-pullable://istio/node-agent-k8s@sha256:4d3adf12f4b8039766768689071de6099951131d59f438a5faab780370c4a549
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 31 Mar 2020 15:28:22 +0200
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Environment:
ENABLE_WORKLOAD_SDS: false
ENABLE_INGRESS_GATEWAY_SDS: true
INGRESS_GATEWAY_NAMESPACE: istio-system (v1:metadata.namespace)
Mounts:
/var/run/ingress_gateway from ingressgatewaysdsudspath (rw)
/var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-6hxfd (ro)
istio-proxy:
Container ID: docker://59ff351eb6dc5fe8951bdebd01db04df8664d402e1c44f37770e6ed259886637
Image: docker.io/istio/proxyv2:1.5.1
Image ID: docker-pullable://istio/proxyv2@sha256:3ad9ee2b43b299e5e6d97aaea5ed47dbf3da9293733607d9b52f358313e852ae
Ports: 15020/TCP, 80/TCP, 443/TCP, 31400/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 15090/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
proxy
router
--domain
$(POD_NAMESPACE).svc.cluster.local
--log_output_level=default:info
--drainDuration
45s
--parentShutdownDuration
1m0s
--connectTimeout
10s
--serviceCluster
istio-ingressgateway
--zipkinAddress
zipkin:9411
--proxyAdminPort
15000
--statusPort
15020
--controlPlaneAuthPolicy
NONE
--discoveryAddress
istio-pilot:15010
State: Running
Started: Tue, 31 Mar 2020 15:28:28 +0200
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 500m
memory: 256Mi
Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
NODE_NAME: (v1:spec.nodeName)
POD_NAME: istio-ingressgateway-66c8bbb77b-j99j6 (v1:metadata.name)
POD_NAMESPACE: istio-system (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
HOST_IP: (v1:status.hostIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
ISTIO_AUTO_MTLS_ENABLED: true
ISTIO_META_POD_NAME: istio-ingressgateway-66c8bbb77b-j99j6 (v1:metadata.name)
ISTIO_META_CONFIG_NAMESPACE: istio-system (v1:metadata.namespace)
ISTIO_METAJSON_LABELS: {"app":"istio-ingressgateway","chart":"gateways","heritage":"Helm","istio":"ingressgateway","release":"RELEASE-NAME"}
ISTIO_META_CLUSTER_ID: Kubernetes
SDS_ENABLED: false
ISTIO_META_WORKLOAD_NAME: istio-ingressgateway
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
ISTIO_META_USER_SDS: true
ISTIO_META_ROUTER_MODE: standard
Mounts:
/etc/certs from istio-certs (ro)
/etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
/etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
/var/run/ingress_gateway from ingressgatewaysdsudspath (rw)
/var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-6hxfd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
ingressgatewaysdsudspath:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.istio-ingressgateway-service-account
Optional: true
ingressgateway-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-certs
Optional: true
ingressgateway-ca-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-ca-certs
Optional: true
istio-ingressgateway-service-account-token-6hxfd:
Type: Secret (a volume populated by a Secret)
SecretName: istio-ingressgateway-service-account-token-6hxfd
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m11s (x94307 over 2d4h) kubelet, gke-awear-mini-default-pool-15a06374-dftr Readiness probe failed: HTTP probe failed with statuscode: 503
pods istio-security-post-install-1.5.1-gldz9
kubectl describe pods istio-security-post-install-1.5.1-gldz9
Name: istio-security-post-install-1.5.1-gldz9
Namespace: istio-system
Priority: 0
Node: gke-awear-mini-default-pool-15a06374-ddf1/10.132.0.21
Start Time: Tue, 31 Mar 2020 15:28:20 +0200
Labels: app=security
chart=security
controller-uid=a014bc31-866f-447c-badf-fcfeff108c7d
heritage=Helm
job-name=istio-security-post-install-1.5.1
release=RELEASE-NAME
Annotations: sidecar.istio.io/inject: false
Status: Succeeded
IP: 10.60.1.12
Controlled By: Job/istio-security-post-install-1.5.1
Containers:
kubectl:
Container ID: docker://999a395e1b04c2598d02cc801d5bd22d8dfd62c74a28df5fead3161ef8176d34
Image: docker.io/istio/kubectl:1.5.1
Image ID: docker-pullable://istio/kubectl@sha256:83ea57063cf3344a2462c5bbaa5b125810f2e8ef7283d2ba3bfd9393e624b80f
Port: <none>
Host Port: <none>
Command:
/bin/bash
/tmp/security/run.sh
/tmp/security/custom-resources.yaml
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 31 Mar 2020 15:28:32 +0200
Finished: Tue, 31 Mar 2020 15:28:45 +0200
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/tmp/security from tmp-configmap-security (rw)
/var/run/secrets/kubernetes.io/serviceaccount from istio-security-post-install-account-token-2bwl2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tmp-configmap-security:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-security-custom-resources
Optional: false
istio-security-post-install-account-token-2bwl2:
Type: Secret (a volume populated by a Secret)
SecretName: istio-security-post-install-account-token-2bwl2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Warning Unhealthy 2m11s (x94307 over 2d4h) kubelet, gke-awear-mini-default-pool-15a06374-dftr Readiness probe failed: HTTP probe failed with statuscode: 503
Well your ingress is unhealthy...
vagababov
on 2 Apr 2020
I have no idea how to fix this:
https://github.com/istio/istio/issues/12446
JCzz
on 3 Apr 2020
Related issues
vagababov
·
3Comments
greghaynes
·
4Comments
ahmetb
·
5Comments
xpepermint
·
6Comments
evankanderson
·
3Comments