Kfserving: Exception: Bad response from application: 503 when sending a prediction request through ingress and IAP for KFServing GCP/IAP Example
/kind bug
What steps did you take and what happened:
When trying KFServing GCP/IAP example from this page https://github.com/kubeflow/kfserving/tree/master/docs/samples/gcp-iap, I get a 404 error when testing the external predict endpoint using iap_request.py
Just for information, when sending requests, I am using the service account which was created when deploying Kubeflow 1.1 using gcp-blueprints (name of this service account is ${KF_NAME}-user)
I am note sure to understand the note at the end of the page :
_Making requests to the service may be blocked (403) by the new istio sidecar container until a new AuthorizationPolicy is added that allows access to this inference URI from a specified ServiceAccount or Namespace._
I am not sure this is the cause of the 404 error, but Please could you give some clarification about this note ? What kind of AuthorizationPolicy must be added and how ?
And how could I solve the 404 error below ?
What did you expect to happen:
Return code 200
Anything else you would like to add:
Here are the commands I executed
export PROJECT=xxxx
gcloud config set project ${PROJECT}
export ZONE=yyyyyy
gcloud config set compute/zone ${ZONE}
export KF_NAME=zzzzz
kubens kfnamespace
cd ~/repos/kubeflow/kfserving/docs/samples/gcp-iap/
kubectl apply -f sklearn-iap-with-authz.yaml
kubectl apply -f virtual-service.yaml
I check that the pod is successfully running
kubectl get po
NAME READY STATUS RESTARTS AGE
sklearn-iap-predictor-default-wl4zb-deployment-56fbfc47f4-m2vcr 3/3 Running 0 25s
I set the service account as the active account
export SERVICE_ACCOUNT=my_sa_name
export KEY_FILE=${HOME}/.config/gcloud/${SERVICE_ACCOUNT}-credentials.json
gcloud config set account ${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com
export GOOGLE_APPLICATION_CREDENTIALS=${KEY_FILE}
I get the value for INGRESS_DNS (which must be updated in make-prediction.sh)
kubectl -n istio-system get ingress
I update the variables in make-prediction.sh
vi make-prediction.sh
INFERENCE_SERVICE=sklearn-iap
INPUT_PATH=@./iris-input.json
PROJECT='xxxx'
NAMESPACE='kfnamespace'
INGRESS_DNS=${KF_NAME}.endpoints.${PROJECT}.cloud.goog
export IAP_CLIENT_ID='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.apps.googleusercontent.com'
SERVICE_URL=https://${INGRESS_DNS}/kfserving/${NAMESPACE}/${INFERENCE_SERVICE}:predict
I send the query
python iap_request.py https://zzzzz.endpoints.xxxx.cloud.goog/kfserving/kfnamespace/sklearn-iap:predict aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.apps.googleusercontent.com --input=./iris-input.json
I get the response below (error 404)
~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:979: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zzzzz.endpoints.xxxx.cloud.goog'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
Traceback (most recent call last):
File "iap_request.py", line 143, in <module>
main()
File "iap_request.py", line 136, in main
raise Exception('Bad response from application: {!r} / {!r} / {!r}'.format(
Exception: Bad response from application: 404 / {'x-powered-by': 'Express', 'content-security-policy': "default-src 'none'", 'x-content-type-options': 'nosniff', 'content-type': 'text/html; charset=utf-8', 'content-length': '173', 'date': 'Wed, 11 Nov 2020 08:53:44 GMT', 'x-envoy-upstream-service-time': '1', 'server': 'istio-envoy', 'Via': '1.1 google', 'Alt-Svc': 'clear'} / '<!DOCTYPE html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n<title>Error</title>\n</head>\n<body>\n<pre>Cannot POST /kfserving/kfnamespace/sklearn-iap:predict</pre>\n</body>\n</html>\n'
I checked that ${GOOGLE_APPLICATION_CREDENTIALS} points to the key json file, and it is OK
more ${GOOGLE_APPLICATION_CREDENTIALS}
I checked that the active account is the service account :
gcloud auth list
Environment:
- Istio Version:
- Knative Version:
- KFServing Version:
- Kubeflow version: build version v1beta1 (Kubeflow 1.1 deployed in GCP using gcp-blueprints)
- Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] : gcp-blueprints
- Minikube version: GCP
- Kubernetes version: (use
kubectl version): Major:"1", Minor:"17+", GitVersion:"v1.17.12-gke.1504 - OS (e.g. from
/etc/os-release):
jal06
All 27 comments
Hummm, I did a few errors when setting variables in make-prediction.sh
I didn't set correctly the variables GOOGLE_APPLICATION_CREDENTIALS and INGRESS_DNS
If I modify those 2 variables as below ,
export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/my_sa_name-credentials.json
INGRESS_DNS=kubeflow.endpoints.${PROJECT}.cloud.goog
(for INGRESS_DNS, the 1rst part is kubeflow instead of ${KF_NAME}
I get the error below
python iap_request.py https://kubeflow.endpoints.xxxx.cloud.goog/kfserving/kfnamespace/sklearn-iap:predict aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..apps.googleusercontent.com --input=./iris-input.json
Traceback (most recent call last):
File "/home/jal/anaconda3/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/home/jal/anaconda3/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/home/jal/anaconda3/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
After some investigations, I did the below modifications in make-prediction.sh and in virtual-service.sh and I get now an error 503 :
/home/jal/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:979: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zzzzz.endpoints.xxxx.cloud.goog'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
Traceback (most recent call last):
File "iap_request.py", line 143, in <module>
main()
File "iap_request.py", line 136, in main
raise Exception('Bad response from application: {!r} / {!r} / {!r}'.format(
Exception: Bad response from application: 503 / {'date': 'Thu, 12 Nov 2020 07:56:36 GMT', 'server': 'istio-envoy', 'content-length': '0', 'Via': '1.1 google', 'Alt-Svc': 'clear'} / ''
Below are the modifications I did :
1- in virtual-service.yaml
the gateway cluster-local-gateway is in the namespace knative-serving so the route.host.destination must be set to
host: cluster-local-gateway.knative-serving.svc.cluster.local
instead of
host: cluster-local-gateway.istio-system.svc.cluster.local
k get gateway --all-namespaces
NAMESPACE NAME AGE
istio-system ingressgateway 17d
knative-serving cluster-local-gateway 17d
kubeflow kubeflow-gateway 17d
2- in make-prediction.sh
I defined the variable KF_NAME as below
KF_NAME=zzzzz
(zzzzz is the name of my K8S cluster)
and I use this variable when setting INGRESS_DNS
INGRESS_DNS=${KF_NAME}.endpoints.${PROJECT}.cloud.goog
I removed the character @ from the variable INPUT_PATH
INPUT_PATH=./iris-input.json
instead of
INPUT_PATH=@./iris-input.json
jal06
on 12 Nov 2020
@jal06 are you sure cluster-local-gateway is running in the knative-serving namespace? The gateway resource is different from the actual local ingress gateway deployment pods.
yuzisun
on 18 Nov 2020
@yuzisun,
Thanks for your help.
When I search all the gateways in all namespaces (see below), I see that the gateway cluster-local-gateway is in the namespace knative-serving
kubectl get gateway --all-namespaces
NAMESPACE NAME AGE
istio-system ingressgateway 23d
knative-serving cluster-local-gateway 23d
kubeflow kubeflow-gateway 23d
It looks that the external-ip of services sklearn-iap and sklearn-iap-predictor-default refers to an object cluster-local-gateway which is in the namespace istio-system (see below)
kubectl get svc sklearn-iap -n jal
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sklearn-iap ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 6d22h
kubectl get svc sklearn-iap-predictor-default -n jal
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sklearn-iap-predictor-default ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 6d22h
md5-ee6f7d35f0daa7842f9ca87e15840547
kubectl get all --all-namespaces | grep local
jal service/sklearn-iap ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 6d22h
jal service/sklearn-iap-predictor-default ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 6d22h
md5-5a7d10f5e63aad254053153362281ec3
kubectl get svc sklearn-iap -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-11-11T08:48:15Z"
name: sklearn-iap
namespace: jal
ownerReferences:
- apiVersion: serving.kubeflow.org/v1alpha2
blockOwnerDeletion: true
controller: true
kind: InferenceService
name: sklearn-iap
...
spec:
externalName: cluster-local-gateway.istio-system.svc.cluster.local
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
md5-353a9827e375d146ea33a9231cea272d
kubectl get svc sklearn-iap-predictor-default -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-11-11T08:48:14Z"
labels:
serving.knative.dev/route: sklearn-iap-predictor-default
name: sklearn-iap-predictor-default
namespace: jal
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Route
name: sklearn-iap-predictor-default
...
spec:
externalName: cluster-local-gateway.istio-system.svc.cluster.local
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
Please, let me know if you would like more infos.
Thanks for your help.
jal06
on 18 Nov 2020
Below are the pods in the namespace istio-system
kubectl get po -n istio-system
NAME READY STATUS RESTARTS AGE
backend-updater-0 1/1 Running 0 23d
iap-enabler-6b884cc58-r9lwr 1/1 Running 0 23d
istio-galley-6cc8484c6d-lz25j 2/2 Running 0 23d
istio-galley-6cc8484c6d-smxf7 2/2 Running 0 23d
istio-ingressgateway-795f4f56fb-4vd4n 2/2 Running 0 23d
istio-ingressgateway-795f4f56fb-hmlrx 2/2 Running 0 23d
istio-nodeagent-6x5dm 1/1 Running 0 23d
istio-nodeagent-7w762 1/1 Running 0 8h
istio-nodeagent-dbhmq 1/1 Running 0 23d
istio-nodeagent-fvq2t 1/1 Running 0 18d
istio-pilot-6947c5b48c-8h5tg 2/2 Running 0 23d
istio-pilot-6947c5b48c-jjngp 2/2 Running 0 23d
istio-sidecar-injector-69c4b49dfc-4pr95 1/1 Running 0 23d
istio-sidecar-injector-69c4b49dfc-4v88h 1/1 Running 0 23d
promsd-86d8df4ff4-48ztd 2/2 Running 1 23d
whoami-app-569664b7f-px7s4 1/1 Running 0 23d
@jal06 The problem is that you are missing the istio local gateway installation. Please check here for the install instruction, or you can use the local gateway manifests from kubeflow
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
cluster-local-gateway-f4967d447-jdzgv 1/1 Running 0 35d
cluster-local-gateway-f4967d447-zbghp 1/1 Running 0 15d
grafana-6fc987bd95-skhhf 1/1 Running 0 35d
istio-citadel-55646d8965-pcd6m 1/1 Running 0 35d
istio-egressgateway-7bdb7bf7b5-qf24m 1/1 Running 0 35d
istio-galley-8f5fcd88b-z6fz2 1/1 Running 0 35d
istio-ingressgateway-64dbd4b954-qtr5g 1/1 Running 0 35d
istio-pilot-7bd4b44686-l6phj 2/2 Running 0 35d
istio-policy-6c6648b847-c75dr 2/2 Running 0 35d
istio-sidecar-injector-5d6ff6d758-k8rt2 1/1 Running 0 35d
istio-telemetry-55f585cd8c-bf8km 2/2 Running 0 35d
istio-tracing-669fd4b9f8-56kll 1/1 Running 0 35d
kiali-94f8cbd99-6sg7q 1/1 Running 0 35d
prometheus-776fdf7479-f774h 1/1 Running 0 35d
@yuzisun, thanks for your reply.
I don't understand why I am missing the istio local gateway installation.
@jlewi @Bobgy I followed the documentation for installation of Kubeflow 1.1 in GCP.
During the installation, I saw no error.
In the install documentation , I don't see any specific instruction about the installation of the istio local gateway.
Did I miss something ?
Is it expected that there is a gateway cluster-local-gateway in the namespace knative-serving ? Do I need to delete it ?
FYI, I expect to install a new Kubeflow in my dev environment (GCP) next week. Hopefully I will install the version 1.2 so I will check the status of the istio local gateway with the version 1.2 and I will let you know what I see
If I need to install the istio local gateway by myself, I will try to do it in my dev environment in version 1.2 using the local gateway manifests from kubeflow
Thanks.
jal06
on 18 Nov 2020
@jal06 Can you check the kfdef yaml? the istio local gateway needs to be defined e.g here . Also do not get confused with Istio gateway resource and Istio ingress gateway deployment, gateway resource cluster-local-gateway in knative-serving namespace defines the listeners for the Istio local gateway deployment, the actual istio local gateway deployment runs in istio-system.
yuzisun
on 19 Nov 2020
Hi @yuzisun
Once again, thanks very much for your help.
You are right, thanks for your explanations. If my understanding is correct, the listeners for the Istio local gateway in knative-serving are OK but the istio local gateway in istio-system is missing.
If I look at the yaml files I used to deploy Kubeflow, I can find the yaml definition of the istio local gateway in the upstream/manifests directory
However I don't understand why in the .build directory, I find only the yaml files for deploying the listeners in the namespace knative-serving and no yaml file for deploying the istio local gateway cluster-local-gateway in the namespace istio-system (if I grep for the string cluster-local-gateway in the .build directory, I find only the definition of the listeners as you can see below).
IMHO, if we need to do some additional actions to install the istio local gateway in istio-system, it would be very useful to describe them in the install documentation.
Please find below some details from my install directory
In the upstream directory, I find the file ./upstream/manifests/kfdef/kfctl_k8s_istio.yaml (which aims to kustomize packages in namespace istio-system )
-- from line 23
- kustomizeConfig:
parameters:
- name: namespace
value: istio-system
repoRef:
name: manifests
path: istio/cluster-local-gateway
name: cluster-local-gateway
In the .build directory, I find only 2 files referring to cluster-local-gateway (1 configmap and 1 gateway)
cd ./.build/
grep -r 'cluster-local-gateway' .
./knative/~g_v1_configmap_config-istio.yaml: local-gateway.knative-serving.cluster-local-gateway: cluster-local-gateway.istio-system.svc.cluster.local
./knative/networking.istio.io_v1alpha3_gateway_cluster-local-gateway.yaml: name: cluster-local-gateway
./knative/networking.istio.io_v1alpha3_gateway_cluster-local-gateway.yaml: istio: cluster-local-gateway
- 1rst file =
./knative/~g_v1_configmap_config-istio.yaml
apiVersion: v1
data:
gateway.kubeflow.kubeflow-gateway: istio-ingressgateway.istio-system.svc.cluster.local
local-gateway.knative-serving.cluster-local-gateway: cluster-local-gateway.istio-system.svc.cluster.local
local-gateway.mesh: mesh
reconcileExternalGateway: "false"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: knative-serving-install
app.kubernetes.io/name: knative-serving-install
kustomize.component: knative
networking.knative.dev/ingress-provider: istio
serving.knative.dev/release: v0.11.2
name: config-istio
namespace: knative-serving
- 2nd file =
./knative/networking.istio.io_v1alpha3_gateway_cluster-local-gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
labels:
app.kubernetes.io/component: knative-serving-install
app.kubernetes.io/name: knative-serving-install
kustomize.component: knative
networking.knative.dev/ingress-provider: istio
serving.knative.dev/release: v0.11.2
name: cluster-local-gateway
namespace: knative-serving
spec:
selector:
istio: cluster-local-gateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
Thanks for the investigations, it seems like a bug in gcp manifests.
@jal06 you can look at the makefile to know how manifests are hydrated to .build, you should add the missing resource in one of the kustomize targets we build.
Will be grateful if you can contribute the fix upstream if you verify the fix works for you.
Bobgy
on 20 Nov 2020
@Bobgy, thanks for your guidance,
After digging into the manifests file, some manifests file for cluster-local-gateway are now generated in the .build directory after executing make hydrate.
For generating those manifests, I did 2 modifications
1- I added a folder cluster-local-gateway in instance/kustomize. And I created the file kustomization.yaml
kind: Kustomization
resources:
- ../../../upstream/manifests/istio/cluster-local-gateway/base
2- I added the lines below in hydrate-kcc in the Makefile
mkdir -p $(BUILD_DIR)/cluster-local-gateway
kustomize build --load_restrictor none -o $(BUILD_DIR)/cluster-local-gateway $(KF_DIR)/cluster-local-gateway
Here is the content of the directory .build/cluster-local-gateway after executing make hydrate
ls -lh
total 48K
-rw-r--r-- 1 jal jal 4,6K nov. 20 17:55 apps_v1_deployment_cluster-local-gateway.yaml
-rw-r--r-- 1 jal jal 453 nov. 20 17:55 autoscaling_v2beta1_horizontalpodautoscaler_cluster-local-gateway.yaml
-rw-r--r-- 1 jal jal 182 nov. 20 17:55 '~g_v1_configmap_cluster-local-gateway-parameters-tbbdb2842d.yaml'
-rw-r--r-- 1 jal jal 119 nov. 20 17:55 '~g_v1_namespace_istio-system.yaml'
-rw-r--r-- 1 jal jal 180 nov. 20 17:55 '~g_v1_serviceaccount_cluster-local-gateway-service-account.yaml'
-rw-r--r-- 1 jal jal 123 nov. 20 17:55 '~g_v1_serviceaccount_istio-multi.yaml'
-rw-r--r-- 1 jal jal 837 nov. 20 17:55 '~g_v1_service_cluster-local-gateway.yaml'
-rw-r--r-- 1 jal jal 384 nov. 20 17:55 policy_v1beta1_poddisruptionbudget_cluster-local-gateway.yaml
-rw-r--r-- 1 jal jal 422 nov. 20 17:55 rbac.authorization.k8s.io_v1_clusterrolebinding_cluster-local-gateway-istio-system.yaml
-rw-r--r-- 1 jal jal 358 nov. 20 17:55 rbac.authorization.k8s.io_v1_clusterrole_cluster-local-gateway-istio-system.yaml
-rw-r--r-- 1 jal jal 398 nov. 20 17:55 rbac.authorization.k8s.io_v1_clusterrole_istio-reader.yaml
I am not sure that this result is correct, because surprisingly when comparing this result with the content of the directory upstream/istio/cluster-local-gateway/base , I see different filenames. Below is the list of manifests in upstream/istio/cluster-local-gateway/base
ls -lh
total 48K
-rw-r--r-- 1 jal jal 375 nov. 20 08:16 cluster-role-binding.yaml
-rw-r--r-- 1 jal jal 640 nov. 20 08:16 cluster-role.yaml
-rw-r--r-- 1 jal jal 4,5K nov. 20 08:16 deployment.yaml
-rw-r--r-- 1 jal jal 406 nov. 20 08:16 horizontal-pod-autoscaler.yaml
-rw-r--r-- 1 jal jal 589 nov. 20 08:16 kustomization.yaml
-rw-r--r-- 1 jal jal 62 nov. 20 08:16 namespace.yaml
-rw-r--r-- 1 jal jal 23 nov. 20 08:16 params.env
-rw-r--r-- 1 jal jal 108 nov. 20 08:16 params.yaml
-rw-r--r-- 1 jal jal 288 nov. 20 08:16 pod-disruption-budget.yaml
-rw-r--r-- 1 jal jal 203 nov. 20 08:16 service-account.yaml
-rw-r--r-- 1 jal jal 743 nov. 20 08:16 service.yaml
Any idea why there is such a difference ?
Thanks.
jal06
on 20 Nov 2020
@jal06 kustomize build targetting a folder will generate those special names.
Don't worry, kubernetes resources doesn't matter what their file name is, the content should be good.
Regarding the make rule, hydrate-kcc is for config connector in management cluster. I'd suggest putting this in hydrate-kubeflow of kubeflow cluster (you should look at the kubeflow folder) because that's where we hydrate all kubeflow applications.
Bobgy
on 21 Nov 2020
@Bobgy, thanks for your feedbacks.
Regarding the make rule, I am OK; that's what I did in my Makefile but I was a little bit tired yesterday and I wrote hydrate-kcc instead of hydrate-kubeflow in my comment.
Happy to read that those filenames look OK
My next steps :
- I'll need to add a line
kubectl --context=$(KFCTXT) apply -f ./$(BUILD_DIR)/cluster-local-gatewayinapply-kubeflowrules - I will test. Probably I will deploy a new instance of Kubeflow in my dev platform with the new version 1.2 which was released yesterday (I don't want to disturb the team who is working with the current Kubeflow instance), and I will do the test with this new version 1.2
- if everything works successfully, I will create a PR in the repo gcp-blueprints
- if my tests are not successful, I will give you my feedback in this thread
I'll need a few days to test and create the PR.
Once again, thanks for your help, @Bobgy and @yuzisun
jal06
on 21 Nov 2020
@jal06 thank you for your efforts to fix this!
Note, GCP manifest for 1.2.0 hasn't been finalized, but should be soon. I'm updating documentation and pinning a release. See the 1.2.0 release tracker in GCP blueprint repo. If you want to test now, you can also install from master, it should be working.
Bobgy
on 23 Nov 2020
@Bobgy, I added the modifications above and I did some tests from the master branch
I see that the istio local gateway deployment is deployed correctly in istio-system.
(From my previous comments, I just needed to specify the namespace istio-system in the new kustomization.yaml in folder instance/kustomize)
If I test using the no-authz example, it works.
If I test using the with-auth example , I get an error
Exception: Service account [email protected] does not have permission to access the IAP-protected application.
I don't understand why I get this error.
- I'd like to use the serviceaccount which is automatically created during the deployment of Kubeflow in GCP ([email protected]).
According to this documentation, I must provide in the script the client_id and the secret.
When sending the request, I use the script make-prediction.sh.
In make_prediction.sh, there is a variable for setting the client_id, but there is no variable for the secret.
I probably missed something. Any help would be appreciated
- Below are the actions I did before getting this error
I gave access the serviceAccount access to IAP-protected application
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:${KF_NAME}[email protected] --role=roles/iap.httpsResourceAccessor
I generated a json key file for the serviceAccount user with this command :
gcloud iam service-accounts keys create ${KEY_FILE} \
--iam-account ${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com
I activated the service account
gcloud auth activate-service-account ${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com --key-file=${KEY_FILE}
I checked that my service account is active
gcloud auth list
GOOGLE_APPLICATION_CREDENTIALS is pointing to the key file of the serviceaccount
export GOOGLE_APPLICATION_CREDENTIALS=${KEY_FILE}
I also created a Clusterrole and a rolebinding (I am not sure it is relevant for production)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: inferenceservice-edit
rules:
- apiGroups:
- "*"
resources:
- inferenceservice
verbs:
- "*"
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: inferenceservice-edit
namespace: jal
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: inferenceservice-edit
subjects:
- kind: ServiceAccount
name: default-editor
namespace: jal
- kind: ServiceAccount
name: default
namespace: jal
- kind: ServiceAccount
name: xxxxx-user
namespace: jal
@Bobgy, @yuzisun
Let me know if you would like I open a PR including the modifications above.
Maybe it could make sense to close this issue and open a new one about the Istio Authorization issue.
WDYT ?
jal06
on 23 Nov 2020
@jal06 This is not the Istio authorization issue, I think you need to follow the Authenticating from a service account procedure https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account. @Bobgy probably knows better.
yuzisun
on 24 Nov 2020
@yuzisun. Thanks for the hint.
I followed the procedure Authenticating from a service account procedure
After selecting the GCP project, I can see that some resources are not configured
- there is an error for kube-system/default-http-backend
- there is a warning for Istio-system/istio-ingressgateway
The error displayed about the backend is : “The configuration of the Oauth client is wrong; for solving this problem,. you must reactivate the IAP service.”
I reactivated the IAP service for this resource, and the red error disappeared from the GUI, only a warning is remaining.
For the 2 resources (istio-ingressgateway and http-backend), I checked that the service account that I am using has the roles
- editor,
- IAP-secured Web app user
- viewer
There are some warnings in the container ingress-sds which were logged during the deployment of KF.
kubectl logs istio-ingressgateway-56fbb76cdd-8pjxz -c ingress-sds
2020-11-22T15:13:25.568306Z info sdsServiceLog SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds"
2020-11-22T15:13:25.568430Z info citadel agent monitor has started.
2020-11-22T15:13:25.568408Z info sdsServiceLog Start SDS grpc server for ingress gateway proxy
2020-11-22T15:13:25.568650Z info monitor Monitor server started.
2020-11-22T15:13:31.083681Z warn secretFetcherLog failed load server cert/key pair from secret dns.istio-galley-service-account: server cert or private key is empty
2020-11-22T15:13:32.091990Z warn secretFetcherLog failed load server cert/key pair from secret dns.istio-sidecar-injector-service-account: server cert or private key is empty
2020-11-22T15:15:37.815317Z warn secretFetcherLog failed load server cert/key pair from secret kubeflow-oauth: server cert or private key is empty
Because those warnings appear only when starting Kubeflow, I don’t know if we need to investigate more about them ?
I also checked from this page monitor-iap-setup .
I just notice that the 2 backends are healthy. However their port is 30954 and 32692 while according to the documentation, the Kubeflow traffic is handled by the service 31380, and there is no backend k8s-be-31380
Do you think that there is no port 31380 could cause some problem ?
@Bobgy, please let me know if you need more infos ?
Thanks
jal06
on 24 Nov 2020
@Bobgy is anyone owning the GCP integration for all the Kubeflow components like KFServing?
yuzisun
on 27 Nov 2020
I'm overseeing all the Kubeflow stuff on GCP.
@jal06 would you mind creating an issue in kubeflow/gcp-blueprints? I think it's more relevant there.
The reported issue about IAP looks strange, from first glance, everything looks correct.
I'd suggest separate the two issues for troubleshooting:
- connect to kfserving endpoint
- authenticate via IAP
To troubleshoot "IAP auth", recommend
- verify a chosen page/api request on KF UI works in the browser.
- follow https://cloud.google.com/iap/docs/authentication-howto and figure out how to connect via
curl
If anything doesn't work, you might also try other Google Cloud support channels just for the IAP problem
After you get "IAP auth" working, let's try to put all stages together.
Bobgy
on 30 Nov 2020
@Bobgy,
Thanks for the link.
FYI, I opened an issue for the connection to kfserving endpoint in kubeflow/gcp-blueprints, and I also opened a PR for adding the deployment of the local geteway in the namespace istio-system, which from my understanding, solves the issue.
I will also read accurately the documentation you gave and try the connexion via curl.
After that, I will create another issue about authentication via IAP in kubeflow/gcp-blueprints. Hopefully, it will work, and in this case, it should be useful to add some documentation in Kubeflow (maybe just pointing to this documentation); and if I still have problemns with authentication via IAP, I will describe them.
jal06
on 30 Nov 2020
@Bobgy, I created a new issue #https://github.com/kubeflow/gcp-blueprints/issues/178 about permission via IAP.
Thanks for your help
jal06
on 1 Dec 2020
@jal06 hi, while seeking for a solution for 503, found this and realized I dont have the cluster-local-gateway, either. I made the initial install with following link, edited auto injection to enabled (istio 1.7.1);
https://knative.dev/docs/install/installing-istio/#installing-istio-without-sidecar-injection
But even after trying istioctl install(default profile), I still cannot see a cluster-local-gateway, in istio-system. Is there an easy way (beginners grade) to add this gateway?
$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-767c5487d6-7n28z 1/1 Running 0 28h
istio-ingressgateway-859fdf8486-4lf6b 1/1 Running 4 3d9h
istiod-5c7446f47d-q8sps 1/1 Running 4 3d8h
kiali-89fd7f87b-hjspk 1/1 Running 0 2d9h
prometheus-788c945c9c-mb6jh 2/2 Running 8 3d8h
$ kubectl get gateway --all-namespaces
NAMESPACE NAME AGE
knative-serving cluster-local-gateway 7d1h
knative-serving knative-ingress-gateway 7d1h
knative-serving knative-local-gateway 7d1h
md5-d8556bc6209d2d4a66bd1e4296cfd79f
$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.106.145.6 <none> 3000/TCP 28h
istio-ingressgateway LoadBalancer 10.99.100.94 192.168.1.235 15021:31484/TCP,80:31804/TCP,443:32361/TCP,15443:31580/TCP 7d3h
istiod ClusterIP 10.108.127.128 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 7d3h
kiali ClusterIP 10.96.81.209 <none> 20001/TCP,9090/TCP 6d23h
knative-local-gateway ClusterIP 10.101.29.12 <none> 80/TCP 7d2h
prometheus ClusterIP 10.104.189.176 <none> 9090/TCP 6d23h
md5-d8556bc6209d2d4a66bd1e4296cfd79f
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bert-v2 ExternalName <none> cluster-local-gateway.istio-system.svc.cluster.local <none> 2d22h
bert-v2-predictor-default ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 2d22h
bert-v2-predictor-default-00001 ClusterIP 10.104.87.23 <none> 80/TCP 28m
bert-v2-predictor-default-00001-private ClusterIP 10.108.138.106 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP 28m
bert-v2-predictor-default-00002 ClusterIP 10.96.119.203 <none> 80/TCP 27m
bert-v2-predictor-default-00002-private ClusterIP 10.104.232.192 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP 27m
bert-v2-predictor-default-00003 ClusterIP 10.101.115.249 <none> 80/TCP 6h29m
bert-v2-predictor-default-00003-private ClusterIP 10.106.171.68 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP 6h29m
bert-v2-predictor-default-00004 ClusterIP 10.103.240.106 <none> 80/TCP 6h
bert-v2-predictor-default-00004-private ClusterIP 10.108.71.171 <none> 80/TCP,9090/TCP,9091/TCP,8022/TCP 6h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d4h
latest-bert-v2-predictor-default ExternalName <none> knative-local-gateway.istio-system.svc.cluster.local 80/TCP 2d22h
ontheway16
on 6 Jan 2021
@ontheway16, I did a PR for deploying the local gateway in GCP : #https://github.com/kubeflow/gcp-blueprints/pull/177
After applying this PR in GCP, the example without Istio injection sklearn-iap-no-authz.yaml works.
When adding authorization, the example with Istio injection sklearn-iap-with-authz.yaml doesn't work, there are still at least some authorization issues
If you work on GCP, you can have a look at this PR, there are only 3 lines to add in the Makefile (for platforms other than GCP, I don't know)
jal06
on 7 Jan 2021
@jal06, thank you for the reply. I am working fully local. Also, I am not installing kubeflow, just kfserving standalone.
So this basically means there are issues waiting to be fixed in kfserving setup, to make samples working, correct ?
ontheway16
on 7 Jan 2021
For GCP, there is at least the PR for the local gateway which must be merged.
For other platforms, I don't know. I saw a few other issues (for example https://github.com/kubeflow/website/pull/2409/files )
jal06
on 7 Jan 2021
one thing strange I noticed is that documentation in https://github.com/knative/docs/blob/master/docs/install/installing-istio.md#installing-istio-without-sidecar-injection is different from https://knative.dev/docs/install/installing-istio/#installing-istio-without-sidecar-injection.
There's an additional section in istio operator definition like
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
- name: cluster-local-gateway
enabled: true
label:
istio: cluster-local-gateway
app: cluster-local-gateway
k8s:
service:
type: ClusterIP
ports:
- port: 15020
name: status-port
- port: 80
targetPort: 8080
name: http2
- port: 443
targetPort: 8443
name: https
that seems like the idiomatic way to install cluster-local-gateway with istio
Bobgy
on 7 Jan 2021
Related issues
ontheway16
·
4Comments
iavinas
·
4Comments
jeffgriffith
·
9Comments
songm28
·
9Comments
ace-gabriel
·
3Comments