Pipelines: Cloud Function with Kubeflow Pipelines

[ultrons]

What happened:
Created a kubeflow deployment on a GKE using inverse proxy.
Tried to submit a pipeline execution to this deployment from cloud function.
-> client=kfp.client(host=)
client.list_experiments() returns nothing.
client.create_experiment(...) returns experiment object with all none field.
No exception seen.

What did you expect to happen:
Connect to the deployment and create/list experiment upload a pipeline and run from cloud function.

What steps did you take:
Tried to look for exceptions to understand the error.
none seen.

[A clear and concise description of what the bug is.]
The kfp SDK requires gcloud to be installed for retrieving the GSA token.
https://github.com/kubeflow/pipelines/blob/9ad7d7dd9776ce75a83712f5723db2ef93ba5c26/sdk/python/kfp/_auth.py#L38
There is no exception handling.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Using gcloud python sdk to retrieve this token could be more robust for cloud function like environment.

[parthmishra]

I believe #2896 should address the silently failing part

[rmgogogo]

"with os.popen('gcloud auth print-access-token') as token:"
We need to switch it to Python based codes for Cloud Function.

[rmgogogo]

@Ark-kun

[Bobgy]

@rmgogogo shall we assign a priority too?

[numerology]

"with os.popen('gcloud auth print-access-token') as token:"
We need to switch it to Python based codes for Cloud Function.

@rmgogogo Just to confirm does it mean we cannot use subprocess or os.popen in cloud function?

[Bobgy]

@numerology yes, because cloud function is a sandboxed environment that doesn't allow you to use your own image.
also each function call will be independent too.

[rmgogogo]

We may try to use gcloud python library and then call it like following (didn't verify yet). just leave memo here.

import google.auth
import google.auth.transport.requests
creds, projects = google.auth.default()

# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those

auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)

[ultrons]

@rmgogogo , I tried it from my local machine, It works but I get the following warning:
/opt/miniconda3/envs/py36/lib/python3.6/site-packages/google/auth/_default.py:69: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)

I have not tried it from GCE VM or CAIP notebook.

Do you think it's an acceptable solution or we need to investigate getting credential from service account instead of user level credential.

[ultrons]

Also, I just made a minor edit to your snippet. I set the scope argument for auth.default() to ['googleapis.com/auth/cloud-platform'] ref: https://stackoverflow.com/questions/53472429/how-to-get-a-gcp-bearer-token-programmatically-with-python.

[rmgogogo]

Thanks, let me put it to a GCP-adoption related queue for well track.
Feel free to ping me. I do want to allocate some time slot to move this forward.

[rmgogogo]

TODO: check if we don't specify the credential, whether Cloud Function can use the Service Account automatically (in GCE/GKE, it will, but I'm not sure whether CF also will).

If it won't, then check with CF team to see if we have other options.

[ultrons]

@rmgogogo I will check this and update here shortly.

[ultrons]

@rmgogogo , I saw that you have already verified it. Please let me know if I can help with anything.

[rmgogogo]

It should can work after #3363 got released. Feel free to ping if find any issue.

[amorenocb]

Hi @rmgogogo! Any idea when #3363 will be released? Thank you.

[rmgogogo]

SDK is already released yesterday in 0.4.0

https://pypi.org/project/kfp/