Pipelines: Access output artifacts stored in a bucket outside of the UI container project

[sngahane]

When components need specific GCP credentials, we use .apply(use_gcp_secret(…)) to add secret volume mounts to a task container. This allows you, for example, to get access to a GCS bucket that is not part of the project.

When the UI pod is created, it is automatically assigned a default service account (<cluster-id>-vm@<gcp-project>.iam.gserviceaccount.com). An issue arises when the UI tries to load the output artifacts stored in a bucket it doesn’t have access to.

As a temporary solution, our engineers grant GCS read permissions to the UI service account to view their GCS bucket objects. But we are wondering if there could be a way to use the proper secrets when the call to GCS is made in the UI backend.

Is this something others have thought about, especially in a multi-user context?

[Ark-kun]

/cc @eterna2 This might have already been fixed by your recent work.

[eterna2]

Not for GCP. I only did it for minio and AWS S3.

But I am going to enhance the artifact endpoint to accept k8s secrets params in the next PR, to support more flexible artifact downloads.

Never planned to do for GCP thou. But I can do that too. Just need abit of time to get familiar with gcp sdk.

[pavan-infiswift]

I am facing this same issue. I need the UI to access artifacts from s3. How do I pass secrets to the Pipeline UI? Any update on this?

[eterna2]

@pavan-infiswift

The latest version of kfp UI now supports IAM role for ur pod/node.

If u have kube2iam or equivalent deployed in ur cluster, u just need to make sure ur UI pod has the IAM role annotation.

Otherwise, u will need to give ur ec2 node the IAM role.

Or if you are using API key, u can pass them to the UI pod via environment variables.

see
https://github.com/kubeflow/pipelines/blob/0f0bb1d7e158e16a92440fad096fd776e6db01a4/frontend/server/server.ts#L45

[Bobgy]

For changing secrets, it's the same as use_gcp_secret. Just edit your deployment and mount your provided GCP credentials and set GOOGLE_APPLICATION_CREDENTIALS env.
Or if you use workload identity, it would be easier.

However, multi user support in the UI is something we haven't looked into. Help welcomed in driving a design for this.

[Bobgy]

I moved the multi user related discussion to https://github.com/kubeflow/pipelines/issues/2899 because it's not very obvious in this title.

/close

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

I moved the multi user related discussion to https://github.com/kubeflow/pipelines/issues/2899 because it's not very obvious in this title.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Shasvat0601]

Or if you are using API key, u can pass them to the UI pod via environment variables.
Can you explain in a bit more detail about how should we do this?
@eterna2