Pipelines: issue of Dataproc component in a cluster created not on GCE VM default service account

[rmgogogo]

One case reports issue that

1. cluster is created on a different service account (not GCE VM default account)
2. the dataproc component didn't use the KFP pipeline-runner service account and also not the service account used to create the cluster. It is using the GCE VM default account.

https://github.com/kubeflow/pipelines/blob/master/components/gcp/container/component_sdk/python/kfp_component/google/dataproc/_client.py

We need to make sure the component (pipeline step) is using the certain KSA (e.x. pipeline-runner).

[Bobgy]

If the client doesn't use workload identity bound GSA, that's a bug in gcp python client, we should report it there.

[rmgogogo]

Based on the email, it seems reporter didn't well express the problem. Waiting for replies but keep this open for a while here to see if others also hit some issue.

[mmwebster]

I'm running into an issue with a demo pipeline that seems like it could be related to this. When running the XGBoost demo pipeline as instructed in the KF Pipelines Quickstart guide, the dataproc-create-cluster block fails with

File "kfp_component/google/dataproc/_create_cluster.py", line 74, in create_cluster 
wait_interval)
File "kfp_component/google/dataproc/_create_cluster.py", line 89, in _create_cluster_internal
return _dump_cluster(operation.get('response'))
File "kfp_component/google/dataproc/_create_cluster.py", line 109, in _dump_cluster
cluster.get('clusterName'))
AttributeError: 'NoneType' object has no attribute 'get'

In the GCP console, my Kubernetes node pools are listed as having the {KF_PROJECT}-vm service account, rather than the default GCE service account. In the IAM section, it also appears that this VM account doesn't have sufficient permissions to modify resources in the cluster (only has Logs Writer, Monitoring Metric Writer, Monitoring Viewer, Storage Object Viewer).

Should I post a new issue or does this seem related this issue?

[Bobgy]

@mmwebster Can you share your kubeflow version?
After 1.0.2, pipelines should already have permissions using workload identity, not default service account.

If not, suggest binding workload identity for pipeline-runner service account

[Bobgy]

/close
because the original report has been resolved, it was because of misconfigured workload identity binding.

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

/close
because the original report has been resolved, it was because of misconfigured workload identity binding.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rmgogogo]

/close
because the original report has been resolved, it was because of misconfigured workload identity binding.

I'm wondering whether we publish the WI self test to github or website as a FAQ. I mean the tool to check the WI binding. It seems a FAQ. Will put one to post-1.0 backlog.

[Bobgy]

that's a good point, we should mention it in workload identity doc in troubleshooting section