Flux: GCR Credentials and different namespaces

[emas80]

We are running Flux inside of its own namespace, and the idea is for Flux to manage the workloads in other namespaces.

The docker images are hosted on GCR, and I'm installing Flux by using the Helm chart, using the version 1.12.0 of the Chart.

I've been doing some testing, and I've read some documentation, in particular the doc about the troubleshooting (https://docs.fluxcd.io/en/latest/troubleshooting.html#flux-doesn-t-seem-to-be-able-to-use-my-imagepullsecrets) and the issue #1485.

I've seen that Flux is unable to list the docker image tags unless we create one secret of type docker-registry and we associate it to the default service account of the namespace.
I've tried to associate the docker registry secret to the service account used by Flux, or even to specify the

dockercfg:
  enabled: true
  secretName: "my-docker-registry"
  configFileName: /dockercfg/config.json

inside of the Helm values.yaml, but Flux is unable to list the docker images as it does not find any registry credentials

ts=2019-08-21T11:47:47.212984069Z caller=warming.go:180 component=warmer canonical_name=gcr.io/GCP_PROJECT/my-api auth={map[]} err="requesting tags: denied: Failed to read tags for host 'gcr.io', repository '/v2/GCP_PROJECT/my-api/tags/list'"

If I assume this is the expected behaviour, I would like to update the documentation that in my opinion is not so clear on this point in the section

Why are my images not showing up in the list of images?

If instead this is not the expected behavior, and I should be able to provide Flux with the credentials for accessing to GCR once and for all, what should be the right way to configure it?

For our use case, the namespaces should be dynamically generated - they can be created or deleted, and Flux should be able to manage the workloads without any human intervention, adding or removing them.
It would be hard for us to add the imagePullSecret to every namespace.

Our cluster is running on GKE and we are using the workload identity (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
I have tried adding the permissions to use GCR (basically, the storageAdmin permission) to the default service account by using the workload identity annotation on the service account, but it does not work, it really wants the imagePullSecret.

[mgenov]

I'm experiencing the same issue on my side too. Flux was working as expected until I've upgraded the cluster to use the workload-identity.

[vic3lord]

For me:
Flux works perfectly fine pulling new images if the registry belongs to the same project but the other flux I have on other project, cannot access to the main project's gcr.

I am able to pull images correctly from different projects using Storage Object Viewer role as documented to allow cross project pulling

[marratj]

For me:
Flux works perfectly fine pulling new images if the registry belongs to the same project but the other flux I have on other project, cannot access to the main project's gcr.

I am able to pull images correctly from different projects using Storage Object Viewer role as documented to allow cross project pulling

I have the same issue.

GKE clusters with Workload Identity enabled. Flux can read images from the GCR registry in the same project, but not from other projects, despite the nodes are able to pull the images just fine with their GCP node service account.

Even when explicitly assigning the GCP service account to Flux's Kubernetes Service Account, it's not able to read the images.

[marratj]

Update to my comment, the following works:

• configure an explicit GCP Service Account and assign the iam.workloadIdentityUser role to that with correct Identity Namespace config in the project where Flux is running on GKE
• set the iam.gke.io/gcp-service-account annotation in to that in the Flux K8s SA.
• grant storage.objectViewer permissions to that GCP SA on the underlying GCR bucket in the other project

With that, Flux is able to see the images from the registry in the other project.

So you need to explicitly configure Workload Identity and Bucket permissions for the Service Account under that Flux is running.

[hiddeco]

@marratj this would actually be a good addition to our documentation, do you think you can put this into a PR?

[kozejonaz]

Update to my comment, the following works:

* configure an explicit GCP Service Account and assign the `iam.workloadIdentityUser` role to that with correct Identity Namespace config in the project where Flux is running on GKE

* set the `iam.gke.io/gcp-service-account` annotation in to that in the Flux K8s SA.

* grant `storage.objectViewer` permissions to that GCP SA on the underlying GCR bucket in the other project

With that, Flux is able to see the images from the registry in the other project.

So you need to explicitly configure Workload Identity and Bucket permissions for the Service Account under that Flux is running.

Thanks @marratj ! We were just hit with this after activating workload identity for our clusters and this worked nicely.

[marratj]

@marratj this would actually be a good addition to our documentation, do you think you can put this into a PR?

@hiddeco sorry for the late reply, I lost track of the issue and just stumbled upon it again right now. If still wanted, I surely can create a PR to include this into the docs. In which section should I put it? Guides? Troubleshooting?