Cert-manager: !!! cert-manager cannot access GoogleCloud API

[munjal-patel]

Hello,

I am installing cert-manager using this script:

#!/bin/sh

CURRENT_PROJECT_ID=$(gcloud config get-value core/project)
read -p "Provide GCP Project ID for cluster [$CURRENT_PROJECT_ID]: "  PROJECT_ID
if [ -z "$PROJECT_ID" ]; then
    PROJECT_ID="$CURRENT_PROJECT_ID"
fi

kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true --overwrite=true
kubectl label namespace cert-manager istio-injection=disabled --overwrite=true

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=$(gcloud config get-value core/account)

CLOUD_DNS_SA=cert-manager-cloud-dns-admin
gcloud --project $PROJECT_ID iam service-accounts delete $CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com
gcloud --project $PROJECT_ID iam service-accounts create $CLOUD_DNS_SA \
    --display-name "Service Account to support ACME DNS-01 challenge."

CLOUD_DNS_SA=$CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member serviceAccount:$CLOUD_DNS_SA \
     --role roles/dns.admin

KEY_DIRECTORY=`mktemp -d`
gcloud iam service-accounts keys create $KEY_DIRECTORY/cloud-dns-key.json \
     --iam-account=$CLOUD_DNS_SA
kubectl delete secret --namespace cert-manager cloud-dns-key
kubectl create secret --namespace cert-manager generic cloud-dns-key \
     --from-file=key.json=$KEY_DIRECTORY/cloud-dns-key.json
rm -rf $KEY_DIRECTORY

CERT_MANAGER_VERSION=0.6.1
DOWNLOAD_URL=https://github.com/jetstack/cert-manager/archive/v${CERT_MANAGER_VERSION}.tar.gz

wget $DOWNLOAD_URL
tar xzf v${CERT_MANAGER_VERSION}.tar.gz

cd cert-manager-${CERT_MANAGER_VERSION}

kubectl apply -f deploy/manifests/00-crds.yaml
kubectl apply -f deploy/manifests/cert-manager.yaml

cd ../
rm -rf cert-manager-${CERT_MANAGER_VERSION}
rm v${CERT_MANAGER_VERSION}.tar.gz

However, when it tries to validate DNS-01 challenge, it keeps failing with:

challenges controller: Re-queuing item "istio-system/nebula-certificate-1671088635-1" 
due to error processing: GoogleCloud API call failed: googleapi: 
Error 403: Forbidden, forbidden

I have spent almost 2 days trying various different settings / combinations / versions / re-installs .. but have not been able to resolve this issue.

Would really appreciate any help I can get here.

[munjal-patel]

Duplicate of #1532

[munjal-patel]

Is there anything we can do here? This is really hurting our _adoption_ of cert-manager and _go-live._

[munjal-patel]

After cloning https://github.com/jetstack/cert-manager and tracing through the problem manually, I have figured it out.

Turns out Google Cloud does not like re-use of the sa email.

In my script above, I am re-using the same sa email every time I setup the cluster by deleting the old one and creating the new one.

CLOUD_DNS_SA=cert-manager-cloud-dns-admin
gcloud --project $PROJECT_ID iam service-accounts delete $CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com
gcloud --project $PROJECT_ID iam service-accounts create $CLOUD_DNS_SA \
    --display-name "Service Account to support ACME DNS-01 challenge."

CLOUD_DNS_SA=$CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member serviceAccount:$CLOUD_DNS_SA \
     --role roles/dns.admin

Instead of that, I am doing this now to ensure a new unique sa email is used everytime. Timestamp helps me figure out the which one is the older one that should be removed manually.

CLOUD_DNS_SA=certmgr-cdns-admin-$(date +%s)
gcloud --project $PROJECT_ID iam service-accounts create $CLOUD_DNS_SA \
    --display-name "Service Account to support ACME DNS-01 challenge."

CLOUD_DNS_SA=$CLOUD_DNS_SA@$PROJECT_ID.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member serviceAccount:$CLOUD_DNS_SA \
     --role roles/dns.admin