Actions-runner-controller: Failed calling webhook...context deadline exceeded

[raj-saxena]

Hi,

I have been trying to get this working for some time now referring the guide in the README but it always fails at the last step.
I have tried both the approaches of authentication - Github App as well as PAT.

I tried installing Runners as well as RunnerDeployments but the step fails with similar errors

Here are the steps that I followed -

# Install cert-manager
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace actions-runner-system \
  --version v0.15.1 \
  --set installCRDs=true

# Create necessary resources in actions-runner-system namespace
kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml

# Create secret with github PAT
kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_token=<GH_PAT>

# Copied the Runner.yml file from the example and changed the repo name
kubectl apply -f Runner.yaml

# Error for Runner
Error from server (InternalError): error when creating "Runner.yaml": Internal error occurred: failed calling webhook "mutate.runner.actions.summerwind.dev": Post https://webhook-service.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=30s: context deadline exceeded

# Error for RunnerDeployment
Error from server (InternalError): error when creating "RunnerDeployment.yaml": Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": Post https://webhook-service.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=30s: context deadline exceeded

If it is relevant, I am on Google Kubernetes Engine and we run nginx-ingress for ingress management instead of default LoadBalancer. Do I need to add some firewall rule or ingress rule?

[mumoshu]

The communication between the k8s api and the addmission webhook should be in-cluster as we can see the in-cluster domain is used in https://webhook-service.actions-runner-system.svc:443, which is also our default setup.

So, my guess is that the admission webhook server isn't working in your env? Can you try running kubectl -n actions-runner-system get po,deploy,svc to see if everything's really ready and working?

[raj-saxena]

Thanks for the suggestions. I investigated further and did a fresh installation on a local minikube as well as a new cluster on GKE and they both worked flawlessly. Seems like something is wrong with my specific cluster. Will investigate deeper internally. Nothing to do here. Thanks again.

[raj-saxena]

For anyone coming here in future - this was an issue because we have a private cluster on GKE and needed to create a firewall rule that could connect master node to the webhook port as per this comment

The only thing different was that we had to enable port 9443 instead of 6443 because that is used for the webhook-service yaml config in the action-runner-controller.yaml.

Here's how we created the firewall

# 1) Retrieve the network tag automatically given to the worker nodes
# NOTE: this only works if you have only one cluster in your GCP project. You will have to manually inspect the result of this command to find the tag for the cluster you want to target
WORKER_NODES_TAG=$(gcloud compute instances list --format='text(tags.items[0])' --filter='metadata.kubelet-config:*' | grep tags | awk '{print $2}' | sort | uniq)

# 2) Take note of the VPC network in which you deployed your cluster
# NOTE this only works if you have only one network in which you deploy your clusters
NETWORK=$(gcloud compute instances list --format='text(networkInterfaces[0].network)' --filter='metadata.kubelet-config:*' | grep networks | awk -F'/' '{print $NF}' | sort | uniq)

SOURCE=$(gcloud container clusters describe <cluster-name> --region <region> | grep masterIpv4CidrBlock| cut -d ':' -f 2 | tr -d ' ')
gcloud compute firewall-rules create k8s-cert-manager --source-ranges $SOURCE --target-tags $WORKER_NODES_TAG  --allow TCP:9443 --network $NETWORK

[mumoshu]

@raj-saxena Thanks for sharing your insight! It must definitely be helpful for others.

@summerwind Would you mind if I added a short note for GCP users trying to run this controller, with a link to @raj-saxena's comment?

[summerwind]

Sorry for the late reply.
I think it's helpful information! Please include it in the README, thanks!