Test-infra: Router quota exceeded error causing GCE tests to fail

Created on 4 Oct 2019 · 19Comments · Source: kubernetes/test-infra

What happened:
W1004 01:31:15.149] Creating router [e2e-51036-95a39-nat-router]...
W1004 01:31:18.991] ....................failed.
W1004 01:31:19.173] ERROR: (gcloud.compute.routers.create) Quota 'ROUTERS' exceeded. Limit: 10.0 globally.

Please provide links to example occurrences, if any:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/51036/pull-kubernetes-e2e-gce-100-performance/1179926617770692608/
https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/51036/pull-kubernetes-kubemark-e2e-gce-big/1179926617825218560/

Anything else we need to know?:
Potential boskos cleaning issue

areboskos kinbug kinfailing-test kinoncall-hotlist

Source

cblecker

All 19 comments

from boskos janitor logs:

jsonPayload: {
  error: "exit status 1"   
  level: "error"   
  msg: "failed to clean up project k8s-jkns-e2e-gke-ci-canary, error info: Activated service account credentials for: [[email protected]]
ERROR: (gcloud.compute.disks.delete) unrecognized arguments: --global 

To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS
Error try to delete resources disks: CalledProcessError()
[=== Start Janitor on project 'k8s-jkns-e2e-gke-ci-canary' ===]
[=== Activating service_account /etc/service-account/service-account.json ===]
[=== Finish Janitor on project 'k8s-jkns-e2e-gke-ci-canary' with status 1 ===]
"   
 }

BenTheElder on 4 Oct 2019

cc @krzyzacy

BenTheElder on 4 Oct 2019

it looks like the image was last updated a month ago https://github.com/kubernetes/test-infra/commit/0fd634d70423b71b85bba2bc0687c0fbb732c31e

gcloud compute disks delete --help
NAME
    gcloud compute disks delete - delete Google Compute Engine persistent disks

SYNOPSIS
    gcloud compute disks delete DISK_NAME [DISK_NAME ...] [--zone=ZONE]
        [GCLOUD_WIDE_FLAG ...]

DESCRIPTION
    gcloud compute disks delete deletes one or more Google Compute Engine
    persistent disks. Disks can be deleted only if they are not being used by
    any virtual machine instances.

POSITIONAL ARGUMENTS
     DISK_NAME [DISK_NAME ...]
        Names of the disks to delete.

FLAGS
     --zone=ZONE
        Zone of the disks to delete. If not specified and the compute/zone
        property isn't set, you may be prompted to select a zone.

        To avoid prompting when this flag is omitted, you can set the
        compute/zone property:

            $ gcloud config set compute/zone ZONE

# gcloud compute disks delete --help | tail
    --flags-file, --flatten, --format, --help, --log-http, --project, --quiet,
    --trace-token, --user-output-enabled, --verbosity. Run $ gcloud help for
    details.

NOTES
    These variants are also available:

        $ gcloud alpha compute disks delete
        $ gcloud beta compute disks delete

# gcloud compute disks delete --global
ERROR: (gcloud.compute.disks.delete) unrecognized arguments: --global 

To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS

BenTheElder on 4 Oct 2019

I can't tell what actually broke when yet here. AFAICT we're running an image from august since then and haven't been having issues, also, the previous image has the same missing --global flag ...

BenTheElder on 4 Oct 2019

https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=hack%2Fe2e-internal%2Fe2e-up.sh#ea6679417165f10786e6

cblecker on 4 Oct 2019

$ kubectl get po -n=test-pods -l=app=boskos-janitor-nongke
NAME                                     READY   STATUS    RESTARTS   AGE
boskos-janitor-nongke-7c78646b5d-8rwjm   1/1     Running   0          6d6h
boskos-janitor-nongke-7c78646b5d-wm2fn   1/1     Running   0          6d9h
boskos-janitor-nongke-7c78646b5d-xj5sl   1/1     Running   0          6d8h
boskos-janitor-nongke-7c78646b5d-xl2bw   1/1     Running   0          6d3h

BenTheElder on 4 Oct 2019

I execed to the pods and unsurprisingly they do seem to be running the janitor script from when the image was updated, so I don't think there were any terribly recent changes actually deployed.

BenTheElder on 4 Oct 2019

@krzyzacy feel free to punt this back, but I don't feel that I have the context on what happened here.
@dims can you fill us in on the router issue with cluster-api-provider-gcp?

BenTheElder on 4 Oct 2019

we are obviously not cleaning up routers in https://github.com/kubernetes/test-infra/blob/master/boskos/janitor/gcp_janitor.py#L35-L67

also seems gcloud deprecated some flags (that --global one), but should be unrelated.

krzyzacy on 4 Oct 2019

👀1

Thanks @krzyzacy Sen!

@BenTheElder the new CAPG job uses boskos to acquire a project to create the actual cluster (uses kind to boostrap and then gcp to run the actual cluster) seems to have ended up with some problems. I do try to clean that up here, but some runs may have run into trouble and ended up leaking.
https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/master/hack/ci/e2e-conformance.sh#L101-L105