Test-infra: Image publishing jobs fail with cloud build quota errors

Created on 28 Aug 2019 · 7Comments · Source: kubernetes/test-infra

What happened:
TestGrid alerts for some of the image publishing jobs.

2019/08/21 17:20:58 Failed to push some images: [error running [gcloud builds submit --config kettle/cloudbuild.yaml --substitutions _GIT_TAG=v20190821-92d6d70 --project k8s-testimages --gcs-log-dir gs://k8s-testimages-scratch/logs --gcs-source-staging-dir gs://k8s-testimages-scratch/source gs://k8s-testimages-scratch/source/71e37026-b191-4853-8ebf-8b489fc2abd8.tgz]: exit status 1]

Please provide links to example occurrences, if any:

https://k8s-testgrid.appspot.com/sig-testing-images#kettle
https://k8s-testgrid.appspot.com/sig-testing-images#bootstrap
https://k8s-testgrid.appspot.com/sig-testing-images#velodrome
https://k8s-testgrid.appspot.com/sig-testing-images#image-builder

The errors for kubekins-e2e look unrelated and potentially resolved?
https://k8s-testgrid.appspot.com/sig-testing-images#kubekins-e2e

/assign @fejta @michelle192837 @Katharine

kinbug kinoncall-hotlist

Source

cjwagner

All 7 comments

Review https://github.com/kubernetes/test-infra/pull/14105?

fejta on 28 Aug 2019

👀2 🚀1

kettle:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-test-infra-push-kettle/1164222888736198657

Step #0: Configuring tzdataYour build timed out. Use the [--timeout=DURATION] flag to change the timeout threshold.
ERROR: (gcloud.builds.submit) build 1dc7e43e-1de3-4443-89ec-d7d3dff43e80 completed with status "TIMEOUT"

velodrome:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-test-infra-push-velodrome/1165017664762941440

Step #0: Creating shadow workspace and symlinking source into "./gopath/src/k8s.io/test-infra".
Step #0: Documentation at https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/go/README.md
Step #0: Running: go build -o velodrome/transform/transform ./velodrome/transform
Step #0: velodrome/transform/fetcher.go:25:2: cannot find package "github.com/golang/glog" in any of:
Step #0:    /usr/local/go/src/github.com/golang/glog (from $GOROOT)
Step #0:    /workspace/gopath/src/github.com/golang/glog (from $GOPATH)
Step #0: velodrome/transform/influx.go:26:2: cannot find package "github.com/influxdata/influxdb/client/v2" in any of:
Step #0:    /usr/local/go/src/github.com/influxdata/influxdb/client/v2 (from $GOROOT)
Step #0:    /workspace/gopath/src/github.com/influxdata/influxdb/client/v2 (from $GOPATH)
Step #0: velodrome/transform/fetcher.go:26:2: cannot find package "github.com/jinzhu/gorm" in any of:
Step #0:    /usr/local/go/src/github.com/jinzhu/gorm (from $GOROOT)
Step #0:    /workspace/gopath/src/github.com/jinzhu/gorm (from $GOPATH)
Step #0: velodrome/transform/influx.go:27:2: cannot find package "github.com/jinzhu/gorm/dialects/mysql" in any of:
Step #0:    /usr/local/go/src/github.com/jinzhu/gorm/dialects/mysql (from $GOROOT)
Step #0:    /workspace/gopath/src/github.com/jinzhu/gorm/dialects/mysql (from $GOPATH)
Step #0: velodrome/transform/influx.go:28:2: cannot find package "github.com/spf13/cobra" in any of:
Step #0:    /usr/local/go/src/github.com/spf13/cobra (from $GOROOT)
Step #0:    /workspace/gopath/src/github.com/spf13/cobra (from $GOPATH)

fejta on 28 Aug 2019

/unassign @michelle192837

fejta on 28 Aug 2019

/kind oncall-hotlist

Looks like we don't have enough cloudbuild quota?

https://k8s-testgrid.appspot.com/sig-testing-images#velodrome
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-test-infra-push-velodrome/1185703182316081152
https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/post-test-infra-push-velodrome/1185703182316081152/artifacts/
https://storage.googleapis.com/kubernetes-jenkins/logs/post-test-infra-push-velodrome/1185703182316081152/artifacts/transform.log

Created [https://cloudbuild.googleapis.com/v1/projects/k8s-testimages/builds/f146ba4a-6fe1-430b-a0de-1d5691f7d0f4].
Logs are available at [https://console.cloud.google.com/gcr/builds/f146ba4a-6fe1-430b-a0de-1d5691f7d0f4?project=597082399059].
ERROR: (gcloud.builds.submit) RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'cloudbuild.googleapis.com/get_requests' and limit 'GetRequestsPerMinutePerProject' of service 'cloudbuild.googleapis.com' for consumer 'project_number:684195865762'.
- '@type': type.googleapis.com/google.rpc.Help
  links:
  - description: Google developer console API key
    url: https://console.developers.google.com/project/684195865762/apiui/credential

fejta on 26 Oct 2019

Katharine I assume this has been fixed?

fejta on 5 Dec 2019

@fejta Are we possibly still hitting this?