Terraform-provider-google: Service Usage Quota

[antoinecastex]

Following this
https://www.terraform.io/docs/providers/google/r/service_usage_consumer_quota_override.html

I need to modify BigQuery daily Query Quota per user

esource "google_service_usage_consumer_quota_override" "override" {
provider = google-beta
project = var.PROJECT_ID
service = "bigquery.googleapis.com"
metric = "bigquery.googleapis.com%2Fquota%2Fquery%2Fusage"
limit = "%2Fd%2Fproject"
override_value = "1048576"
force = true

Result is OK but nothing changed

Second test

resource "google_service_usage_consumer_quota_override" "override" {
provider = google-beta
project = var.PROJECT_ID
service = "bigquery.googleapis.com"
metric = "bigquery.googleapis.com/quota/query/usage"
limit = "d/project"
override_value = "1048576"
force = true

result :

The requested URL /v1beta1/projects/aca-sbx-z1234/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com/quota/query/usage/limits/d/project/consumerOverrides/?alt=json&force=true was not found on this server. That’s all we know.

Please help

[edwardmedia]

@antoinecastex The error seems clear what is wrong. Likely the settings for metric and limit are not correct. How did you decide there are the right ones.
Can you also post the debug logs for the Terraform script execution?

[antoinecastex]

Thanks @edwardmedia for reply

i'm pretty sure this is the right one because by API i'm doing like this (and that's work very well )
body = "projects/"+project_name+"/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com%2Fquota%2Fquery%2Fusage/limits/%2Fd%2Fproject"
request = SERVICE_QUOTA_USAGE_MANAGER.services().consumerQuotaMetrics().limits().consumerOverrides().create(parent=body, body={'overrideValue': quota}, force=True)
response = request.execute()

So my first test in the issue is the exact same parameters in TF mode, no Errors , but nothing changed on the project

My second test is try to modify a little bit and just go the error i coy paste on the issue , nothing more

[nozhang]

any update? I have same problem

[edwardmedia]

@antoinecastex it is OK when I tested using below code. I see the quota data updated in the GCP Console each time I apply the new config.

  # google_service_usage_consumer_quota_override.override will be updated in-place
  ~ resource "google_service_usage_consumer_quota_override" "override" {
        dimensions     = {}
        force          = false
        id             = "projects/tf-test-issue7194/services/servicemanagement.googleapis.com/consumerQuotaMetrics/servicemanagement.googleapis.com%2Fdefault_requests/li
mits/%2Fmin%2Fproject/consumerOverrides/Cg1RdW90YU92ZXJyaWRl"
        limit          = "%2Fmin%2Fproject"
        metric         = "servicemanagement.googleapis.com%2Fdefault_requests"
        name           = "Cg1RdW90YU92ZXJyaWRl"
      ~ override_value = "110" -> "115"
        project        = "tf-test-issue7194"
        service        = "servicemanagement.googleapis.com"
    }
Plan: 0 to add, 1 to change, 0 to destroy.

resource "google_service_usage_consumer_quota_override" "override" {
  provider       = google-beta
  project        = google_project.my_project.project_id
  service        = "servicemanagement.googleapis.com"
  metric         = "servicemanagement.googleapis.com%2Fdefault_requests"
  limit          = "%2Fmin%2Fproject"
  override_value = "115"
}

[antoinecastex]

@edwardmedia thanks, the differences are the metric and the limit, because i'm trying to do that for bigquery, not servicemanagement ...

How. to know the exact value ?

What's the name field ?

thanks

[edwardmedia]

Below is the test using your code

  # google_service_usage_consumer_quota_override.override2 will be updated in-place
  ~ resource "google_service_usage_consumer_quota_override" "override2" {
        dimensions     = {}
        force          = true
        id             = "projects/tf-test-issue7194/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com%2Fquota%2Fquery%2Fusage/limits/%2Fd%2Fproject/consumerOverrides/Cg1RdW90YU92ZXJyaWRl"
        limit          = "%2Fd%2Fproject"
        metric         = "bigquery.googleapis.com%2Fquota%2Fquery%2Fusage"
        name           = "Cg1RdW90YU92ZXJyaWRl"
      ~ override_value = "900000" -> "850000"
        project        = "tf-test-issue7194"
        service        = "bigquery.googleapis.com"
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Below are the API requests and responses. Every time I updated the overrideValue, I do see the new values in the responses, which I think the quota service has accepted the new value. But same as what you were experiencing, I don't see the new values showing up on GCP Console UI. I don't have the answer for that. This happens for bigqery quota, not above example (servicemanagement)

PATCH /v1beta1/projects/tf-test-issue7194/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com%2Fquota%2Fquery%2Fusage/limits/%2Fd%
2Fproject/consumerOverrides/Cg1RdW90YU92ZXJyaWRl?alt=json HTTP/1.1
{
 "overrideValue": "850000"
}


GET /v1beta1/projects/tf-test-issue7194/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com%2Fquota%2Fquery%2Fusage/limits/%2Fd%2F
project/consumerOverrides/?alt=json HTTP/1.1

---[ RESPONSE ]--------------------------------------
{
   "overrides": [
     {
       "name": "projects/560998363126/services/bigquery.googleapis.com/consumerQuotaMetrics/bigquery.googleapis.com%2Fquota%2Fquery%2Fusage/limits/%2Fd%2Fpro
ject/consumerOverrides/Cg1RdW90YU92ZXJyaWRl",
       "overrideValue": "850000"
     }
   ]
 }

[antoinecastex]

so your conclusion @edwardmedia is that it's working well, there is only a bug in the console ?

[antoinecastex]

This is really crazy @edwardmedia you right

'overrideValue': '1048576

Shame on the console , how can the dev team let something so bad like that in production ? crazy !

[edwardmedia]

@antoinecastex I will reach out other teams to see how much they can help us.

[edwardmedia]

@antoinecastex you can also verify what value was set in the Quota service by running Terraform import. In my case "overrideValue" is "850000"

[nozhang]

@antoinecastex question, how do you know the limit string like _limit = "%2Fd%2Fproject"_ ?

[antoinecastex]

@nozhang very good question !

it's impossible to use the REST Api to that so , we try to look at the query send through the browser when we do that in the UI , we copy paste the full query and have discovered this ....

crazy !

[edwardmedia]

@antoinecastex It is good now. I was looking at a different Quota. To retest it using above code, I see the updates on Quota limit under below Quota Let me know if this solves your problem

BigQuery API | Query usage per day

[antoinecastex]

@edwardmedia what code have you used ?

you say now you can see the change in the UI ?

[edwardmedia]

@antoinecastex I used below code to verify the limit on BigQuery API | Query usage per day

resource "google_service_usage_consumer_quota_override" "override2" {
  provider       = google-beta
  project        = google_project.my_project.project_id
  service = "bigquery.googleapis.com"
  metric = "bigquery.googleapis.com%2Fquota%2Fquery%2Fusage"
  limit = "%2Fd%2Fproject"
  override_value = "830000"  #<--- updating this
  force = true
}

[antoinecastex]

@edwardmedia yes this code is mine, it's working very well, the api return OK

but the web console is not seeing any modifications ...

Service: BigQuery API
Name: Requests per day
Quota Metric: bigquery.googleapis.com/unlimited_requests

Requests per day | Unlimited
-- | --

[edwardmedia]

@antoinecastex what do you see under Query usage per day?

Service: BigQuery API
Name: Query usage per day
Quota Metric: bigquery.googleapis.com/quota/query/usage

[antoinecastex]

@edwardmedia that's working perfectly

Limit:1,048,576 MiB (1 TiB)Current usage:
0 MiB
(0%)7 days peak:
0 MiB
(0%)

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!