Terraform-provider-google: Provider ignores project setting - uses remote state project

[kevinohara80]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

v0.11.8

Affected Resource(s)

All google_* resources

Terraform Configuration Files

variables.tf

variable "google_region" {
  description = "The GCP region for the project"
  default     = "us-central1"
}

main.tf

# local variables
locals {
  project = "prod-888888"
}

terraform {
  backend "gcs" {
    credentials = "~/.gcloud/terraform-key.json"
    project     = "devops-tools"
    bucket      = "mynewco-terraform-state"
  }
}

provider "google" {
  credentials = "${file("~/.gcloud/terraform-key.json")}"
  project     = "${local.project}"
  region      = "${var.google_region}"
  version     = "~> 1.19"
}

resource "google_project_services" "project_services" {
  project  = "${local.project}"
  services = ["pubsub.googleapis.com"]
}

resource "google_pubsub_topic" "mytopic" {
  project = "${local.project}"
  name    = "test-topic"
}

Debug Output

Debug Log

Expected Behavior

I have two GCP projects set up, one called devops-tools and another called prod-888888. I have the terraform backend configured to use the devops-tools project for remote state. When I configure a GCP provider to use the prod project, I expect all subsequent resources to be created in that project.

Actual Behavior

When the above terraform is applied, terraform is attempting to create the resources in the devops-tools' (project id 999999999999) project despite the provider being configured to utilize theprod-888888` project.

Steps to Reproduce

1. Create a two projects in Google Cloud called devtools and prod
2. Create a Service Account in the devtools project
3. Generate a key for the Service Account and place it at ~/.gcloud/terraform-key.json locally
4. Add the Service Account to the prod project through Google IAM
5. Create the terraform file above
6. Run terraform apply

Important Factoids

It's important to note that the Service Account Key is for a Service Account that originates from the devtools-tools project. However, this service account was also added to the prod-8888 project so the credentials are valid. It seems as if terraform is simply ignoring the project configuration from the provider and also the resources.

[Chupaka]

Isn't you problem that your credentials are from devtools project, and that project is used for API calls?

Like this one: https://github.com/terraform-providers/terraform-provider-google/issues/2194

[kevinohara80]

@Chupaka Those credentials were generated from the Service Account in the devtools project, but that same service account was added to the prod-888888 project. Therefore the API calls to the prod project will work.

Another thing I noticed is that the issue only seems to be with resource creation. Adding a data source returns the correct information about the configured project:

locals {
  project = "prod-888888"
}

terraform {
  backend "gcs" {
    credentials = "~/.gcloud/terraform-key.json"
    project     = "devtools-219114"
    bucket      = "mynewco-terraform-state"
  }
}

provider "google" {
  credentials = "${file("~/.gcloud/terraform-key.json")}"
  project     = "${local.project}"
  region      = "${var.google_region}"
  version     = "~> 1.19"
}


data "google_project" "project" {}

output "project_name" {
  value = "${data.google_project.project.name}"
}

output "project_id" {
  value = "${data.google_project.project.id}"
}

produces the following output...

$ terraform apply -auto-approve
data.google_project.project: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

project_id = prod-888888
project_name = prod

[emilymye]

I'm having some trouble recreating this issue - using essentially different values for the config you gave, I am able to generate resources under my local.project.

I'm also a little confused - your debug logs show PUT "/v1/projects/prod-888888/topics/test-topic", which is your local.project.

[kevinohara80]

@emilymye Yeah, I saw that too but the operation does fail and the fail message spits out the project id of the devtools project. I just ran it again and and verified the project numbers.

It feels like something is defaulting to the originating project for the service account key. Perhaps it's just in the error logging since we are seeing a PUT to the correct project but that doesn't explain why it's failing.

[emilymye]

Oh weird - actually, your debug logs are saying the Cloud Pub/Sub API hasn't been enabled for 999999999999 ( I assume you redacted this ID). If this was a different error than the one you saw, could you send me the logs from that error? As for this one, could you add a depends_on = [ "google_project_services. project_services" ] to your pubsub topic for me?

[emilymye]

And as a final comment, you should probably be using google_project_service instead of google_project_services (note plural) - google_project_services will actually set the exact set of enabled APIs, which means that all other APIs will be disabled.

[kevinohara80]

@emilymye Sorry for the delay. Yes, I redacted the ID. The error you mentioned is the same one I saw. It's trying to create resources including the google_project_services in the devtools project instead of the prod-8888 project which is specified in the provider (and the resources themselves).

Thanks for the suggestion on using google_project_service resources instead. I'll use that going forward.

[kevinohara80]

@emilymye We hit this problem again today on a fresh project set up using this guide.

Interestingly, this issue seems to be isolated to creating google_pubsub_topic resources only. In my terraform file, I'm able to create other resources like service accounts with no problem. I'm testing other resources right now but so far, the pubsub topics are the only issue.

[kevinohara80]

Doing some browsing, it looks like it could be related to #2346 and #2469. The first time I tried to create the resource I neglected to add the project Id. After that error, I added the project Id and that's when the resource began using the other Google Project.

[sergei-ivanov]

@kevinohara80 You may have run into the same problem that I described here. Try enabling pubsub.googleapis.com service in the devops-tools project and see if it solves the issue.

[kevinohara80]

@sergei-ivanov Thanks for the link. They definitely look related. Unfortunately we had to move our project to AWS because we've encountered so many API issues.

[danawillow]

Hi @kevinohara80, I'm sorry to hear that! I'm going to go ahead and close this issue because it doesn't seem like there's anything to do here. I've talked to teams at Google about getting better docs about which project APIs have to be enabled on, but in the meantime, it's the one that the service account comes from, rather than the one that the resource is being created in. I hope that you're able to get things working with GCP, and if you have any other questions please keep filing issues or join us in the terraform channel in the GCP slack!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!