Terraform-provider-google: data.google_project.id breaks configs in 3.0.0-beta.1

[universalvishwa]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.14

Affected Resource(s)

Every resource that needs to specify the project ID reference. This effects both google and google-beta provider both.

• google_....

Terraform Configuration Files

data "google_project" "project" {
  project_id = "my-project-7jewd"
}

resource "google_compute_global_forwarding_rule" "ipv4-primary" {
  name        = "primary"
  ip_address  = "${data.google_compute_global_address.ipv4.address}"
  ip_protocol = "TCP"

  port_range = "80"
  target     = "${google_compute_target_tcp_proxy.default.self_link}"
  project    = "${data.google_project.project.id}"
}
````
The above is one of the effected resources. Anywhere I refer the `project = "${data.google_project.project.id}"` it fails.

So, instead I had to update it to `project = "${data.google_project.project.project_id}"` to get it to work. Also, Terraform no longer identifies old resources from the existing state after I did this change, it starts to recreate everything and even fail at some points. May be because I had to do that update I mentioned to those depending resources. This causes a chain re-action accross the entire deployment.

The main problem here is in the older 2.x.x provider versions the output of 
`${data.google_project.project.id}` is something like 'my-project-7jewd`. But now the resulting output is something like 'projects/my-project-7jewd'.

I think this is a clear bug that needs to be addressed asap. and make sure the output of `${data.google_project.project.id}` remains consistent with the earlier versions.


### Debug Output

Initializing provider plugins...

• Checking for available provider plugins on https://releases.hashicorp.com...
• Downloading plugin for provider "google" (3.0.0-beta.1)...
• Downloading plugin for provider "google-beta" (3.0.0-beta.1)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

• provider.google: version = "~> 3.0"
• provider.google-beta: version = "~> 3.0"

In the above init output you can clearly see that providers are sourcing a beta release of the providers. This was released earlier today.

<!---
Please provide a link to a GitHub Gist containing the complete debug output. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

To obtain the debug output, see the [Terraform documentation on debugging](https://www.terraform.io/docs/internals/debugging.html).
--->

### Panic Output

Causes this type of output...
```Error: Error applying plan:

1 error occurred:
        * google_compute_target_tcp_proxy.default: 1 error occurred:
        * google_compute_target_tcp_proxy.default: Error waiting to create TargetTcpProxy: Error waiting for Creating TargetTcpProxy: error while retrieving operation: googleapi: Error 400: Invalid value 'projects/my-project-7jewd'. Values must match the following regular expression: '(?:(?:[-a-z0-9]{1,63}\.)*(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?):)?(?:[0-9]{1,19}|(?:[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?))', invalidParameter

Expected Behavior

For new 3.x.x provider upgrades the data source google_project, attribute id should be the project ID instead of having a prefix projects/ along with the project ID.

data "google_project" "project" {
  project_id = "my-project-7jewd"
}

output "gcp_project_id" {
  value = "${data.google_project.project.id}"
}

Outputs:

gcp_project_id = my-project-7jewd

Actual Behavior

data "google_project" "project" {
  project_id = "my-project-7jewd"
}

output "gcp_project_id" {
  value = "${data.google_project.project.id}"
}

Outputs:

gcp_project_id = projects/my-project-7jewd

Steps to Reproduce

1. terraform apply

[paddycarver]

hey @universalvishwa, sorry for the issues here. The bug you noticed will probably still affect 0.12 users, but I notice you're on Terraform v0.11.14. 3.0.0-beta.1 went out to Terraform 0.11 users on accident today as a release mishap; it should be cleared up now. If you haven't applied yet, you should be able to remove your downloaded copy of the provider and run init again, and get 2.20.0.

Sorry for the problem.

[universalvishwa]

@paddycarver ,

When I run re-init it still shows like this,

Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "google" (3.0.0-beta.1)...
- Downloading plugin for provider "google-beta" (2.20.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 3.0"
* provider.google-beta: version = "~> 2.20"```

The google provider is 3.0.0-beta.1. I assume both google and google-beta both should be 2.20, Right?

[paddycarver]

That's correct. Can I ask what OS and architecture you're using?

[universalvishwa]

My local dev is done on Mac OS X, Our actual build on CentOS linux.

[paddycarver]

And both are giving you 3.0.0-beta.1 with Terraform 0.11? Are they amd64 machines?

[paddycarver]

We've just discovered a caching issue, where the cache wasn't updated like we thought it was, which was leading some users to get the uncorrected binary. Please let me know if you continue to have problems after this point.

[danawillow]

I'm going to un-waiting-response this, since the actual bug in question seems pretty clear.

The actual bugfix here is either:

1. add DSFs on all project fields and allow both forms in getProject
2. have the data source (and probably the project resource) return the old id format

My preference is option 1, but I could be convinced otherwise. @rileykarson, feel free to tell me to shut up if I'm stepping on your toes here.

[paddycarver]

Oh, sorry, I hijacked the issue. My bad. I had forgotten about the actual bug we were talking about.

[rileykarson]

The impact of changing google_project.id is large enough that preserving the old value makes sense to me. It's a little sad that the format won't be consistent, but it's a tremendously dangerous change otherwise.

I'm low availability for a bit, so handing over the issue to you @danawillow.

[danawillow]

Handing over to @slevenick because IDs, feel free to just assign it back to me if you need an extra pair of hands though.

[slevenick]

Yeah, I'm tempted to move the id on this one back to just the project-id rather than projects/project-id.

Changing the config from project = "${data.google_project.project.id}" => project = "${data.google_project.project.project_id}" will fix the problem, but I would expect a large number of people to be using this, as it was used in several of the tests in the past.

[slevenick]

@universalvishwa I'm concerned about the report of resources being recreated due to this. I'm unable to reproduce locally using a sample config with several resources including google_compute_global_forwarding_rule though.

When upgrading providers I do see the diff:

      ~ project = "my-project-id" -> "projects/my-project-id" # forces replacement

but changing from project.id to project.project_id causes a terraform plan to show no diff. Are you able to provide debug logs for this issue after changing all instances of project.id -> project.project_id?

[dedamico]

@slevenick this type of issue (confusion related to project_id) appears to happen for other resources as well. A customer reported to us (HC support) that when using the google_dns_managed_zone this no longer works in 3.0.0-beta.1.

I was able to recreate by doing the following steps:

Terraform version: 0.12.10

1. Create a Cloud DNS zone in Cloud console (name = daved-test-zone)
2. run terraform plan on the following code with GCP provider pinned to 2.20.0:

data "google_dns_managed_zone" "delegation-zone" {
name = "daved-test-zone"
project = "optimum-bonbon-237812"
}

1. terraform plan runs successfully.
2. Remove version constraint and remove .terraform directory.
3. Run terraform init -- v3.0.0-beta.1 is downloaded.
4. Run terraform plan, receive the following error:

googleapi: Error 404: The 'parameters.managedZone' resource named 'projects/optimum-bonbon-237812/managedZones/daved-test-zone' does not exist., notFound

I wanted to add this comment on this GitHub issue since it looks like 3.0.0-beta.1 changes how project ID is calculated, but please let me know if you need me to file a separate issue in GitHub for this.

Thanks!

[universalvishwa]

@paddycarver, It seems like the provider downloading is fixed now. Refer the output below.
```Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

• Checking for available provider plugins on https://releases.hashicorp.com...
• Downloading plugin for provider "google" (2.20.0)...
• Downloading plugin for provider "google-beta" (2.20.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

• provider.google: version = "~> 2.20"
• provider.google-beta: version = "~> 2.20"

Terraform has been successfully initialized!

But, this also caused an issue in creating instance groups. Refer the error below. (I'm not sure if I should open a another bug report for this. Let me know, if so) This is similar to what @slevenick is experiencing I suppose.

*Terraform version : 0.11.14 and 0.12.12*

...
google_compute_region_instance_group_manager.default: Refreshing state... (ID: projects/my-project-7jewd/regions/us-east1...roupManagers/my-instance-grp)

Error: Error refreshing state: 1 error occurred:
* module.us-east1.google_compute_region_instance_group_manager.default: 1 error occurred:
* module.us-east1.google_compute_region_instance_group_manager.default: google_compute_region_instance_group_manager.default: Invalid region instance group manager specifier. Expecting either {projectId}/{region}/{name} or {name}, where {projectId} and {region} will be derived from the provider.
```

[slevenick]

@dedamico That is a separate issue, looks like the google_dns_managed_zone datasource is currently broken. I filed an issue here: https://github.com/terraform-providers/terraform-provider-google/issues/4939

and a fix will be coming soon via https://github.com/GoogleCloudPlatform/magic-modules/pull/2712

@universalvishwa it looks like you are using a module to manage the instance group manager? Can you either provide the config that you are using that causes this error or point me at the module that you are using?

I believe the issues are caused by a config using the .id field from some resource, which is causing issues now that id formats have been updated. There should be existing fields on the resources that contain the same information that should be used rather than the id field

[universalvishwa]

@slevenick, I found the problem. When I run the init, the data source for google_project still returns this (id = projects/my-project-7jewd). The below output is was what is shown in terraform show command sourcing form the remote state file. I tried to do terraform taint on the data source and re-run init to re-initialize the data source, but it doesn't allow me taint it.

data.google_project.project:
  id = projects/my-project-7jewd
  billing_account = XXXXX-XXXXX-XXXXXX
  folder_id = 83037301714054
  name = my-project
  number = 749318403431
  org_id = 
  project_id = my-project-7jewd

[slevenick]

The id field should not be used as an input to other resources. Use the project_id field instead, like: project = data.google_project.project_id

Does that solve the issue?

[universalvishwa]

I suppose it will. In addition to that, may I ask why is it recommended to do so? (Just for me my understanding)

More over, this was the original issues I had initially.

[slevenick]

The issue is that the format of the id field changed from provider version 2.x -> 3.0.0. In 2.20 and below it was recommended to not use the id field as an input to other resources because the format was not documented and it was likely to change (as it has here).

We decided to move ids to be similar to the self_link field on many resources, so that it can be used as an input for other resources in the future. To do this we had to update the id formats of many resources and datasources. Many resources changed from an id of name to for example: projects/{project-name}/zones/{zone}/instances/{name}

This allows us to potentially send the id as the fully qualified path for the resource, and have standardization across most resources. But it does cause issues for people who were depending on the id field in configs in versions 2.x. The recommended fix is to update configs to not use the id field until 3.0.0

[slevenick]

Closing this, if there are further questions either re-open this or file a different issue

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!