Terraform: 0.12 upgrade shows plan with changed resource but no change is shown and a hidden empty list vs. None change happens.
Terraform Version
Terraform v0.12.1 with google provider v2.4.1.
I want to do an upgrade from Terraform v0.11.14 with google provider v2.4.1 to Terraform v0.12.1 (or later) with the same google provider.
I built these versions locally. The google provider code is unchanged between Terraform v0.11.14 and v0.12.1 to isolate the failing component. While the google provider code is unchanged, I built it against the corresponding Terraform versions; I used google provider 2.4.1 because it builds against both, Terraform 0.11.14 and Terraform 0.12.1. Below, I will use terraform12 to refer to my v0.12.1 build and terraform11 to refer to my v0.11.14 build.
Terraform Configuration Files
A default setup, copy&pasted from the docs. I create a google_compute_instance where the disk output field gets populated. For this, I create and attach a disk.
resource "google_compute_disk" "default" {
project = "${google_project.my_project.id}"
name = "test-disk"
image = "debian-8-jessie-v20170523"
type = "pd-ssd"
zone = "us-central1-f"
labels = {
environment = "dev"
}
physical_block_size_bytes = 4096
}
resource "google_compute_attached_disk" "default" {
project = "${google_project.my_project.id}"
disk = "${google_compute_disk.default.self_link}"
instance = "${google_compute_instance.default.self_link}"
}
resource "google_compute_instance" "default" {
project = "${google_project.my_project.id}"
name = "attached-disk-instance"
machine_type = "n1-standard-1"
zone = "us-central1-f"
boot_disk {
initialize_params {
image = "debian-cloud/debian-9"
}
}
network_interface {
network = "default"
}
lifecycle {
ignore_changes = ["attached_disk"]
}
}
Debug Output
Available on request. Nothing special here.
Crash Output
No crash.
Expected Behavior
terraformv11 apply creates the resources. Afterwards, terraform11 plan shows No changes. Infrastructure is up-to-date.
Upgrading to terraform12: terraform12 plan should show No changes. Infrastructure is up-to-date.
Actual Behavior
terraform12 plan shows the following plan:
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
[...]
google_compute_disk.default: Refreshing state... [id=test-disk]
google_compute_instance.default: Refreshing state... [id=attached-disk-instance]
google_compute_image.debian_testing: Refreshing state... [id=debian-testing]
google_compute_attached_disk.default: Refreshing state... [id=attached-disk-instance:test-disk]
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# google_compute_instance.default will be updated in-place
~ resource "google_compute_instance" "default" {
can_ip_forward = false
cpu_platform = "Intel Haswell"
deletion_protection = false
id = "attached-disk-instance"
instance_id = "2071836455626482860"
label_fingerprint = "42WmSpB8rSM="
labels = {}
machine_type = "n1-standard-1"
metadata = {}
metadata_fingerprint = "vYoMe6PbG1A="
name = "attached-disk-instance"
project = "my-test-project"
self_link = "https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/instances/attached-disk-instance"
tags = []
tags_fingerprint = "42WmSpB8rSM="
zone = "us-central1-f"
attached_disk {
device_name = "persistent-disk-1"
mode = "READ_WRITE"
source = "https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/test-disk"
}
boot_disk {
auto_delete = true
device_name = "persistent-disk-0"
source = "https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/attached-disk-instance"
initialize_params {
image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20190514"
size = 10
type = "pd-standard"
}
}
network_interface {
name = "nic0"
network = "https://www.googleapis.com/compute/v1/projects/my-test-projectglobal/networks/default"
network_ip = "10.128.0.4"
subnetwork = "https://www.googleapis.com/compute/v1/projects/my-test-project/regions/us-central1/subnetworks/default"
subnetwork_project = "my-test-project"
}
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
preemptible = false
}
timeouts {}
}
Plan: 0 to add, 1 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
In my opinion, there are tow bugs:
- Terraform
planshould not show any change for a 0.11 to 0.12 upgrade in this case. - The
planoutput does not show what is changed.
Steps to Reproduce
terraform11 initterraform11 apply: Create everything withterraform11terraform plan: Verify thatNo changes. Infrastructure is up-to-date.rm -rf .terraformterraform12 initterraform12 plan: The plan output showsPlan: 0 to add, 1 to change, 0 to destroy.but no visible change. The bug is visible here and the unexpected plan dumped above is shown.
The follow steps are just gathering further debug output.
terraform plan -out=tfplanterraform show -json tfplan- Copy state into
ipython3(I use variable namedstate); in the python shellimport pprint, jsonand dump the interesting change withpprint.pprint([r for r in json.loads(state)['resource_changes'] if r['change']['actions'] != ['no-op']]).
Additional Context
Inspecting the JSON state, I see the following:
[{'address': 'google_compute_instance.default',
'change': {'actions': ['update'],
'after': {'allow_stopping_for_update': None,
'attached_disk': [{'device_name': 'persistent-disk-1',
'disk_encryption_key_raw': '',
'disk_encryption_key_sha256': '',
'mode': 'READ_WRITE',
'source': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/test-disk'}],
'boot_disk': [{'auto_delete': True,
'device_name': 'persistent-disk-0',
'disk_encryption_key_raw': '',
'disk_encryption_key_sha256': '',
'initialize_params': [{'image': 'https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20190514',
'size': 10,
'type': 'pd-standard'}],
'source': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/attached-disk-instance'}],
'can_ip_forward': False,
'cpu_platform': 'Intel Haswell',
'create_timeout': None,
'deletion_protection': False,
'description': None,
'disk': [],
'guest_accelerator': [],
'hostname': '',
'id': 'attached-disk-instance',
'instance_id': '2071836455626482860',
'label_fingerprint': '42WmSpB8rSM=',
'labels': {},
'machine_type': 'n1-standard-1',
'metadata': {},
'metadata_fingerprint': 'vYoMe6PbG1A=',
'metadata_startup_script': '',
'min_cpu_platform': '',
'name': 'attached-disk-instance',
'network_interface': [{'access_config': [],
'address': '',
'alias_ip_range': [],
'name': 'nic0',
'network': 'https://www.googleapis.com/compute/v1/projects/my-test-project/global/networks/default',
'network_ip': '10.128.0.4',
'subnetwork': 'https://www.googleapis.com/compute/v1/projects/my-test-project/regions/us-central1/subnetworks/default',
'subnetwork_project': 'my-test-project'}],
'project': 'my-test-project',
'scheduling': [{'automatic_restart': True,
'on_host_maintenance': 'MIGRATE',
'preemptible': False}],
'scratch_disk': [],
'self_link': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/instances/attached-disk-instance',
'service_account': [],
'tags': [],
'tags_fingerprint': '42WmSpB8rSM=',
'timeouts': {'create': None,
'delete': None,
'update': None},
'zone': 'us-central1-f'},
'after_unknown': {},
'before': {'allow_stopping_for_update': None,
'attached_disk': [{'device_name': 'persistent-disk-1',
'disk_encryption_key_raw': '',
'disk_encryption_key_sha256': '',
'mode': 'READ_WRITE',
'source': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/test-disk'}],
'boot_disk': [{'auto_delete': True,
'device_name': 'persistent-disk-0',
'disk_encryption_key_raw': '',
'disk_encryption_key_sha256': '',
'initialize_params': [{'image': 'https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20190514',
'size': 10,
'type': 'pd-standard'}],
'source': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/disks/attached-disk-instance'}],
'can_ip_forward': False,
'cpu_platform': 'Intel Haswell',
'create_timeout': None,
'deletion_protection': False,
'description': None,
'disk': None,
'guest_accelerator': [],
'hostname': '',
'id': 'attached-disk-instance',
'instance_id': '2071836455626482860',
'label_fingerprint': '42WmSpB8rSM=',
'labels': {},
'machine_type': 'n1-standard-1',
'metadata': {},
'metadata_fingerprint': 'vYoMe6PbG1A=',
'metadata_startup_script': '',
'min_cpu_platform': '',
'name': 'attached-disk-instance',
'network_interface': [{'access_config': [],
'address': '',
'alias_ip_range': [],
'name': 'nic0',
'network': 'https://www.googleapis.com/compute/v1/projects/my-test-project/global/networks/default',
'network_ip': '10.128.0.4',
'subnetwork': 'https://www.googleapis.com/compute/v1/projects/my-test-project/regions/us-central1/subnetworks/default',
'subnetwork_project': 'my-test-project'}],
'project': 'my-test-project',
'scheduling': [{'automatic_restart': True,
'on_host_maintenance': 'MIGRATE',
'preemptible': False}],
'scratch_disk': [],
'self_link': 'https://www.googleapis.com/compute/v1/projects/my-test-project/zones/us-central1-f/instances/attached-disk-instance',
'service_account': [],
'tags': [],
'tags_fingerprint': '42WmSpB8rSM=',
'timeouts': {'create': None,
'delete': None,
'update': None},
'zone': 'us-central1-f'}},
'mode': 'managed',
'name': 'default',
'provider_name': 'google',
'type': 'google_compute_instance'}]
The only difference here is:
after: 'disk': []
before: 'disk': None
References
- Potentially related: https://github.com/hashicorp/terraform/issues/21347#issuecomment-495782365
diekmann
All 7 comments
Hi @diekmann,
Thanks for filing the issue. I'm not able to replicate this plan, and I think this is likely taken care in a more recent google provider release. Can you try with the latest version?
jbardin
on 11 Jun 2019
Thanks for checking.
I tried again, this time with the official binaries:
- https://releases.hashicorp.com/terraform/0.11.14/terraform_0.11.14_linux_amd64.zip is
terraform11 - https://releases.hashicorp.com/terraform/0.12.1/terraform_0.12.1_linux_amd64.zip is
terraform12
I also let Terraform download the official provider binary each time. I made sure to cleanup the downloaded providers whenever switching from terraform11 to terraform12.
First, I pinned the version of the google provider to the one closest to 2.4.1 which is officially compatible with Terraform 0.12 and 0.11.
provider "google" {
version = ">= 2.5.1, <= 2.5.1"
}
During init, I made sure I saw Downloading plugin for provider "google" (terraform-providers/google) 2.5.1...
Again, I created the resources with terraform11 and then upgraded to terraform12, expecting to see No changes. Infrastructure is up-to-date.
The terraform12 plan problem exists on my freshly imaged machine and I can reproduce it.
Hence, if the 3 resources were created with Terraform 0.11.14 and google provider 2.5.1 and I subsequently upgrade to Terraform 0.12.1 and google provider 2.5.1, I see this strange plan shown in the initial bug report.
Without deleting the resources, I remove the version constraint and run terraform12 init -upgrade. I get "google" (terraform-providers/google) 2.8.0....
The terraform12 plan problem exists.
Hence, if the 3 resources were created with Terraform 0.11.14 and google provider 2.5.1 and I subsequently upgrade to Terraform 0.12.1 and google provider 2.8.0, I see this strange plan shown in the initial bug report.
I do a full cleanup, also removing the three example resources to re-create them with the most up to date google provider version.
I try again:
$ terraform11 init
...
- Downloading plugin for provider "google" (2.8.0)...
...
$ terraform11 apply
...
`Apply complete! Resources: 3 added, 0 changed, 0 destroyed.`
$ rm -rf .terraform/
$ terraform12 init
...
- Downloading plugin for provider "google" (terraform-providers/google) 2.8.0...
$ terraform12 plan
The terraform12 plan problem exists.
Hence, if the 3 resources were created with Terraform 0.11.14 and google provider 2.8.0 and I subsequently upgrade to Terraform 0.12.1 and google provider 2.8.0, I see this strange plan shown in the initial bug report.
Is this a 0.12 upgrade one-time thing?
When I run terraform12 apply once, I get the following output:
...
Terraform will perform the following actions:
# google_compute_instance.default will be updated in-place
....
plan output without change as shown above
....
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
google_compute_instance.default: Modifying... [id=attached-disk-instance]
google_compute_instance.default: Modifications complete after 4s [id=attached-disk-instance]
Seems like it's doing nothing.
Further runs of terraform12 plan correctly show No changes. Infrastructure is up-to-date..
Side note: do we care about this? Story telling time with an anecdote :smiley_cat:
This bug may seem like a harmless change which a user may occur only once during a 0.11 to 0.12 upgrade. However, these small differences can have huge semantical differences. So far, I could not find a critical flaw following from this bug here, but I found a very similar issue once: In a google_compute_instance, I saw access_config = []. I changed it to access_config {}. This small change (= [] vs. {}) caused the machine to get a public IP address, and exposed the machine to the Internet. For security, the machine should not have any Internet connectivity. Since this incident, I'm very careful about the seemingly irrelevant differences between "empty values".
diekmann
on 11 Jun 2019
Thanks for info @diekmann,
The 2.8.0 version of the google provider should contain what's necessary for it to reach a steady state with this resource. While we've tried to minimize the disruption, it is certainly possible that the provider under Terraform 0.11 simply did not store enough information to make the upgrade from the 0.11 state smoothly.
Regarding the access_config = [] vs access_config {}, the new configuration language has been designed to remove that sort of ambiguity, so that sort of unexpected change can't happen. This will improve as providers get more access to the new plugin protocol and strong type system in the future.
I'lll see what I can do to help improve the upgrade process here, since while this change seems benign, if one of those fields required replacing the resource it would make upgrading the resource quite difficult.
jbardin
on 11 Jun 2019
Thanks James! Are you planning a Terraform v0.11.15 which will make the transition to 0.12 smoother? Can a 0.12.2 potentially catch this case?
Fun fact: I made the change in access_config to prepare for 0.12 since access_config = [] is illegal in 0.12. But not checking the plan there was my failure. On the other hand, anything which minimizes the plan for a 0.11 to 0.12 upgrade reduces the noise and helps to catch such errors.
diekmann
on 11 Jun 2019
There really isn't anything that 0.11 could do in this case. If there is anything that can be done, the fixes are going to be in the SDK which means it would be included in a subsequent provider release.
jbardin
on 11 Jun 2019
OK, this one looks like we might be able to handle it a a little better in the state upgrade path. The issue here is that the upgrader is working with the cty.Type of the state, which is sufficient to know how to encode and decode it, but it is not sufficient to differentiate between nested blocks and in the config. The disk block does not exist in the old state, so is returned as null from the upgrade process, but since blocks are never null, just empty, the plan changes that value to an empty list block.
jbardin
on 12 Jun 2019
I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
![hashibot[bot] picture](https://avatars2.githubusercontent.com/in/8332?v=4&s=40)
hashibot[bot]
on 29 Mar 2020
Related issues
ketzacoatl
·
3Comments
c4milo
·
3Comments
allomov
·
3Comments
sprokopiak
·
3Comments
larstobi
·
3Comments
Most helpful comment
OK, this one looks like we might be able to handle it a a little better in the state upgrade path. The issue here is that the upgrader is working with the
cty.Typeof the state, which is sufficient to know how to encode and decode it, but it is not sufficient to differentiate between nested blocks and in the config. Thediskblock does not exist in the old state, so is returned asnullfrom the upgrade process, but since blocks are nevernull, just empty, the plan changes that value to an empty list block.