Terraform: Real and shadow states do not match

Created on 7 Nov 2016  ยท  12Comments  ยท  Source: hashicorp/terraform

After updating to Terraform 0.7.9, I'm seeing an "Experimental feature failure". I'll provide the output from the command here, but it may be difficult to provide reproduction steps as we have a few private modules that we are using.

Here is the output from terraform apply, I have redacted a bunch of sensitive fields:

Experimental feature failure! Please report a bug.

This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.

While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.

The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!

You can report an issue at: https://github.com/hashicorp/terraform/issues

3 error(s) occurred:

* apply operation: Real and shadow states do not match! Real state:

<no state>
module.iam:
  aws_iam_instance_profile.instance_profile:
    ID = test-20161107221623836236165q3b
    arn = arn:aws:iam::REDACTED:instance-profile/test-20161107221623836236165q3b
    name = test-20161107221623836236165q3b
    name_prefix = test-
    path = /
    roles.# = 1
    roles.548918501 = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
  aws_iam_role.role:
    ID = test-201611072216228566408916wh
    arn = arn:aws:iam::REDACTED:role/test-201611072216228566408916wh
    assume_role_policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }

    name = test-201611072216228566408916wh
    name_prefix = test-
    path = /
    unique_id = REDACTED

    Dependencies:
      data.template_file.role_policy
  aws_iam_role_policy.packages:
    ID = test-201611072216228566408916wh:packages
    name = packages
    policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::REDACTED/*",
          "arn:aws:s3:::REDACTED/*"
        ]
      }
    ]
  }

    role = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
      data.template_file.packages
  aws_iam_role_policy.puppet:
    ID = test-201611072216228566408916wh:puppet
    name = puppet
    policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::REDACTED/*"
      }
    ]
  }

    role = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
      data.template_file.puppet
  data.template_file.packages:
    ID = 38c04cfd1692bd1f674a0ffca2cdf26407c72b89bb50c9b6e43d3921e83aed97
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::REDACTED/*",
          "arn:aws:s3:::REDACTED/*"
        ]
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::${bucket}/deb/*",
          "arn:aws:s3:::${bucket}/ubuntu/*"
        ]
      }
    ]
  }

    vars.% = 1
    vars.bucket = REDACTED
  data.template_file.puppet:
    ID = 9dd85f0b9893dc883a4dfa084450270c827d9f7e1159b632b51136d98811c54e
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::REDACTED/*"
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::${bucket}/*"
      }
    ]
  }

    vars.% = 1
    vars.bucket = REDACTED
  data.template_file.role_policy:
    ID = f2b05e93e4aaaf673536470607c186b983c02095ecea8ddb7a1dbf24e11a9765
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }


  Outputs:

  instance_profile = test-20161107221623836236165q3b
  role = test-201611072216228566408916wh
module.instance:
  aws_instance.instance:
    ID = REDACTED
    ami = REDACTED
    associate_public_ip_address = false
    availability_zone = ap-southeast-2a
    disable_api_termination = false
    ebs_block_device.# = 0
    ebs_optimized = false
    ephemeral_block_device.# = 0
    iam_instance_profile = test-20161107221623836236165q3b
    instance_state = running
    instance_type = c3.large
    key_name = 
    monitoring = false
    network_interface_id = 
    private_dns = REDACTED
    private_ip = REDACTED
    public_dns = REDACTED
    public_ip = REDACTED
    root_block_device.# = 0
    security_groups.# = 1
    security_groups.3233087073 = base
    source_dest_check = true
    subnet_id = 
    tags.% = 3
    tags.Name = Test
    tags.realm = test
    tags.role = test
    tenancy = default
    user_data = 2400f7cce084dd0f39843159d24c21cdd80c5823
    vpc_security_group_ids.# = 0

    Dependencies:
      data.terraform_remote_state.foundation
      module.ami
      module.user_data
  data.terraform_remote_state.foundation:
    ID = 2016-11-07 22:18:20.738815458 +0000 UTC
    backend = local
    common_security_group = base
    common_security_group_id = REDACTED
    config.% = 1
    config.path = /REDACTED/../foundation/terraform.tfstate
    configuration_bucket = REDACTED
    elb_logs_bucket = REDACTED
    external_route53_zone_id = REDACTED
    internal_route53_zone_id = REDACTED
    s3_logs_bucket = REDACTED

  Outputs:

  availability_zone = [ap-southeast-2a]
  id = [REDACTED]
  private_dns = [REDACTED]
  private_ip = [REDACTED]
  public_dns = [REDACTED]
  public_ip = [REDACTED]
module.instance.ami:
  data.aws_ami.ami:
    ID = REDACTED
    architecture = x86_64
    block_device_mappings.# = 2
    block_device_mappings.215362657.device_name = sdb
    block_device_mappings.215362657.ebs.% = 0
    block_device_mappings.215362657.no_device = 
    block_device_mappings.215362657.virtual_name = ephemeral0
    block_device_mappings.2294479958.device_name = sdc
    block_device_mappings.2294479958.ebs.% = 0
    block_device_mappings.2294479958.no_device = 
    block_device_mappings.2294479958.virtual_name = ephemeral1
    creation_date = 2016-11-07T13:10:07.000Z
    description = role=base os=linux distribution=ubuntu codename=trusty
    filter.# = 5
    filter.1353251163.name = state
    filter.1353251163.values.# = 1
    filter.1353251163.values.0 = available
    filter.2504063201.name = description
    filter.2504063201.values.# = 1
    filter.2504063201.values.0 = role=base os=linux distribution=ubuntu codename=trusty
    filter.279282180.name = is-public
    filter.279282180.values.# = 1
    filter.279282180.values.0 = false
    filter.3834012668.name = root-device-type
    filter.3834012668.values.# = 1
    filter.3834012668.values.0 = instance-store
    filter.400084487.name = virtualization-type
    filter.400084487.values.# = 1
    filter.400084487.values.0 = paravirtual
    hypervisor = xen
    image_id = ami-b07f40d3
    image_location = REDACTED
    image_type = machine
    kernel_id = aki-c362fff9
    most_recent = true
    name = base (trusty pv.instance-store) 1478523605
    owner_id = REDACTED
    product_codes.# = 0
    public = false
    root_device_name = /dev/sda1
    root_device_type = instance-store
    state = available
    state_reason.% = 2
    state_reason.code = UNSET
    state_reason.message = UNSET
    tags.# = 0
    virtualization_type = paravirtual

  Outputs:

  id = ami-b07f40d3
module.instance.user_data:
  data.template_cloudinit_config.user_data:
    ID = 3270850169
    base64_encode = false
    gzip = false
    part.# = 1
    part.0.content = REDACTED

  --MIMEBOUNDARY--


    Dependencies:
      data.template_file.bootstrap
  data.template_file.bootstrap:
    ID = 0095c43d576b525c63afb2d70d454644afa6d57fdbbf6813ed5b1910967d6c33
    rendered = REDACTED

    template = REDACTED

    vars.% = 3
    vars.branch = master
    vars.realm = test
    vars.role = test

  Outputs:

  rendered = REDACTED

  --MIMEBOUNDARY--

Shadow state:

<no state>
module.iam:
  aws_iam_instance_profile.instance_profile:
    ID = test-20161107221623836236165q3b
    arn = arn:aws:iam::REDACTED:instance-profile/test-20161107221623836236165q3b
    name = test-20161107221623836236165q3b
    name_prefix = test-
    path = /
    roles.# = 1
    roles.548918501 = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
  aws_iam_role.role:
    ID = test-201611072216228566408916wh
    arn = arn:aws:iam::REDACTED:role/test-201611072216228566408916wh
    assume_role_policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }

    name = test-201611072216228566408916wh
    name_prefix = test-
    path = /
    unique_id = AROAIKCQ3M4PN6HFEVMEA

    Dependencies:
      data.template_file.role_policy
  aws_iam_role_policy.packages:
    ID = test-201611072216228566408916wh:packages
    name = packages
    policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::REDACTED/*",
          "arn:aws:s3:::REDACTED/*"
        ]
      }
    ]
  }

    role = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
      data.template_file.packages
  aws_iam_role_policy.puppet:
    ID = test-201611072216228566408916wh:puppet
    name = puppet
    policy = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::REDACTED/*"
      }
    ]
  }

    role = test-201611072216228566408916wh

    Dependencies:
      aws_iam_role.role
      data.template_file.puppet
  data.template_file.packages:
    ID = 38c04cfd1692bd1f674a0ffca2cdf26407c72b89bb50c9b6e43d3921e83aed97
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::REDACTED/*",
          "arn:aws:s3:::REDACTED/*"
        ]
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect":"Allow",
        "Action": "s3:GetObject*",
        "Resource": [
          "arn:aws:s3:::${bucket}/deb/*",
          "arn:aws:s3:::${bucket}/ubuntu/*"
        ]
      }
    ]
  }

    vars.% = 1
    vars.bucket = freelancer-packages
  data.template_file.puppet:
    ID = 9dd85f0b9893dc883a4dfa084450270c827d9f7e1159b632b51136d98811c54e
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::REDACTED/*"
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeTags",
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::${bucket}/git/puppet/*"
      }
    ]
  }

    vars.% = 1
    vars.bucket = freelancer-packages
  data.template_file.role_policy:
    ID = f2b05e93e4aaaf673536470607c186b983c02095ecea8ddb7a1dbf24e11a9765
    rendered = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }

    template = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "ec2.amazonaws.com"
        }
      }
    ]
  }


  Outputs:

  instance_profile = test-20161107221623836236165q3b
  role = test-201611072216228566408916wh
module.instance:
  data.terraform_remote_state.foundation:
    ID = 2016-11-07 22:18:20.738815458 +0000 UTC
    backend = local
    common_security_group = base
    common_security_group_id = REDACTED
    config.% = 1
    config.path = REDACTED/../foundation/terraform.tfstate
    configuration_bucket = REDACTED
    elb_logs_bucket = REDACTED
    external_route53_zone_id = REDACTED
    internal_route53_zone_id = REDACTED
    s3_logs_bucket = REDACTED
module.instance.ami:
  data.aws_ami.ami:
    ID = REDACTED
    architecture = x86_64
    block_device_mappings.# = 2
    block_device_mappings.215362657.device_name = sdb
    block_device_mappings.215362657.ebs.% = 0
    block_device_mappings.215362657.no_device = 
    block_device_mappings.215362657.virtual_name = ephemeral0
    block_device_mappings.2294479958.device_name = sdc
    block_device_mappings.2294479958.ebs.% = 0
    block_device_mappings.2294479958.no_device = 
    block_device_mappings.2294479958.virtual_name = ephemeral1
    creation_date = 2016-11-07T13:10:07.000Z
    description = role=base os=linux distribution=ubuntu codename=trusty
    filter.# = 5
    filter.1353251163.name = state
    filter.1353251163.values.# = 1
    filter.1353251163.values.0 = available
    filter.2504063201.name = description
    filter.2504063201.values.# = 1
    filter.2504063201.values.0 = role=base os=linux distribution=ubuntu codename=trusty
    filter.279282180.name = is-public
    filter.279282180.values.# = 1
    filter.279282180.values.0 = false
    filter.3834012668.name = root-device-type
    filter.3834012668.values.# = 1
    filter.3834012668.values.0 = instance-store
    filter.400084487.name = virtualization-type
    filter.400084487.values.# = 1
    filter.400084487.values.0 = paravirtual
    hypervisor = xen
    image_id = REDACTED
    image_location = REDACTED
    image_type = machine
    kernel_id = aki-c362fff9
    most_recent = true
    name = base (trusty pv.instance-store) 1478523605
    owner_id = REDACTED
    product_codes.# = 0
    public = false
    root_device_name = /dev/sda1
    root_device_type = instance-store
    state = available
    state_reason.% = 2
    state_reason.code = UNSET
    state_reason.message = UNSET
    tags.# = 0
    virtualization_type = paravirtual

  Outputs:

  id = REDACTED
module.instance.user_data:
  data.template_cloudinit_config.user_data:
    ID = 3270850169
    base64_encode = false
    gzip = false
    part.# = 1
    part.0.content = REDACTED

    part.0.content_type = text/x-shellscript
    part.0.filename = bootstrap.sh
    part.0.merge_type = 
    rendered = REDACTED

  --MIMEBOUNDARY--


    Dependencies:
      data.template_file.bootstrap
  data.template_file.bootstrap:
    ID = 0095c43d576b525c63afb2d70d454644afa6d57fdbbf6813ed5b1910967d6c33
    rendered = REDACTED

    template = REDACTED

    vars.% = 3
    vars.branch = master
    vars.realm = test
    vars.role = test

  Outputs:

  rendered = REDACTED

  --MIMEBOUNDARY--


* apply operation: Real and shadow diffs do not match! Real diff:

module.iam:
module.instance:

Shadow diff:

module.iam:
module.instance:
  CREATE: aws_instance.instance
    ami:                         "" => "REDACTED" (forces new resource)
    associate_public_ip_address: "" => "false" (forces new resource)
    availability_zone:           "" => "ap-southeast-2a" (forces new resource)
    ebs_block_device.#:          "" => "<computed>"
    ebs_optimized:               "" => "false" (forces new resource)
    ephemeral_block_device.#:    "" => "<computed>"
    iam_instance_profile:        "" => "test-20161107221623836236165q3b" (forces new resource)
    instance_state:              "" => "<computed>"
    instance_type:               "" => "c3.large" (forces new resource)
    key_name:                    "" => "<computed>" (forces new resource)
    monitoring:                  "" => "false"
    network_interface_id:        "" => "<computed>"
    placement_group:             "" => "<computed>" (forces new resource)
    private_dns:                 "" => "<computed>"
    private_ip:                  "" => "<computed>" (forces new resource)
    public_dns:                  "" => "<computed>"
    public_ip:                   "" => "<computed>"
    root_block_device.#:         "" => "<computed>"
    security_groups.#:           "" => "1" (forces new resource)
    security_groups.3233087073:  "" => "base" (forces new resource)
    source_dest_check:           "" => "true"
    subnet_id:                   "" => "<computed>" (forces new resource)
    tags.%:                      "" => "3"
    tags.Name:                   "" => "Test"
    tags.realm:                  "" => "test"
    tags.role:                   "" => "test"
    tenancy:                     "" => "<computed>" (forces new resource)
    user_data:                   "" => "2400f7cce084dd0f39843159d24c21cdd80c5823" (forces new resource)


* apply operation: 1:3: unknown variable accessed: var.iam_instance_profile in:

${var.iam_instance_profile}

This is not an error. Your terraform operation completed successfully
and your real infrastructure is unaffected by this message.

Here is the Terraform code that was used:

module "iam" {
  source      = "../../../../modules/iam"
  puppet_role = "test"
}

module "instance" {
  source = "../../../../modules/instance"

  ami_codename            = "trusty"
  ami_virtualization_type = "paravirtual"
  ami_storage_type        = "instance-store"
  availability_zones      = ["ap-southeast-2a"]
  instance_type           = "c3.large"
  iam_instance_profile    = "${module.iam.instance_profile}"

  name         = "Test"
  puppet_realm = "test"
  puppet_role  = "test"
}
bug core

Most helpful comment

Exactly the same problem here:

apply operation: Real and shadow states do not match!
apply operation: Real and shadow diffs do not match!
apply operation: 1:3: unknown variable accessed: var.vpc_id

Is there a way to disable "experimental features" that are causing this error? It fills up my CI logs with a bunch of jumble I don't want.

All 12 comments

Exactly the same problem here:

apply operation: Real and shadow states do not match!
apply operation: Real and shadow diffs do not match!
apply operation: 1:3: unknown variable accessed: var.vpc_id

Is there a way to disable "experimental features" that are causing this error? It fills up my CI logs with a bunch of jumble I don't want.

I also get the apply operation: Real and shadow states do not match! message, and the error message of experimental features is too long, I need to scroll my screen to check my apply result.

Disable "experimental features" may be not good for development, simple/verbose flag for experimental features message seems be better

Same issue here

apply operation: Real and shadow diffs do not match! Real diff
* apply operation: 1:3: unknown variable accessed: var.keypair_devops in:

and so on...

Would be great to disable this output so I can see what actually changed.

Thanks! Fixed in master, will be in 0.7.10

@mitchellh I am still seeing this issue with 0.7.11:

ubuntu@ip-10-0-0-248:~/demos/infra/terraform/providers/aws/network/dev$ terraform --version
Terraform v0.7.11

Destroy complete! Resources: 32 destroyed.

Experimental feature failure! Please report a bug.

This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.

While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.

The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!

You can report an issue at: https://github.com/hashicorp/terraform/issues

1 error(s) occurred:

  • apply operation: Real and shadow states do not match! Real state:


Outputs:

bastion_user = ec2-user

module.network:

Outputs:

bastion_user = ec2-user
module.network.bastion:

Outputs:

user = ec2-user
module.network.nat:

module.network.private_subnet:

module.network.public_subnet:

module.network.vpc:

Shadow state:


Outputs:

bastion_user = ec2-user
boot_key_name = sa-demo-bootkey-20161115

module.network:

Outputs:

bastion_user = ec2-user
boot_key_name = sa-demo-bootkey-20161115
module.network.bastion:

Outputs:

user = ec2-user
module.network.nat:

module.network.private_subnet:

module.network.public_subnet:

module.network.vpc:

This is not an error. Your terraform operation completed successfully
and your real infrastructure is unaffected by this message.

I am also seeing a similar error message to what @ctindel posted above, running on terraform 0.7.11

I am also getting this error message on terraform 0.7.11

1 error(s) occurred:

Experimental feature failure! Please report a bug.

This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.

While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.

The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!

You can report an issue at: https://github.com/hashicorp/terraform/issues

2 error(s) occurred:

* plan operation: Real and shadow states do not match! Real state:

Also get this message since moving from 0.7.8 => 0.7.11. Have truncated the huge output which seems to include everything from all modules.

Update: I am seeing this when planning/applying on a subset of targets.

Experimental feature failure! Please report a bug.

This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.

While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.

The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!

You can report an issue at: https://github.com/hashicorp/terraform/issues

1 error(s) occurred:

* apply operation: Real and shadow states do not match! Real state:

@mitchellh I get the desire to test these features out but I provide Terraform to a few hundred engineers who get these messages now and it creates a very poor experience. Are you opposed to disabling them by default or providing a method (environment variable?) to disable the messaging?

@bigkraig We improved the experience in 0.7.13 by making the error message much shorter and putting the bulk of it into a log file.

Disabling by default is scary simply because for short term UX we may risk long term stability.

I'm going to lock this issue because it has been closed for _30 days_ โณ. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Was this page helpful?
0 / 5 - 0 ratings