After updating to Terraform 0.7.9, I'm seeing an "Experimental feature failure". I'll provide the output from the command here, but it may be difficult to provide reproduction steps as we have a few private modules that we are using.
Here is the output from terraform apply, I have redacted a bunch of sensitive fields:
Experimental feature failure! Please report a bug.
This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.
While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.
The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!
You can report an issue at: https://github.com/hashicorp/terraform/issues
3 error(s) occurred:
* apply operation: Real and shadow states do not match! Real state:
<no state>
module.iam:
aws_iam_instance_profile.instance_profile:
ID = test-20161107221623836236165q3b
arn = arn:aws:iam::REDACTED:instance-profile/test-20161107221623836236165q3b
name = test-20161107221623836236165q3b
name_prefix = test-
path = /
roles.# = 1
roles.548918501 = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
aws_iam_role.role:
ID = test-201611072216228566408916wh
arn = arn:aws:iam::REDACTED:role/test-201611072216228566408916wh
assume_role_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
name = test-201611072216228566408916wh
name_prefix = test-
path = /
unique_id = REDACTED
Dependencies:
data.template_file.role_policy
aws_iam_role_policy.packages:
ID = test-201611072216228566408916wh:packages
name = packages
policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::REDACTED/*",
"arn:aws:s3:::REDACTED/*"
]
}
]
}
role = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
data.template_file.packages
aws_iam_role_policy.puppet:
ID = test-201611072216228566408916wh:puppet
name = puppet
policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::REDACTED/*"
}
]
}
role = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
data.template_file.puppet
data.template_file.packages:
ID = 38c04cfd1692bd1f674a0ffca2cdf26407c72b89bb50c9b6e43d3921e83aed97
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::REDACTED/*",
"arn:aws:s3:::REDACTED/*"
]
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::${bucket}/deb/*",
"arn:aws:s3:::${bucket}/ubuntu/*"
]
}
]
}
vars.% = 1
vars.bucket = REDACTED
data.template_file.puppet:
ID = 9dd85f0b9893dc883a4dfa084450270c827d9f7e1159b632b51136d98811c54e
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::REDACTED/*"
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::${bucket}/*"
}
]
}
vars.% = 1
vars.bucket = REDACTED
data.template_file.role_policy:
ID = f2b05e93e4aaaf673536470607c186b983c02095ecea8ddb7a1dbf24e11a9765
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
Outputs:
instance_profile = test-20161107221623836236165q3b
role = test-201611072216228566408916wh
module.instance:
aws_instance.instance:
ID = REDACTED
ami = REDACTED
associate_public_ip_address = false
availability_zone = ap-southeast-2a
disable_api_termination = false
ebs_block_device.# = 0
ebs_optimized = false
ephemeral_block_device.# = 0
iam_instance_profile = test-20161107221623836236165q3b
instance_state = running
instance_type = c3.large
key_name =
monitoring = false
network_interface_id =
private_dns = REDACTED
private_ip = REDACTED
public_dns = REDACTED
public_ip = REDACTED
root_block_device.# = 0
security_groups.# = 1
security_groups.3233087073 = base
source_dest_check = true
subnet_id =
tags.% = 3
tags.Name = Test
tags.realm = test
tags.role = test
tenancy = default
user_data = 2400f7cce084dd0f39843159d24c21cdd80c5823
vpc_security_group_ids.# = 0
Dependencies:
data.terraform_remote_state.foundation
module.ami
module.user_data
data.terraform_remote_state.foundation:
ID = 2016-11-07 22:18:20.738815458 +0000 UTC
backend = local
common_security_group = base
common_security_group_id = REDACTED
config.% = 1
config.path = /REDACTED/../foundation/terraform.tfstate
configuration_bucket = REDACTED
elb_logs_bucket = REDACTED
external_route53_zone_id = REDACTED
internal_route53_zone_id = REDACTED
s3_logs_bucket = REDACTED
Outputs:
availability_zone = [ap-southeast-2a]
id = [REDACTED]
private_dns = [REDACTED]
private_ip = [REDACTED]
public_dns = [REDACTED]
public_ip = [REDACTED]
module.instance.ami:
data.aws_ami.ami:
ID = REDACTED
architecture = x86_64
block_device_mappings.# = 2
block_device_mappings.215362657.device_name = sdb
block_device_mappings.215362657.ebs.% = 0
block_device_mappings.215362657.no_device =
block_device_mappings.215362657.virtual_name = ephemeral0
block_device_mappings.2294479958.device_name = sdc
block_device_mappings.2294479958.ebs.% = 0
block_device_mappings.2294479958.no_device =
block_device_mappings.2294479958.virtual_name = ephemeral1
creation_date = 2016-11-07T13:10:07.000Z
description = role=base os=linux distribution=ubuntu codename=trusty
filter.# = 5
filter.1353251163.name = state
filter.1353251163.values.# = 1
filter.1353251163.values.0 = available
filter.2504063201.name = description
filter.2504063201.values.# = 1
filter.2504063201.values.0 = role=base os=linux distribution=ubuntu codename=trusty
filter.279282180.name = is-public
filter.279282180.values.# = 1
filter.279282180.values.0 = false
filter.3834012668.name = root-device-type
filter.3834012668.values.# = 1
filter.3834012668.values.0 = instance-store
filter.400084487.name = virtualization-type
filter.400084487.values.# = 1
filter.400084487.values.0 = paravirtual
hypervisor = xen
image_id = ami-b07f40d3
image_location = REDACTED
image_type = machine
kernel_id = aki-c362fff9
most_recent = true
name = base (trusty pv.instance-store) 1478523605
owner_id = REDACTED
product_codes.# = 0
public = false
root_device_name = /dev/sda1
root_device_type = instance-store
state = available
state_reason.% = 2
state_reason.code = UNSET
state_reason.message = UNSET
tags.# = 0
virtualization_type = paravirtual
Outputs:
id = ami-b07f40d3
module.instance.user_data:
data.template_cloudinit_config.user_data:
ID = 3270850169
base64_encode = false
gzip = false
part.# = 1
part.0.content = REDACTED
--MIMEBOUNDARY--
Dependencies:
data.template_file.bootstrap
data.template_file.bootstrap:
ID = 0095c43d576b525c63afb2d70d454644afa6d57fdbbf6813ed5b1910967d6c33
rendered = REDACTED
template = REDACTED
vars.% = 3
vars.branch = master
vars.realm = test
vars.role = test
Outputs:
rendered = REDACTED
--MIMEBOUNDARY--
Shadow state:
<no state>
module.iam:
aws_iam_instance_profile.instance_profile:
ID = test-20161107221623836236165q3b
arn = arn:aws:iam::REDACTED:instance-profile/test-20161107221623836236165q3b
name = test-20161107221623836236165q3b
name_prefix = test-
path = /
roles.# = 1
roles.548918501 = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
aws_iam_role.role:
ID = test-201611072216228566408916wh
arn = arn:aws:iam::REDACTED:role/test-201611072216228566408916wh
assume_role_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
name = test-201611072216228566408916wh
name_prefix = test-
path = /
unique_id = AROAIKCQ3M4PN6HFEVMEA
Dependencies:
data.template_file.role_policy
aws_iam_role_policy.packages:
ID = test-201611072216228566408916wh:packages
name = packages
policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::REDACTED/*",
"arn:aws:s3:::REDACTED/*"
]
}
]
}
role = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
data.template_file.packages
aws_iam_role_policy.puppet:
ID = test-201611072216228566408916wh:puppet
name = puppet
policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::REDACTED/*"
}
]
}
role = test-201611072216228566408916wh
Dependencies:
aws_iam_role.role
data.template_file.puppet
data.template_file.packages:
ID = 38c04cfd1692bd1f674a0ffca2cdf26407c72b89bb50c9b6e43d3921e83aed97
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::REDACTED/*",
"arn:aws:s3:::REDACTED/*"
]
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect":"Allow",
"Action": "s3:GetObject*",
"Resource": [
"arn:aws:s3:::${bucket}/deb/*",
"arn:aws:s3:::${bucket}/ubuntu/*"
]
}
]
}
vars.% = 1
vars.bucket = freelancer-packages
data.template_file.puppet:
ID = 9dd85f0b9893dc883a4dfa084450270c827d9f7e1159b632b51136d98811c54e
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::REDACTED/*"
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::${bucket}/git/puppet/*"
}
]
}
vars.% = 1
vars.bucket = freelancer-packages
data.template_file.role_policy:
ID = f2b05e93e4aaaf673536470607c186b983c02095ecea8ddb7a1dbf24e11a9765
rendered = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
template = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
}
Outputs:
instance_profile = test-20161107221623836236165q3b
role = test-201611072216228566408916wh
module.instance:
data.terraform_remote_state.foundation:
ID = 2016-11-07 22:18:20.738815458 +0000 UTC
backend = local
common_security_group = base
common_security_group_id = REDACTED
config.% = 1
config.path = REDACTED/../foundation/terraform.tfstate
configuration_bucket = REDACTED
elb_logs_bucket = REDACTED
external_route53_zone_id = REDACTED
internal_route53_zone_id = REDACTED
s3_logs_bucket = REDACTED
module.instance.ami:
data.aws_ami.ami:
ID = REDACTED
architecture = x86_64
block_device_mappings.# = 2
block_device_mappings.215362657.device_name = sdb
block_device_mappings.215362657.ebs.% = 0
block_device_mappings.215362657.no_device =
block_device_mappings.215362657.virtual_name = ephemeral0
block_device_mappings.2294479958.device_name = sdc
block_device_mappings.2294479958.ebs.% = 0
block_device_mappings.2294479958.no_device =
block_device_mappings.2294479958.virtual_name = ephemeral1
creation_date = 2016-11-07T13:10:07.000Z
description = role=base os=linux distribution=ubuntu codename=trusty
filter.# = 5
filter.1353251163.name = state
filter.1353251163.values.# = 1
filter.1353251163.values.0 = available
filter.2504063201.name = description
filter.2504063201.values.# = 1
filter.2504063201.values.0 = role=base os=linux distribution=ubuntu codename=trusty
filter.279282180.name = is-public
filter.279282180.values.# = 1
filter.279282180.values.0 = false
filter.3834012668.name = root-device-type
filter.3834012668.values.# = 1
filter.3834012668.values.0 = instance-store
filter.400084487.name = virtualization-type
filter.400084487.values.# = 1
filter.400084487.values.0 = paravirtual
hypervisor = xen
image_id = REDACTED
image_location = REDACTED
image_type = machine
kernel_id = aki-c362fff9
most_recent = true
name = base (trusty pv.instance-store) 1478523605
owner_id = REDACTED
product_codes.# = 0
public = false
root_device_name = /dev/sda1
root_device_type = instance-store
state = available
state_reason.% = 2
state_reason.code = UNSET
state_reason.message = UNSET
tags.# = 0
virtualization_type = paravirtual
Outputs:
id = REDACTED
module.instance.user_data:
data.template_cloudinit_config.user_data:
ID = 3270850169
base64_encode = false
gzip = false
part.# = 1
part.0.content = REDACTED
part.0.content_type = text/x-shellscript
part.0.filename = bootstrap.sh
part.0.merge_type =
rendered = REDACTED
--MIMEBOUNDARY--
Dependencies:
data.template_file.bootstrap
data.template_file.bootstrap:
ID = 0095c43d576b525c63afb2d70d454644afa6d57fdbbf6813ed5b1910967d6c33
rendered = REDACTED
template = REDACTED
vars.% = 3
vars.branch = master
vars.realm = test
vars.role = test
Outputs:
rendered = REDACTED
--MIMEBOUNDARY--
* apply operation: Real and shadow diffs do not match! Real diff:
module.iam:
module.instance:
Shadow diff:
module.iam:
module.instance:
CREATE: aws_instance.instance
ami: "" => "REDACTED" (forces new resource)
associate_public_ip_address: "" => "false" (forces new resource)
availability_zone: "" => "ap-southeast-2a" (forces new resource)
ebs_block_device.#: "" => "<computed>"
ebs_optimized: "" => "false" (forces new resource)
ephemeral_block_device.#: "" => "<computed>"
iam_instance_profile: "" => "test-20161107221623836236165q3b" (forces new resource)
instance_state: "" => "<computed>"
instance_type: "" => "c3.large" (forces new resource)
key_name: "" => "<computed>" (forces new resource)
monitoring: "" => "false"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>" (forces new resource)
private_dns: "" => "<computed>"
private_ip: "" => "<computed>" (forces new resource)
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "1" (forces new resource)
security_groups.3233087073: "" => "base" (forces new resource)
source_dest_check: "" => "true"
subnet_id: "" => "<computed>" (forces new resource)
tags.%: "" => "3"
tags.Name: "" => "Test"
tags.realm: "" => "test"
tags.role: "" => "test"
tenancy: "" => "<computed>" (forces new resource)
user_data: "" => "2400f7cce084dd0f39843159d24c21cdd80c5823" (forces new resource)
* apply operation: 1:3: unknown variable accessed: var.iam_instance_profile in:
${var.iam_instance_profile}
This is not an error. Your terraform operation completed successfully
and your real infrastructure is unaffected by this message.
Here is the Terraform code that was used:
module "iam" {
source = "../../../../modules/iam"
puppet_role = "test"
}
module "instance" {
source = "../../../../modules/instance"
ami_codename = "trusty"
ami_virtualization_type = "paravirtual"
ami_storage_type = "instance-store"
availability_zones = ["ap-southeast-2a"]
instance_type = "c3.large"
iam_instance_profile = "${module.iam.instance_profile}"
name = "Test"
puppet_realm = "test"
puppet_role = "test"
}
Exactly the same problem here:
apply operation: Real and shadow states do not match!
apply operation: Real and shadow diffs do not match!
apply operation: 1:3: unknown variable accessed: var.vpc_id
Is there a way to disable "experimental features" that are causing this error? It fills up my CI logs with a bunch of jumble I don't want.
I also get the apply operation: Real and shadow states do not match! message, and the error message of experimental features is too long, I need to scroll my screen to check my apply result.
Disable "experimental features" may be not good for development, simple/verbose flag for experimental features message seems be better
Same issue here
apply operation: Real and shadow diffs do not match! Real diff
* apply operation: 1:3: unknown variable accessed: var.keypair_devops in:
and so on...
Would be great to disable this output so I can see what actually changed.
Thanks! Fixed in master, will be in 0.7.10
@mitchellh I am still seeing this issue with 0.7.11:
ubuntu@ip-10-0-0-248:~/demos/infra/terraform/providers/aws/network/dev$ terraform --version
Terraform v0.7.11
Destroy complete! Resources: 32 destroyed.
Experimental feature failure! Please report a bug.
This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.
While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.
The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!
You can report an issue at: https://github.com/hashicorp/terraform/issues
1 error(s) occurred:
Outputs:
bastion_user = ec2-user
module.network:
Outputs:
bastion_user = ec2-user
module.network.bastion:
Outputs:
user = ec2-user
module.network.nat:
module.network.private_subnet:
module.network.public_subnet:
module.network.vpc:
Shadow state:
Outputs:
bastion_user = ec2-user
boot_key_name = sa-demo-bootkey-20161115
module.network:
Outputs:
bastion_user = ec2-user
boot_key_name = sa-demo-bootkey-20161115
module.network.bastion:
Outputs:
user = ec2-user
module.network.nat:
module.network.private_subnet:
module.network.public_subnet:
module.network.vpc:
This is not an error. Your terraform operation completed successfully
and your real infrastructure is unaffected by this message.
I am also seeing a similar error message to what @ctindel posted above, running on terraform 0.7.11
I am also getting this error message on terraform 0.7.11
1 error(s) occurred:
Experimental feature failure! Please report a bug.
This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.
While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.
The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!
You can report an issue at: https://github.com/hashicorp/terraform/issues
2 error(s) occurred:
* plan operation: Real and shadow states do not match! Real state:
Also get this message since moving from 0.7.8 => 0.7.11. Have truncated the huge output which seems to include everything from all modules.
Update: I am seeing this when planning/applying on a subset of targets.
Experimental feature failure! Please report a bug.
This is not an error. Your Terraform operation completed successfully.
Your real infrastructure is unaffected by this message.
While running, Terraform sometimes tests experimental features in the
background. These features cannot affect real state and never touch
real infrastructure. If the features work properly, you see nothing.
If the features fail, this message appears.
The following failures happened while running experimental features.
Please report a Terraform bug so that future Terraform versions that
enable these features can be improved!
You can report an issue at: https://github.com/hashicorp/terraform/issues
1 error(s) occurred:
* apply operation: Real and shadow states do not match! Real state:
@mitchellh I get the desire to test these features out but I provide Terraform to a few hundred engineers who get these messages now and it creates a very poor experience. Are you opposed to disabling them by default or providing a method (environment variable?) to disable the messaging?
@bigkraig We improved the experience in 0.7.13 by making the error message much shorter and putting the bulk of it into a log file.
Disabling by default is scary simply because for short term UX we may risk long term stability.
I'm going to lock this issue because it has been closed for _30 days_ โณ. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Most helpful comment
Exactly the same problem here:
apply operation: Real and shadow states do not match!apply operation: Real and shadow diffs do not match!apply operation: 1:3: unknown variable accessed: var.vpc_idIs there a way to disable "experimental features" that are causing this error? It fills up my CI logs with a bunch of jumble I don't want.