Terraform: Remote state not created when using modules.

Created on 18 Aug 2017 · 43Comments · Source: hashicorp/terraform

I have recently jumped back to modules with Terraform latest version. I am creating a 2-tier infrastructure using aws instances, autoscaling groups and vpc resources. This is the layout:

├───autoscaling
├───instance
├───remote_state
│   └───vpc
└───vpc

Each folder contains a main.tf and vars.tf, with the exception of vpc that also contains an outputs.tf file with the resources I need for autoscaling and instance templates. My remote config in vpc script looks like this:

terraform {
  backend "local" {
    path = "../remote_state/vpc/terraform.tfstate"
  }
}
...

And the remote state statement in the instance and autoscaling modules is as follows:

data "terraform_remote_state" "vpc" {
  backend = "local"
  config {
      path = "../remote_state/vpc/terraform.tfstate" 
  }
}

I have tested this script alone and it plans, applies correctly and create the desired remote state file I need for my other modules. However, when I load it as module and plan with the rest of the resources I get errors like the following:

>terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.network: Refreshing state...
data.terraform_remote_state.vpc: Refreshing state...
Error running plan: 2 error(s) occurred:

* module.autoscaling.aws_security_group.web: 1 error(s) occurred:

* module.autoscaling.aws_security_group.web: Resource 'data.terraform_remote_state.vpc' does not have attribute 'vpc_id' for variable 'data.terraform_remote_state.vpc.vpc_id'
* module.instance.aws_instance.module_instance: 1 error(s) occurred:

* module.instance.aws_instance.module_instance: Resource 'data.terraform_remote_state.network' does not have attribute 'subnet_id' for variable 'data.terraform_remote_state.network.subnet_id'

This is my main script that contains all three modules:

provider "aws" {
  region = "${var.aws_region}"
}

module "network" {
  source = "./vpc/"

  cidr = "${var.cidr}"
  subnet_cidr = "${var.subnet_cidr}"
}

module "instance" {
  source = "./instance/"

  ami_id = "${var.ami_id}"
  size = "${var.size}"
  logging_key = "${var.logging_key}"
}

module "autoscaling" {
  source = "./autoscaling"

  image_id = "${var.image_id}"
  inst_size = "${var.inst_size}"
  ssh_key = "${var.ssh_key}"
  ssh_port = "${var.ssh_port}"
  http_port ="${var.http_port}"
  min_size = "${var.min_size}"
  max_size ="${var.max_size}"
}

I have tested creating only my network module too. In this case, no remote state gets created and I still get the above errors when I use the other 2 modules. I also tested commenting the parameters that use the remote state data and confirmed I can plan and apply with no errors. I appreciate any help or guide.

Terraform Version

v0.10.1
v0.10.2

Affected resources

remote state file

Expected behavior

Plan and apply the module script.

Actual behavior

Terraform does not create a remote state and throws errors when planning and applying the module script.

bug core

Source

HebertCL

👍1

Most helpful comment

I'm running into the same thing with the latest terraform. My code looks like this

data "terraform_remote_state" "core" {
  backend = "s3"

  config {
    bucket = "m-devops-terraform"
    key    = "core-common/terraform.tfstate"
    region = "us-east-1"
  }
}

module "metadata" {
  source = "../modules/metadata/"

  tag_environment  = "${data.terraform_remote_state.core.tag_environment}"
  tag_maintainer   = "${data.terraform_remote_state.core.tag_maintainer}"
  tag_application  = "${data.terraform_remote_state.core.tag_application}"
  tag_run_always   = "true"
  tag_department   = "${data.terraform_remote_state.core.tag_department}"
  tag_project_name = "${data.terraform_remote_state.core.tag_project_name}"
}

module "bastion" {
  source           = "../modules/bastion"
  region           = "${data.terraform_remote_state.core.region}"
  security_groups  = "${data.terraform_remote_state.core.external_ssh},${data.terraform_remote_state.core.internal_ssh}"
  vpc_id           = "${data.terraform_remote_state.core.vpc_id}"
  key_name         = "${data.terraform_remote_state.core.key_name}"
  subnet_id        = "${data.terraform_remote_state.core.external_subnets.0.id}"
  tag_environment  = "${module.metadata.tag_environment}"
  tag_maintainer   = "${module.metadata.tag_maintainer}"
  tag_application  = "${module.metadata.tag_application}"
  tag_run_always   = "true"
  tag_department   = "${module.metadata.tag_department}"
  tag_project_name = "${module.metadata.tag_project_name}"
  name             = "${module.metadata.tag_environment}-ssh"
}

When I look at the state for my core in s3, the outputs are all there.

{
  "version": 3,
  "terraform_version": "0.10.2",
  "serial": 1,
  "lineage": "******************",
  "modules": [
    {
      "path": [
        "root"
      ],
      "outputs": {
        "availability_zones": {
          "sensitive": false,
          "type": "list",
          "value": [
            "us-east-2a",
            "us-east-2b",
            "us-east-2c"
          ]
        },
        "domain_name": {
          "sensitive": false,
          "type": "string",
          "value": "core-net.local"
        },
        "environment": {
          "sensitive": false,
          "type": "string",
          "value": "staging"
        },
        "external_elb": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "external_route_tables": {
          "sensitive": false,
          "type": "string",
          "value": "rtb-******************"
        },
        "external_ssh": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "external_subnets": {
          "sensitive": false,
          "type": "list",
          "value": [
            "subnet-******************",
            "subnet-******************",
            "subnet-******************"
          ]
        },
        "internal_elb": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "internal_route_tables": {
          "sensitive": false,
          "type": "string",
          "value": "rtb-******************,rtb-******************,rtb-******************"
        },
        "internal_ssh": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "internal_subnets": {
          "sensitive": false,
          "type": "list",
          "value": [
            "subnet-******************",
            "subnet-******************",
            "subnet-******************"
          ]
        },
        "key_name": {
          "sensitive": false,
          "type": "string",
          "value": "devo******************"
        },
        "log_bucket_id": {
          "sensitive": false,
          "type": "string",
          "value": "******************-core-network-test-staging-logs"
        },
        "region": {
          "sensitive": false,
          "type": "string",
          "value": "us-east-2"
        },
        "tag_application": {
          "sensitive": false,
          "type": "string",
          "value": "core-network"
        },
        "tag_department": {
          "sensitive": false,
          "type": "string",
          "value": "operations"
        },
        "tag_environment": {
          "sensitive": false,
          "type": "string",
          "value": "staging"
        },
        "tag_maintainer": {
          "sensitive": false,
          "type": "string",
          "value": "devops@******************.com"
        },
        "tag_project_name": {
          "sensitive": false,
          "type": "string",
          "value": "d******************-core"
        },
        "vpc_id": {
          "sensitive": false,
          "type": "string",
          "value": "vpc-******************"
        },
        "vpc_security_group": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "zone_id": {
          "sensitive": false,
          "type": "string",
          "value": "Z3******************"
        }
      },
      "resources": {
        "data.aws_caller_identity.current": {
          "type": "aws_caller_identity",
          "depends_on": [],
          "primary": {
            "id": "2017-08-18 20:54:45.466849791 +0000 UTC",
            "attributes": {
              "account_id": "******************",
              "arn": "arn:aws:iam::******************3:user/******************.com",
              "id": "2017-08-18 20:54:45.466849791 +0000 UTC",
              "user_id": "AIDAILJPLKR24UNC7CDIK"
            },
            "meta": {},
            "tainted": false
          },
          "deposed": [],
          "provider": ""
        }
      },
      "depends_on": []
    },
    {
      "path": [
        "root",
        "defaults"
      ],
      "outputs": {
        "domain_name_servers": {
          "sensitive": false,
          "type": "string",
          "value": "10.30.0.2"
        }
      },
      "resources": {},
      "depends_on": []
    },

Some reason when calling them from the remote state it can't find them. My output for the "core" remote data looks like below.What am I doing in correctly?

```output.tf
// The region in which the infra lives.
output "region" {
value = "${var.region}"
}

// The internal route53 zone ID.
output "zone_id" {
value = "${module.dns.zone_id}"
}

// Security group for internal ELBs.
output "internal_elb" {
value = "${module.security_groups.internal_elb}"
}

// Security group for external ELBs.
output "external_elb" {
value = "${module.security_groups.external_elb}"
}

// Security group for internal ELBs.
output "internal_ssh" {
value = "${module.security_groups.internal_ssh}"
}

// Security group for external ELBs.
output "external_ssh" {
value = "${module.security_groups.external_ssh}"
}

// Comma separated list of internal subnet IDs.
output "internal_subnets" {
value = "${module.vpc.internal_subnets}"
}

// Comma separated list of external subnet IDs.
output "external_subnets" {
value = "${module.vpc.external_subnets}"
}

// S3 bucket ID for ELB logs.
output "log_bucket_id" {
value = "${module.s3_logs.id}"
}

// The internal domain name, e.g "stack.local".
output "domain_name" {
value = "${module.dns.name}"
}

// The environment of the stack, e.g "prod".
output "environment" {
value = "${var.environment}"
}

// The VPC availability zones.
output "availability_zones" {
value = "${module.vpc.availability_zones}"
}

// The VPC security group ID.
output "vpc_security_group" {
value = "${module.vpc.security_group}"
}

// The VPC ID.
output "vpc_id" {
value = "${module.vpc.id}"
}

// Comma separated list of internal route table IDs.
output "internal_route_tables" {
value = "${module.vpc.internal_rtb_id}"
}

// The external route table ID.
output "external_route_tables" {
value = "${module.vpc.external_rtb_id}"
}

output "tag_environment" {
value = "${module.metadata.tag_environment}"
}

output "tag_department" {
value = "${module.metadata.tag_department}"
}

output "tag_application" {
value = "${module.metadata.tag_application}"
}

output "tag_maintainer" {
value = "${module.metadata.tag_maintainer}"
}

output "tag_project_name" {
value = "${module.metadata.tag_project_name}"
}

output "key_name" {
value = "${var.key_name}"
}
```

moos3 on 21 Aug 2017

👍4

All 43 comments

Hi @HebertCL! Sorry this isn't working properly.

Thanks for the detailed reproduction steps. Agreed that this is some odd behavior. Hopefully we can duplicate your result here and figure out what's going awry.

apparentlymart on 18 Aug 2017

@HebertCL Reading over your (very detailed) description, it is not clear to me if you ran terraform apply on your VPC configuration before running the dependant configuration.

Just wanted to be clear, the terraform remote state data provider is a read-only mechanism, and does not cause terraform to apply the remote configuration. I've seen this as a common misunderstanding in the community.

It's also worth noting that the remote state data provider reads only outputs of the root module in the remote resources. So if you have a module and you want to access one of it's outputs using the data provider, you'll need to add another output block to the root module that references the output of the submodule. See root outputs only from the provider docs.

nbering on 18 Aug 2017

@nbering I did tried applying only the network module prior applying the other two. It applied correctly but it didn't generated a terraform.tfstate file under my remote_state directory.

The plan output I pasted above was just to show the errors I get when I try to plan or apply even if the network module was previously created.

Thanks for the outputs root outputs reference, this is something new for me that I need to take a look at.

HebertCL on 18 Aug 2017

I'm running into the same thing with the latest terraform. My code looks like this

data "terraform_remote_state" "core" {
  backend = "s3"

  config {
    bucket = "m-devops-terraform"
    key    = "core-common/terraform.tfstate"
    region = "us-east-1"
  }
}

module "metadata" {
  source = "../modules/metadata/"

  tag_environment  = "${data.terraform_remote_state.core.tag_environment}"
  tag_maintainer   = "${data.terraform_remote_state.core.tag_maintainer}"
  tag_application  = "${data.terraform_remote_state.core.tag_application}"
  tag_run_always   = "true"
  tag_department   = "${data.terraform_remote_state.core.tag_department}"
  tag_project_name = "${data.terraform_remote_state.core.tag_project_name}"
}

module "bastion" {
  source           = "../modules/bastion"
  region           = "${data.terraform_remote_state.core.region}"
  security_groups  = "${data.terraform_remote_state.core.external_ssh},${data.terraform_remote_state.core.internal_ssh}"
  vpc_id           = "${data.terraform_remote_state.core.vpc_id}"
  key_name         = "${data.terraform_remote_state.core.key_name}"
  subnet_id        = "${data.terraform_remote_state.core.external_subnets.0.id}"
  tag_environment  = "${module.metadata.tag_environment}"
  tag_maintainer   = "${module.metadata.tag_maintainer}"
  tag_application  = "${module.metadata.tag_application}"
  tag_run_always   = "true"
  tag_department   = "${module.metadata.tag_department}"
  tag_project_name = "${module.metadata.tag_project_name}"
  name             = "${module.metadata.tag_environment}-ssh"
}

When I look at the state for my core in s3, the outputs are all there.

{
  "version": 3,
  "terraform_version": "0.10.2",
  "serial": 1,
  "lineage": "******************",
  "modules": [
    {
      "path": [
        "root"
      ],
      "outputs": {
        "availability_zones": {
          "sensitive": false,
          "type": "list",
          "value": [
            "us-east-2a",
            "us-east-2b",
            "us-east-2c"
          ]
        },
        "domain_name": {
          "sensitive": false,
          "type": "string",
          "value": "core-net.local"
        },
        "environment": {
          "sensitive": false,
          "type": "string",
          "value": "staging"
        },
        "external_elb": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "external_route_tables": {
          "sensitive": false,
          "type": "string",
          "value": "rtb-******************"
        },
        "external_ssh": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "external_subnets": {
          "sensitive": false,
          "type": "list",
          "value": [
            "subnet-******************",
            "subnet-******************",
            "subnet-******************"
          ]
        },
        "internal_elb": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "internal_route_tables": {
          "sensitive": false,
          "type": "string",
          "value": "rtb-******************,rtb-******************,rtb-******************"
        },
        "internal_ssh": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "internal_subnets": {
          "sensitive": false,
          "type": "list",
          "value": [
            "subnet-******************",
            "subnet-******************",
            "subnet-******************"
          ]
        },
        "key_name": {
          "sensitive": false,
          "type": "string",
          "value": "devo******************"
        },
        "log_bucket_id": {
          "sensitive": false,
          "type": "string",
          "value": "******************-core-network-test-staging-logs"
        },
        "region": {
          "sensitive": false,
          "type": "string",
          "value": "us-east-2"
        },
        "tag_application": {
          "sensitive": false,
          "type": "string",
          "value": "core-network"
        },
        "tag_department": {
          "sensitive": false,
          "type": "string",
          "value": "operations"
        },
        "tag_environment": {
          "sensitive": false,
          "type": "string",
          "value": "staging"
        },
        "tag_maintainer": {
          "sensitive": false,
          "type": "string",
          "value": "devops@******************.com"
        },
        "tag_project_name": {
          "sensitive": false,
          "type": "string",
          "value": "d******************-core"
        },
        "vpc_id": {
          "sensitive": false,
          "type": "string",
          "value": "vpc-******************"
        },
        "vpc_security_group": {
          "sensitive": false,
          "type": "string",
          "value": "sg-******************"
        },
        "zone_id": {
          "sensitive": false,
          "type": "string",
          "value": "Z3******************"
        }
      },
      "resources": {
        "data.aws_caller_identity.current": {
          "type": "aws_caller_identity",
          "depends_on": [],
          "primary": {
            "id": "2017-08-18 20:54:45.466849791 +0000 UTC",
            "attributes": {
              "account_id": "******************",
              "arn": "arn:aws:iam::******************3:user/******************.com",
              "id": "2017-08-18 20:54:45.466849791 +0000 UTC",
              "user_id": "AIDAILJPLKR24UNC7CDIK"
            },
            "meta": {},
            "tainted": false
          },
          "deposed": [],
          "provider": ""
        }
      },
      "depends_on": []
    },
    {
      "path": [
        "root",
        "defaults"
      ],
      "outputs": {
        "domain_name_servers": {
          "sensitive": false,
          "type": "string",
          "value": "10.30.0.2"
        }
      },
      "resources": {},
      "depends_on": []
    },