Terraform: Import resources into Terraform

[piavlo]

Use case , manage infrastructure environments both existing and created from scratch, by same terraform configs. For example in development environments we want to create everything from scratch and destroy everything then we finish working with it. In production we want to be able to add new resources to it with terraform but not conflict with resources that already exist but not managed by terrafrom yet.

For ex:

resource "aws_subnet" "b" {
    vpc_id = "${aws_vpc.terraform.id}"
    cidr_block = "${var.vpc_network_prefix}.${lookup(var.vpc_subnet_suffix, "b")}"
    map_public_ip_on_launch = true
    availability_zone = "${var.region}b"
    if defined(var.subnet_b_id) { load_by_id(${var.subnet_b_id}) }

}

if terraform has not yet created this resource it will check if subnet_b_id is defined and if resource with such id exists in provider api it will import this resource into terraform based on data in api instead of creating it.

[sparkprime]

I think this doesn't need language support, instead you just need the ability to import the existing subnet into the tfstate file (and give it the name "b" in the process). Then Terraform would reuse the existing one instead of creating a new one to fill the "b" slot. I.e. that conditional if defined(...) essentially already exists in the core terraform planning logic to check if a resource already exists in the tfstate.

[nogweii]

Would the recommended way forward for this be then to create a script that traverses an AWS network, getting all Route53 records, VPC information, instance data, etc, and generating the appropriate tfstate JSON + example .tf files?

[armon]

Correct. Ideally the tool would be able to scan resources on various provides and with operator assistance map it to config logical names.

[carimura]

+1 to this one for us. Keeps us from being able to launch new clusters of our service into an existing VPC. Any update on this one since Dec?

[phinze]

No specific update on the feature itself, it's still on the roadmap.

But I wanted to jump in here to link an excellent article by @phrawzty on a strategy for dealing with existing resources with current versions of terraform:

https://www.dark.ca/2015/01/27/handling-extant-resources-in-terraform/

[clstokes]

+1, would love to see this (soon).

[AlexanderEkdahl]

An alternative approach could be something inspired by terraform taint where the user could execute something along the lines of terraform add [options] name id. For example terraform add aws_security_group.name sg-4bdc302e and it would be marked as existing but without attributes in the .tfstate-file. Upon the next refresh or apply the attributes would be populated.

[johnrengelman]

I like @AlexanderEkdahl suggestion. This would be awesome and we could keep people out of the state file - @mitchellh @catsby, thoughts? I'd be willing to do a POC for something like this.

[mitchellh]

Thats a pretty neat idea. Could help basic stuff... What do you think @phinze?

[phinze]

Agreed - I think that would be a really valuable baby step to take. :footprints: POC away @johnrengelman!

[johnrengelman]

Cool.

[johnrengelman]

I put together a very simple POC for discussion - https://github.com/hashicorp/terraform/pull/2022

[blalor]

Separate from @johnrengelman's POC (where I commented, too), I feel like a missing part of this conversation is actually generating the Terraform config. I'd expect it to still be a relatively manual, iterative process, but I'd want terraform import <resource_id> <provider_id> to generate config appropriate for a .tf file, probably with a helper for the various _association resource types. Even outputting JSON a la .tfjson would be a huge help.

[ketzacoatl]

+1 to @blalor's comment, though I would advocate for taking this in separate.. the first being coverage for inport to begin with.. generating config has more nuances and work in general.

[meylor]

Others may find this tool helpful. I tested the EC2, ELB, security group, and VPC capabilities and it was able to extract existing AWS infrastructure into resources. https://github.com/dtan4/terraforming

[jwaldrip]

:+1: I would love to see this incorporated into terraform.

[nrcxcia]

Would love to see this as well, this functionality would allow us to start using terraform at my company to manage existing infrastructure. +1

[ckelner]

:+1:

[danielcbright]

:+1:

[smcavoy-travelbird]

I'll join the party. This would be a _huge_ feature.
(AWS first please)

[leopepe]

:+1:

[mikerahmati]

I'd like to see this incorporated into terraform too :+1:

[antonbabenko]

yes, yes, yes :+1: I would love to see this incorporated into terraform also, because it should have wider community support.

@dtan4 great work!

[JesperTerkelsen]

:+1:

[br0ch0n]

:+1: I'd think the terraform equivalent of puppet's 'resource' command would be good: https://docs.puppetlabs.com/references/3.8.latest/man/resource.html

[Deserved]

:+1: It would help a lot!

[JacobASeverson]

:+1:

[zstarer]

:+1: for sure.

[josephholsten]

could folks giving a :+1: try out the patch in #2022 or #3345? Those implementations seem to be the way forward.

[realloc]

:+1:

[preflightsiren]

+1 do you think this would cover a similar idea I've had to allow for stub resources / read-only resources? essentially it would allow you define a resource that already exists (eg. vpc/subnets), to allow you to reference them in aws_instances for example, but without having terraform manage the resources and try to destroy/recreate them?

[ross]

+1 do you think this would cover a similar idea I've had to allow for stub resources / read-only resources? essentially it would allow you define a resource that already exists (eg. vpc/subnets), to allow you to reference them in aws_instances for example, but without having terraform manage the resources and try to destroy/recreate them?

Not as it stands, but something like it/built upon it could potentially allow that. I'd assume you'd need some sort of read-only meta property that would say you can refresh this thing, but don't ever try to change it. If you had that or something like it this would allow you to get it in to your state file initially. That said it's definitely out of scope for what this PR is trying to do.

[alfredopalhares]

:+1:

[kha0S]

A way to import existing resources would be awesome :+1:

[rdark]

@kha0S - I've had some luck with https://github.com/dtan4/terraforming

[kha0S]

@rdark Thanks. Will give it a try :+1:

[aldarund]

+1

[rokka-n]

:+1:

[JigarS91]

I am trying to create new subnets in existing vpc.
I have added vpc id in variables.tf file as

variable "aws_vpc" {
default = "vpc id"
}

Getting below error:-
aws_subnet.private: Error creating subnet: InvalidVpcID.NotFound: The vpc ID "vpc id not found"
Not sure how to go ahead with this one. Need help !

[preflightsiren]

@JigarS91 https://www.terraform.io/community.html is the place to ask these questions :)

[rizvfe00]

:+1:

[avdhoot]

+1 it will be good to have feature in terraform.

[cbroglie]

terraforming is working OK for me too. My process is essentially to export all resources into tfstate form, then rename the ones I'm importing, and delete the others. Would be great to have this functionality built into terraform.

[allandrick]

This is a bit of a show stopper for my application - I don't want to edit the tfstate if I can avoid it... any update on whether this will be moving forward in one form or another?

[Pryz]

:+1:

[robertwe]

:+1:

[TerribleDev]

:+1:

[LusciousPear]

+1, this makes it much easier to recover from partial applies

[cyakimov]

👍

[davidfic]

👍

[jvasallo]

👍

[nitdana]

+1

[manojlds]

Folks, github has had reactions for a while. Please stop the +1 and thumbs up.

[brandon-dacrib]

seconded on a request for an update, clearly this is something that the community wants. If the response is that hey this will be included in version 0.6.20 then so be it.

[johnrengelman]

Looking at the work on the master branch, parts of this (or maybe all) will be in 0.7.

[mitchellh]

The basics of this are in master and will be part of 0.7. See the website subfolder for docs until it is released. More fancy features such as config generation will come, but 0.7 will be able to import resources into Terraform.

This is just the beginning, but we're finally getting there!

[ozbillwang]

Anyone interesting the converter, you need to follow up this ticket:

https://github.com/hashicorp/hcl/issues/162

[sergeylanzman]

You can try new solution https://github.com/GoogleCloudPlatform/terraformer
Full import existing resources, include tf and tfstate. Full support cloud providers options
Support Google cloud and aws

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.