Terraform: Track resource renaming through configuration

[BogdanSorlea]

Hey,

Although this idea might get refuted, I think it could ease the pain of refactoring quite a lot. Example...

Let's say I have a terraform-based setup with two instances (does not really matter how many) and some SGs created, all located in the same folder. Thus, let's say that my SG, whose "name" (terraform resource name) is integration, SG which is attached to both instances. Therefore, when specifying the SG ID to associate to the instances, we reference it by ${aws_security_group.integration.id}.

Now let's say we move the SG declaration to a separate module, security-groups, which is located outside the working directory and is referenced accordingly. Inside this module, we define the output

output "integration_sg_id" {
  value = "${aws_security_group.integration.id}"
}

Because of this change, the SG ID associated to the instances needs to change to ${module.security-groups.integration_sg_id}. Now, assuming that before the refactoring we have actually spawned these resources, just applying the new manifest will result in errors, requiring to destroy and re-create the stack.

For "large-scale" deployments, destroying and recreating is not always an option, so I am suggesting to somehow allow the defining of some resource equivalence maps, i.e. a file where you could say something like

resource "equivalence" "integration-refactoring-20160926" {
  this = "aws.security-groups.integration"
  that = "module.security-groups.integration_sg_id"
}

Of course, there would be something related to scope that would have to be figured out - and, of course, it is very much a bit of a hairy problem, but I can assure you that at least for our setup it is very crucial.

If applied, this mechanism should probably be envisioned as a database incremental migration routine: the state file should keep track of all the equivalence definitions that have already been applied, add a new equivalence resource to the state file after it has been successfully applied and update the structure/values of/in the state file to reflect the post-apply state.

Please let me know if there's anything that is not very clear.

[apparentlymart]

Hi @BogdanSorlea! Thanks for this feature request.

Do you think you could achieve some of what you're asking here using the terraform state mv command? This doesn't allow the resource to appear under two different names, but it _is_ intended to allow you to easily rename a resource or move it into a module without recreating it.

The workflow in this case would be:

• Move the security group _config_ into the security-groups module and define `output "integration_sg_id" there.
• Add the module "security-groups" definition to the existing project and change interpolations to "${module.security-groups.integration_sg_id}".
• Run terraform state mv aws_security_group.integration module.security-groups.aws_security_group.integration to move the state about the existing resource into the module.
• Run terraform plan and see that no changes are required, because the config and the state are already matching.

The main difference between this and what you are describing, I think, is that using terraform state mv requires that you complete the refactoring all in a single operation, whereas your approach could potentially allow moving the resource but deferring the update of all of the referencing interpolations until some later time.

[BogdanSorlea]

Hey, thanks, stupid of me, I wasn't aware of the state move command, I will try it out and let you know.

[afeld]

state mv works well, but only if you're running apply manually; there's not a good way to work it into deployment pipelines. Does feel like there could be some way to help with migrations of resources between modules, and being able to specify an equivalence/alias would be a way to do that.

[jamesarosen]

Run terraform state mv aws_security_group.integration module.security-groups.aws_security_group.integration to move the state about the existing resource into the module.
Run terraform plan and see that no changes are required, because the config and the state are already matching.

The problem with this flow is that I would have to run the mv before opening a pull request on my infrastructure repo. That means changing global state (the terraform remote's state) before code review.

For a flow that involves review by a second engineer, I think it would be very useful to have something like

terraform plan -with=name.before:name.after

[mcintyre321]

I think this is a really important feature. Refactoring the terraform resource names is a pretty common things, and it really needs to work with CI and source control.

Other approaches might be:

resource "something" "newname" {
    previous_names: ["oldname"]
}

Terraform should then upgrade any oldnames to the newname.

Another option could be for terraform to attempt to detect renames, by having the developer mark an output variable as a 'resource identifier'. e.g. suppose you have a resource that creates an s3 bucket with a specific name and path. During plan time, terraform could look for any deleted resources with that arn, and if they match that of one being created, it can treat that as a renamed resource

[jlsjonas]

Just found this ticket after a 2nd search through open issues, it looks like this one even predates https://github.com/hashicorp/terraform/pull/18347 where it was said that state mv shouldn't be used, which contradicts what was suggested as a workaround here.

Can we get consensus between the contributors or in some other way make a _battle plan_ so that the community can contribute to a renaming feature?

[afeld]

Thought I had posted it here before, but I started tfmv a couple years ago to try and address this problem. Haven't gotten back to it, but the problem just bit me again. Would love some help on it if there's interest. aidan.[email protected]

[philomory]

I really, _really_ wish Terraform had the second feature @mcintyre321 suggests, where specific attributes of a resource could be flagged as a resource identifier, similar in some ways to a Puppet "namevar", but potentially specified on a per-resource basis by the user. This would not only help with refactoring, it would also potentially allow for automatic importing of existing resources without resorting to terraform import.

[jlsjonas]

it would also potentially allow for automatic importing of existing resources without resorting to terraform import.

While I'd really like the feature, and allowing to define an identifier would certainly make this more manageable for moving.
For importing however, I'd argue that this would have to be an explicit action (an extra "these resources were already identified , do you want to import them?" question before/after confirming applying, CLI flag when non-interactive) to avoid importing a naming collision, with the resource ending up in 2 TF states

[philomory]

@jlsjonas For sure, I do agree there should be a confirmation before auto-importing; I only mean "auto" in the sense that you don't have to type potentially hundreds of terraform import statements manually.

[apparentlymart]

Hi all,

Making it so that all side-effects are made using the standard terraform plan and terraform apply workflow, rather than various secondary commands, is a long-term design goal for Terraform, but unfortunately it's also something that we've identified as being too big a piece of work to happen prior to stabilizing Terraform as 1.0, which is the goal all of our current work is heading towards.

With that said, I can acknowledge that this is something we intend to do eventually but I also want to be clear with you all that it is not going to happen in the near future. I can't predict far into the future with any certainty, but I strongly suspect that it will be _at least_ a year before we'd be able to begin design and implementation work for this huge shift in Terraform's current model of plans, because work to stabilize Terraform's current featureset is taking priority. We don't intend to accept community contributions for the feature because the changes required would be too disruptive to Terraform's internals to happen concurrently with the stabilization work.

I understand that this is annoying, and I want to have this new workflow at least as much as you all do, but I also want to be honest about the current situation in case it inspires any of you to seek third-party external solutions in the meantime.

[mwarkentin]

I've come across https://github.com/minamijoyo/tfmigrate

Haven't used it personally but maybe helpful for some people in this thread.