Atlantis: Feature Request: Support Resource Import

[tedwardd]

Terraform support importing existing resources in to the TF state file by way of the terraform import command. Typical usage looks like this:

1. Create resource that represents existing resource.
2. Import Resource
3. Plan to compare resource definition to reality
4. Make changes as necessary and plan until plan shows now changes (or desired changes
5. Apply changes

[lkysow]

Hi Ted, thanks for the ticket! So the workflow with Atlantis would be something like:

1. Create pull request with resource in terraform file
2. Autoplan would say that it's creating the resource
3. Run atlantis import -w workspace -d dir aws_instance.example i-abcd1234
4. Run atlantis plan -w workspace -d dir to regen the plan
5. Plan should have no changes
6. Merge pull request

How does that sound?

[tedwardd]

Hey Luke,

Yeah, that sounds correct. We are still running v0.3.8 in production so I did not account for the autoplan in my original workflow but I believe it would be fine. Autoplan would show that it was creating a new resource but once atlantis import is run, the resource file is mapped to the target resource ID and subsequent runs of atlantis plan would show no changes needing to be made unless something in the resource definition differs from what you're attempting to import.

[rhughes1]

Is this on the roadmap for 0.5.x?

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

[majormoses]

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

What I do currently is create a PR with a no-op change (such as add a comment) which with auto plan enabled will create the lock making it safe to do imports from the CLI locally without worrying that someone will be able to run a plan/apply through atlantis. After I have done the imports I just do another plan and then it will be business as usual.

[rhughes1]

Interesting idea to do for the interim.

Just thinking from an auditing perspective it would be nice to see what was/wasn't imported.

[lkysow]

Is this on the roadmap for 0.5.x?

No, server-side config is my focus right now.

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

How would Atlantis know that you were going to do a resource import ahead of time?

[majormoses]

Just thinking from an auditing perspective it would be nice to see what was/wasn't imported.

If you are using a backend such as s3 and enable versioning you should be able to determine who pushed the latest version and you can diff the versions to see the delta. But ya I really would prefer to have atlantis be able to do it for me so that I can say github pull requests are my audit log. Now it's technically both.

[majormoses]

Yeah, that sounds correct. We are still running v0.3.8 in production so I did not account for the autoplan in my original workflow but I believe it would be fine. Autoplan would show that it was creating a new resource but once atlantis import is run, the resource file is mapped to the target resource ID and subsequent runs of atlantis plan would show no changes needing to be made unless something in the resource definition differs from what you're attempting to import.

If you open you pull request it starts planning, you run an import (locally or via atlantis git(hub|lab) comment) but then you have a lock not just at the atlantis level you have various types of terraform locks. For example terraform always creates its own lock in the form of a serial number if you try to apply a plan from a previous serial terraform will refuse to apply the plan[1] as there is no way to know if the plan is still valid (in this case it would not). Additionally if you use aws you can configure your backend to use a dynamodb locking table as well as an extra safeguard. I think you have auto plan enabled you need to run a plan at least once manually and would be safer to run it a second time to ensure that there is no delta before merging.

[tedwardd]

@majormoses Your comment made me re-evaluate my statement and I see that you're absolutely right about the plans and locks. I guess you'd have to have something where atlantis import triggers the locks and plans to clear, but that would clutter up the PR comments with a useless plan.

But this got me thinking and I realized this is not even the most interesting part of the problem. How would atlantis import know what resource IDs to map to the matching TF resource? Normally, you run terraform import <TF_project_resource> <project_id>. But if we're expecting the user to just run atlantis import and "magic happens", we would need some way to know which resources are intended for which ID's. Two thoughts on how to do that.

1. Command would be more akin to the TF equivalent atlantis import <tf_resource> <gcp_resource_id>
2. User could define mappings in a file within the project, similar to how I'd envision taints configured (see my comment in: #527).

I'm sure there are other ways as well. I, personally, would lean towards something that makes an effort to minimize the already clutter-prone PR comments, though.

[majormoses]

1. Command would be more akin to the TF equivalent atlantis import

Yes this is how I imagined the interface/experience.

User could define mappings in a file within the project, similar to how I'd envision taints configured (see my comment in: #527).

I will respond there as well but I think that the best interface would be to have it be a comment, committing a file to git to import/taint a resource/module seems like a bad idea. If someone forgets to clean it up then it could be a problem. I would vote taint follows the same pattern where you are explicit in what resource you want tainted.

personally, would lean towards something that makes an effort to minimize the already clutter-prone PR comments, though.

Ya I have thought about it but if we want github to be our audit trail I don't see around this. What I personally do in my org is minimize comments if they are not important or relevant. This leaves all the data intact and reduces the need to scroll a bunch of times.

[majormoses]

clutter up the PR comments with a useless plan

I don't know that I would call it useless it can be very helpful to see a plan before importing resources as it can help with recalling all the sub module objects that need to be imported, check for issues with variable/module/data references. It also helps paint a story (an audit log) of what things were like before you made any "manual" changes and then can be compared with a plan after the changes/import takes plan to confirm desired changes or a no-op. I can certainly see where you are coming from though, you can always hide or delete those comments if they really bug you.

[rohitshrivastava04]

can someone recommend a workaround this feature is developed? Is the only way we could achieve it by following what @majormoses suggests?

[thomas-orange]

@rohitshrivastava04 You can accomplish this with an atlantis custom command. I'm working on this now to create atlantis import {resource-identifier-tf} {resource-identifier-infra} which will simply pass the arguments to the terraform import command. You still have the auto-plan, but you won't need to run anything in CLI and you'll have an IaC audit trail in github.

This seems like a common enough use case that this could easily be made part of the core atlantis stack, but until then I'll proceed with my custom command.

[thomas-orange]

After looking closer, it appears that we can't setup custom commands for the github comment interface. The custom commands must run inside of a workflow as part of either init, plan, or apply. Looks like a simple workaround is out of the question on this, and this will need to be part of the official tool.