Atlantis: Run plan/apply for multiple projects in parallel

[lkysow]

Via @psalaberria002, would like Atlantis to run plan and apply in parallel when running a command that needs to run on multiple projects. This would make things much faster.

[jolexa]

We have some repos that works on >10 AWS accounts (each account is a project in atlantis.yaml). When making a change to all the accounts, Atlantis is painfully show while it sequentially works on every account. We expect this usage to increase, by the way.

[majormoses]

I can certainly see the value in this but I think we need to be careful as to how this is introduced. While in the case of aws providers rate limits are more mitigated as _most_ of their api limits are gated at the account level and rate limit awareness is built in but this is not true of all (or even most) providers. For example we use the github provider to manage teams, repos, etc. We have something like 30+ states/projects and if we ran them all at once it would most certainly fail.

[mwarkentin]

Could you make this opt in? atlantis apply --parallel

I'm dealing with about 50 RDS instances right now, each in their own projects - and need to enable multi-az, change instance type, etc. in different steps. This is incredibly slow waiting for each to happen 1 at a time.

My other alternative would be to make 50 separate PRs to "fake" the parallelization but that's tricky to manage as well.

[majormoses]

To better understand the use cases where this is important would it be something that makes sense as adhoc on PRs, repo config, server config, or yes? I am leaning towards it be adhoc on PRs as needed or repo config.

[mwarkentin]

I’m not too familiar with the distinction between repo config and server config, but it seems it would make sense to have a way to set the default behavior as well as enable or disable on an adhoc basis within a PR.

[darrylb-github]

👍 - was just about to ask about this. I have a repo with 4 projects and currently waiting on atlantis to finish planning as it's running them sequentially.

[Fauzyy]

We've been running a simple repo config implementation of parallel plans for a couple weeks in our fork and it's working well: https://github.com/runatlantis/atlantis/compare/v0.8.3..segmentio:v0.8.3-segment0.2.1

Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.

Example config:

version: 3
automerge: true
parallel_plans: true
...

This is for plans only, I feel that making applies parallel could have unintended side-effects.

There's also a flag to set the server-side cap on the # of concurrent goroutines that run per-request via --parallel-plans-pool-size=5 (default of 10)

Would love to get this or something similar into upstream

[kennethtxytqw]

Will this be considered?

[lkysow]

Yeah that looks like a good solution since it avoids the problem of a lot of refactoring by only applying when using different workspaces. Segment's branch looks pretty diverged though so someone would need to make a clean PR.

[psalaberria002]

Does that mean we won't be able to parallelize it when not using workspaces?

[lkysow]

Does that mean we won't be able to parallelize it when not using workspaces?

Yes. But it's a first step. For not using workspaces the locking needs to be figured out so we don't modify a file-system that's got mutations in progress.

[Lowess]

@lkysow quick question as #926 was recently released under 0.13.0.

I am running Atlantis with a custom Terragrunt integration (really similar to the docs example

I wanted to leverage the newly introduced parallel features which is unfortunately not supported for Terragrunt workflows (yet) due to the workspace limitation.

I am wondering if I could _fool_ atlantis by setting different workspaces (even though in terragrunt you don't need them) to make atlantis run plans and applies in parallel for prod / stage configs.

Is this a bad idea or you guys think that could be a valid workaround ? Looking at the plan output from terragrunt it looks good to me but I was wondering if there is something else I should worry about ?

• Here is my Terragrunt layout:

.
├── modules
└── terragrunt
    ├── account-1
    │   ├── account.hcl
    │   └── virginia
    │       ├── prod
    │       ├── region.hcl
    │       └── stage
    └── terragrunt.hcl

• Here is a sample from my atlantis.yaml:

version: 3

parallel_plan: true
parallel_apply: true
automerge: true

projects:
  # Project Anchor
  - &terragrunt
    name: template
    dir: '.null'
    workflow: terragrunt
    autoplan:
      enabled: true
      when_modified:
        - "./terraform/modules/**/*.tf"
        - "**/*.tf"
        - "**/terragrunt.hcl"


  - <<: *terragrunt
    name: account-1-virginia-prod-app-A
    dir: ./terraform/terragrunt/account-1/virginia/prod/app-A
    workspace: prod

  - <<: *terragrunt
    name: account-1-virginia-stage-app-A
    dir: ./terraform/terragrunt/account-1/virginia/stage/app-A
    workspace: stage

  [....]

Thanks for your feedback !

[lkysow]

Hi @Lowess, the reason this only works for workspaces is because Atlantis clones the repo into a separate directory for each workspace so there is no contention. If you set different workspaces then Atlantis will clone into separate directories so it should work.

[angeloskaltsikis]

@Lowess i wonder whether you have tried using Terragrunt with Terraform workspaces and how this experiment went?
We are having multiple Terragrunt files and looking for ways on how we could make the {plan,apply} commands run on parallel as it may take atlantis a couple of hours to run a single command on 100s of Terragrunt leafs.

[ghostsquad]

Having the same problem, in that I have multiple projects in the same repo, all using different backends, etc. I'm not using any "workspaces" (Terraform definition of such), so everything is the "default" workspace. Parallel plans/apply don't work because it thinks that there's a shared lock in place. The first of 6 projects runs successfully, and the rest are reported that they cannot run because they are locked.

2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Running plans in parallel
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-1/us-west-2/prod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-2/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/prod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-1/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/tools/default"
2020/10/12 22:57:06+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "cd \"${PWD%\"${REPO_REL_DIR}\"}\" && jb install\n" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:06+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "jsonnet -J \"${PWD%\"${REPO_REL_DIR}\"}/vendor\" terrasonnet.tf.jsonnet > terrasonnet.tf.json\n" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:10+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 init -input=false -no-color -upgrade" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:10+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 workspace show" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:13+0000 [EROR] myorg/terraform-stuff#43: Running "/atlantis/data/bin/terraform0.13.4 plan -input=false -refresh -no-color -out \"/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod/account-1-prod-us-west-2-default.tfplan\"" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod": exit status 1
2020/10/12 22:58:14+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 plan -input=false -refresh -no-color" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 23:00:30+0000 [INFO] server: Parsed comment as command="unlock" verbose=false dir="" workspace="" project="" flags=""

[richstokes]

+1 it would be awesome if we could plan in parallel multiple projects, assuming they are all on different backends.

[Sebor]

I've faced with the same issue. I use terragrunt without workspaces and I'd like to plan\apply in parallel.
Does anybody know a workaround?

[mwarkentin]

Finally got a chance to try this out as we added a subset of projects which use Terraform Workspaces. It looks like I missed this caveat though:

Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.

I had assumed that it would allow us to plan the subset in parallel, and other projects would continue to function as before, however when we opened a PR with multiple (standard) projects each using the default workspace, ended up with The default workspace is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again. for all but one of the plans.

Would it be possible to support this use case? Maybe by supporting the parallel_[plan|apply] configuration at the project level instead of for the top level atlantis config?