Atlantis: Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects

[mlehner616]

I have a repo that uses the default workspace but there are a number of different project folders.

Atlantis version: 0.8.3
Terraform version: v0.12.8

version: 3
projects:
  - name: qa
    dir: qa_acct/qa_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false
  - name: staging
    dir: prod_acct/staging_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false
  - name: prod
    dir: prod_acct/prod_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false

Plans are generated for all three projects as normal after commenting exactly atlantis plan.
Immediately afterword, commenting atlantis apply attempts to apply all three environments as expected. In this case, there was an apply error due to an AWS IAM policy being misconfigured and the plans were not successfully applied. A commit was pushed to fix this issue and another atlantis apply was submitted. Note, there was not another atlantis plan after the fix commit was pushed. Atlantis behaved as if it had forgotten about the failed plans and assumed they had been applied successfully when, in fact, they had not been. I believe the expected behavior should be to reject the apply since new commits were made and force another plan be run, correct?

The result was the following:

Ran Apply for 0 projects:

Automatically merging because all plans have been successfully applied.



md5-f414515a772ebb9ef0a9b5efa5f5dbab



Locks and plans deleted for the projects and workspaces modified in this pull request:

* dir: `prod_acct/prod_env` workspace: `default`
* dir: `prod_acct/staging_env` workspace: `default`
* dir: `qa_acct/qa_env` workspace: `default`

[lkysow]

Yeah it's a bug. If autoplan had been enabled then there would have been new plans generated and the apply wouldn't have worked.

[mlehner616]

@lkysow Thanks for the confirmation. This bug is killing us right now. We want people to be able see non-locking plans being run (in our normal CI pipleine) before Approvals are submitted so they can actually validate their code before blocking other development. If we wanted to dig into solving this, where would be a good place to start looking? I took a really quick glance through the repo and nothing jumped out at me.

Thank you for building this tool by the way, I really appreciate the work that went into this.

[lkysow]

After re-reading the ticket, this isn't technically a bug (although for your use-case it may as well be). Atlantis is just doing what you told it, it's up to you to run atlantis plan if you've pushed a new commit and don't have autoplan running. Coupled with automerge is the real issue here. If you didn't have automerge you'd quickly realize that you didn't re-run plan and there wouldn't be an issue.

Also if you were running with the -d or -p flags you'd get an error that "the plan doesn't exist for that project, please run plan". When we added the apply-all command (i.e. atlantis apply) we didn't replicate the behaviour. I'm not sure if it ever makes sense to not give an error in this case but I'd at least like to add a flag that lets you keep the old behaviour in case people were relying on it.

If we were to add some functionality to detect this case, it would be here: https://github.com/runatlantis/atlantis/blob/master/server/events/project_command_builder.go#L204 after Atlantis has found no pending plans. It could then exit with an error in this case.

I think a path forward may be:

• new flag --allow-no-plan-apply which defaults to false now (breaking change)
• thread that flag through and then check it at the line above

[mlehner616]

Well, I actually think your original interpretation made sense to me. To clarify, we would never want atlantis to apply without having the most up to date plan saved and locked.

What we’re doing instead is just running validate, fmt, lint andterraform plan --lock=false in vanilla gitlab CI. Devs open an MR and need to fix any issues there, and get all approvals first, before the atlantis plan. The problem we were solving by doing it this way was autoplan opening locks too early in the process and thus blocking other MRs that were ready to be applied.

I still think this is a bug. Yes I wanted autoplan disabled but that just means I want the developers to run it if and only if all the pre-apply requirements are met. I would expect the apply step to run the same validation that the plan is locked and up to date and apply based on that. Turning off autoplan shouldn’t affect those checks. What seems to be happening with autoplan disabled is the apply is ignoring the plans and ultimately just applies nothing.

I can confirm there are plans and locks are created when they are supposed to be. It appears that the atlantis apply step is just ignoring those if a second apply is run after this first one fails. Expected behavior would be for the apply step to either force a replan if the MR was updated, or attempt to re-apply the original plan. It’s doing neither of these right now.

[mlehner616]

One thing I did notice was that if the apply does fail, the saved plans are deleted but the locks are left open (this may be the actual bug here). If we removed those after a failed apply, that would basically force the plan step. I don’t know if that’s the best solution but I think it would work.

[kipkoan]

it's up to you to run atlantis plan if you've pushed a new commit and don't have autoplan running.

Our team has autoplan on, but pushing a new commit doesn't cause Atlantis to redo the plan (because Bitbucket).

One thing I did notice was that if the apply does fail, the saved plans are deleted but the locks are left open (this may be the actual bug here). If we removed those after a failed apply, that would basically force the plan step. I don’t know if that’s the best solution but I think it would work.

I agree we should either have Atlantis not delete the plans, or error if an apply is attempted without any plans.

@lkysow - what's the reason for Atlantis to delete the plans after a failed apply? It could have failed because a transient provider issue, and re-running apply on the same plan would later succeed.

[IamGabrielWu]

HI everyone I met this issue too.
Any work in progress to fix this bug ?
I removed the locks as mentioned above and redo "atlantis plan". It still shows "Ran Plan for 0 projects:"

[jamengual]

Running plan on the same PR after a failed to apply should not be any different than if atlantis does not delete the plan, it is just an extra step.

But if someone else in another PR modify the environment you are running plan against you will have a problem no matter what but by re-running a plan you could actually find the drift.

I do not think this is a bug, it is a bit annoying to run plan again but since terraform is idempotent it should only apply the difference.