Atlantis: Support Git Submodules

[osterman]

what

• Support git submodules

why

• It appears that clones are not recursive: https://github.com/runatlantis/atlantis/blob/f057d6d7d9a76c7b8d85c874a10c4f25f7be7465/server/events/working_dir.go#L113
• Support importing terraform root modules without needing terragrunt

use-case

• We're trying to adapt our deployment methodology to work well with Atlantis
• We keep all of our "root" modules in one repo called terraform-root-modules
• We then import these modules into various stages (aka environments or accounts)
• We have been using docker multi-stage, but that won't play well with atlantis detecting changes, so we're willing to compromise
• We want to use git submodules instead, with the expectation that atlantis would be able to detect changes for autoplans

---

@lkysow do you think this would work?

Technically, looks very easy to implement here and we'd be happy contribute the capability.

[lkysow]

Yeah that looks good! Please push a PR. Can you also look into if there are any unintended consequences?

[lkysow]

We should version detect git though and make sure we use the right command for the right version.

[tomer-1]

@osterman a more "backward compatible" approach to this would be:

1. "cd" into the repo
2. run "git submodules update --init --recursive"
   that'll do for all repos, even ones not containing any submodules.
   but this operation should be coupled with any "git clone" operation.
   i'm not aware of the way atlantis manages the repositories after the first clone.
   but if it's doing any git manipulations to prevent re-clones, then those manipulations should contain the steps presented above to achieve an up to date working copy.

[osterman]

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

[tomer-1]

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

could you please elaborate how did you manage implement this?
we have many environments and many repositories, and there are some things that implementing as a submodules is the best way to go

[osterman]

@tomer-1 we ended up having to compromise. Our change detects if a submodule folder is modified and then executes. We were not able to detect changes inside the submodule due to the current implementation relying on the files changed as returned by the github API and not the files changed using git calculus.

[osterman]

@aknysh can add more details

[tomer-1]

@osterman i understand you had to compromise for the changes in the modules,
what i don't fully understand is how did you manage to make atlantis sync the submodules after the clone?

[osterman]

@tomer-1 did you have a chance to look at the PRs where we implement it?

https://github.com/cloudposse/atlantis/pull/11
https://github.com/cloudposse/atlantis/pull/18
https://github.com/cloudposse/atlantis/pull/19

@aknysh can answer any specific questions about these.

[krishnanandchoudhary]

@osterman HI, I am exploring atlantis to use, While the repo has submodules for terraform vars file, it does not checkout submodules. Wondering how to make it work. Thanks

[goobysnack]

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

[lkysow]

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

Does this work? @osterman was saying that that's not enough because the github api doesn't return the list of files changed in the submodule so Atlantis won't know that the files in the submodule have changed and it won't trigger a plan.

[goobysnack]

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

[lkysow]

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

No, you can't add extra args to the git clone command.

[goobysnack]

@lkysow I figured this one out too. I added this to my plan workflow:

- run: git submodule update --init --recursive && ...<removed>

[dcatalano-figure]

another submodule usecase to support is forcing a plan b/c of symlinks ... we have common terraform code IE. tfvars, in a directory that also happens to be a git submodule. We are able to check out the submodule with a custom workflow as described above. There are files in the submodule that are shared / accesses via a symlink in each respective directory. Of course the symlink doesn't change so there is nothing to trigger the plan.

I'd be really awesome to be able to force a plan via GitHub regardless if Atlantis thinks there is a change or not.