Atlantis: Autoplan issue with terraform 0.12 and terragrunt 0.19

[milpog]

I am trying to use atlantis 0.8.3 with terraform 0.12.6 and terragrunt 0.19.16 and I am facing an issue with autoplan functionality.

Terragrunt from version 0.19.x is using terragrunt.hcl files instead of terraform.tfvars. After migration nothing is auto planned as it was previously.

I was even trying to use atlantis.yaml but apparently I cannot figure out how to set it up properly. I tried following configuration but I am still receiving information that no project was suitable for plan.

First approach

version: 3
projects:
  - dir: dev
    autoplan:
      when_modified: ["*.hcl"]

Logs

2019/08/06 19:17:35+0000 [INFO] middleware.go:42 server: POST /events – respond HTTP 200
2019/08/06 19:17:35+0000 [DBUG] project_command_builder.go:102 milpog/infra-live#129: Got workspace lock
2019/08/06 19:17:36+0000 [DBUG] project_command_builder.go:110 milpog/infra-live#129: 3 files were modified in this pull request
2019/08/06 19:17:36+0000 [DBUG] working_dir.go:79 milpog/infra-live#129: Clone directory "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default" already exists, checking if it's at the right commit
2019/08/06 19:17:36+0000 [DBUG] working_dir.go:104 milpog/infra-live#129: Repo was already cloned but is not at correct commit, wanted "ca69b0872f42869f4283ae06a655ec1495de90f5" got "813615c07db85c2d38e6cd786b2d05efb8c98133"
2019/08/06 19:17:36+0000 [INFO] working_dir.go:123 milpog/infra-live#129: Creating dir "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default"
2019/08/06 19:17:39+0000 [DBUG] working_dir.go:189 milpog/infra-live#129: Ran: git clone --branch Test-branch --depth=1 --single-branch https://milpog:<redacted>@github.com/milpog/infra-live.git /home/atlantis/.atlantis/repos/milpog/infra-live/129/default. Output: Cloning into '/home/atlantis/.atlantis/repos/milpog/infra-live/129/default'...
2019/08/06 19:17:39+0000 [INFO] project_command_builder.go:131 milpog/infra-live#129: Successfully parsed atlantis.yaml file
2019/08/06 19:17:39+0000 [DBUG] project_finder.go:84 milpog/infra-live#129: Checking if project at dir "dev" workspace "default" was modified
2019/08/06 19:17:39+0000 [INFO] project_command_builder.go:136 milpog/infra-live#129: 0 projects are to be planned based on their when_modified config
2019/08/06 19:17:39+0000 [INFO] command_runner.go:111 milpog/infra-live#129: Determined there was no project to run plan in

Second approach

version: 3
projects:
  - dir: dev
    autoplan:
      when_modified: ["**/*.hcl"]

Logs

2019/08/06 19:07:01+0000 [INFO] middleware.go:42 server: POST /events – respond HTTP 200
2019/08/06 19:07:03+0000 [DBUG] project_command_builder.go:102 milpog/infra-live#129: Got workspace lock
2019/08/06 19:07:03+0000 [DBUG] project_command_builder.go:110 milpog/infra-live#129: 4 files were modified in this pull request
2019/08/06 19:07:03+0000 [INFO] working_dir.go:123 milpog/infra-live#129: Creating dir "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default"
2019/08/06 19:07:10+0000 [DBUG] working_dir.go:189 milpog/infra-live#129: Ran: git clone --branch Test-branch --depth=1 --single-branch https://finiata-machine-user:<redacted>@github.com/milpog/infra-live.git /home/atlantis/.atlantis/repos/milpog/infra-live/129/default. Output: Cloning into '/home/atlantis/.atlantis/repos/milpog/infra-live/129/default'...
2019/08/06 19:07:10+0000 [INFO] project_command_builder.go:131 milpog/infra-live#129: Successfully parsed atlantis.yaml file
2019/08/06 19:07:10+0000 [DBUG] project_finder.go:84 milpog/infra-live#129: Checking if project at dir "dev" workspace "default" was modified
2019/08/06 19:07:10+0000 [DBUG] project_finder.go:106 milpog/infra-live#129: File "dev/_global/aiven-project-secrets/terragrunt.hcl" matched pattern
2019/08/06 19:07:10+0000 [INFO] project_command_builder.go:136 milpog/infra-live#129: 1 projects are to be planned based on their when_modified config
2019/08/06 19:07:10+0000 [DBUG] project_command_builder.go:138 milpog/infra-live#129: Determining config for project at dir: "dev" workspace: "default"
2019/08/06 19:07:10+0000 [DBUG] global_cfg.go:319 milpog/infra-live#129: Setting apply_requirements: [approved,mergeable] from repos[1], id: /.*/
2019/08/06 19:07:10+0000 [DBUG] global_cfg.go:319 milpog/infra-live#129: Setting workflow: "default" from repos[1], id: /.*/
2019/08/06 19:07:10+0000 [DBUG] global_cfg.go:319 milpog/infra-live#129: Setting allowed_overrides: [] from repos[1], id: /.*/
2019/08/06 19:07:10+0000 [DBUG] global_cfg.go:319 milpog/infra-live#129: Setting allow_custom_workflows: false from repos[1], id: /.*/
2019/08/06 19:07:10+0000 [DBUG] global_cfg.go:171 milpog/infra-live#129: Final settings: apply_requirements: [approved,mergeable], workflow: default
2019/08/06 19:07:10+0000 [INFO] project_locker.go:74 milpog/infra-live#129: Acquired lock with id "milpog/infra-live/dev/default"
2019/08/06 19:07:10+0000 [DBUG] project_command_runner.go:136 milpog/infra-live#129: Acquired lock for project
2019/08/06 19:07:10+0000 [DBUG] working_dir.go:79 milpog/infra-live#129: Clone directory "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default" already exists, checking if it's at the right commit
2019/08/06 19:07:10+0000 [DBUG] working_dir.go:101 milpog/infra-live#129: Repo is at correct commit "fe97a97f8fbf77d764a105e4aeb12b6d7ecf04bd" so will not re-clone
2019/08/06 19:07:11+0000 [DBUG] run_step_runner.go:54 milpog/infra-live#129: Error: exit status 1: running "terragrunt plan -no-color -out \"${PLANFILE}\"" in "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default/dev":
[terragrunt] [/home/atlantis/.atlantis/repos/milpog/infra-live/129/default/dev] 2019/08/06 19:07:10 Running command: terraform --version
[terragrunt] 2019/08/06 19:07:10 Reading Terragrunt config file at /home/atlantis/.atlantis/repos/milpog/infra-live/129/default/dev/terragrunt.hcl
[terragrunt] 2019/08/06 19:07:10 Assuming IAM role arn:aws:iam::606443424440:role/atlantis-allow-full-access-from-other-accounts
[terragrunt] 2019/08/06 19:07:11 Did not find any Terraform files (*.tf) in /home/atlantis/.atlantis/repos/milpog/infra-live/129/default/dev
[terragrunt] 2019/08/06 19:07:11 Unable to determine underlying exit code, so Terragrunt will exit with error code 1
2019/08/06 19:07:11+0000 [INFO] command_runner.go:231 milpog/infra-live#129: Deleting plans because there were errors and automerge requires all plans succeed
2019/08/06 19:07:12+0000 [DBUG] command_runner.go:432 milpog/infra-live#129: Updating DB with pull results

Thanks for help!

[milpog]

My directory structure for files modified in this PR looks like this

  ├── _global
  │   ├── aiven-project
  │   │   └── terragrunt.hcl
  │   └── aiven-project-secrets
  │       └── terragrunt.hcl
  └── eu-central-1
      └── aiven
          └── data-stores
              └── kafka
                  └── terragrunt.hcl

[ldormoy]

I don't think you need a client-side config.

You should rather use a server-side repo config.

I use the same tools and versions as you, here is what works for me:

# repos.yaml
repos:
- id: github.com/my_org/my_repo
  apply_requirements: [approved]
  workflow: terragrunt

workflows:
  terragrunt:
    plan:
      steps:
      - run: terragrunt plan -no-color -out $PLANFILE
    apply:
      steps:
      - run: terragrunt apply -no-color $PLANFILE

With this config, autoplan works fine. The only bug is that it also detects the root terragrunt config file and tries to plan it.

[milpog]

Have you migrated all the files to new *.hcl format? As far as I remember when I still had some configuration files for terragrunt 0.18 it also worked.

My server-side repo config looks like this so it should not be a problem. But I will try again.

# repos lists the config for specific repos.
repos:
  # id can either be an exact repo ID or a regex.
  # If using a regex, it must start and end with a slash.
  # Repo ID's are of the form {VCS hostname}/{org}/{repo name}, ex.
  # github.com/runatlantis/atlantis.
  - id: /.*/

    # apply_requirements sets the Apply Requirements for all repos that match.
    apply_requirements: [approved, mergeable]

    # workflow sets the workflow for all repos that match.
    # This workflow must be defined in the workflows section.
    workflow: default

    # allowed_overrides specifies which keys can be overridden by this repo in
    # its atlantis.yaml file.
    allowed_overrides: []

    # allow_custom_workflows defines whether this repo can define its own
    # workflows. If false (default), the repo can only use server-side defined
    # workflows.
    allow_custom_workflows: false

# workflows lists server-side custom workflows
workflows:
  default:
    plan:
      steps:
        - run: terragrunt plan -no-color -out "${PLANFILE}"
    apply:
      steps:
        - run: terragrunt apply -no-color "${PLANFILE}"

[ldormoy]

Have you migrated all the files to new *.hcl format? As far as I remember when I still had some configuration files for terragrunt 0.18 it also worked.

No I am still in the process, but the migrated files are properly discovered and planned.

My server-side repo config looks like this so it should not be a problem. But I will try again.

Your config looks fine indeed.

[milpog]

@ldormoy I used your configuration but still have the same issue. I tested all combinations with and without atlantis.yaml config.

[milpog]

I went trough atlantis code and found the part where files are filtered for auto plan functionality. It is a little strange that auto plan was run for *.hcl files for me and @ldormoy. I suspect it was because we were replacing our current terraform.tfvars files with terragrunt.hcl ones so atlantis found that given project was modified (because of removed tfvars files) and it run terragrunt plan -no-color -out "${PLANFILE}".

@ldormoy you can check by yourself whether auto plan will be run if you modify any part of the infrastructure where you already have terragrunt.hcl configuration in place. It should not work IMHO because of this part:

https://github.com/runatlantis/atlantis/blob/master/server/events/project_finder.go#L120-L131

// filterToTerraform filters non-terraform files from files.
func (p *DefaultProjectFinder) filterToTerraform(files []string) []string {
    var filtered []string
    for _, fileName := range files {
        // Filter out tfstate files since they usually checked in by accident
        // and regardless, they don't affect a plan.
        if !p.isStatefile(fileName) && strings.Contains(fileName, ".tf") {
            filtered = append(filtered, fileName)
        }
    }
    return filtered
}

It was only working with terragrunt because there is condition which is catching every file name matching *.tf* pattern and terragrunt 0.18.x followed this pattern with terraform.tfvars files for storing infrastructure configuration.

When I have changed following condition and run atlantis locally, auto plan started working also for terragrunt.hcl files.

- if !p.isStatefile(fileName) && strings.Contains(fileName, ".tf") {
+ if !p.isStatefile(fileName) && (strings.Contains(fileName, ".tf") || strings.Contains(fileName, ".hcl")) {
    filtered = append(filtered, fileName)
}

[ldormoy]

oops you are right, I have the same problem.

@lkysow looks like @milpog change could be a solution?

If we go this way, could it be possible to use this opportunity to tackle the root terragrunt configuration issue?

    |__terragrunt.hcl
    |__frontend
    |        |__terragrunt.hcl
    |
    |__backend
    |        |__terragrunt.hcl
    |
    |__modules

The frontend and backend terragrunt.hcl are legit projects, but the the root terragrunt.hcl is a config file that is not meant to be found by atlantis. To my knowledge, all terragrunt repositories work this way (see https://github.com/gruntwork-io/terragrunt#quick-start for details).

[lkysow]

That function isn't used when autoplan.when_modified is set.

Also, it starts looking in the directory specified in your project setting. If you want it to find .hcl files that are further below that directory you need to specify that:

a/
  b/
    c/
      file.hcl

projects:
- dir: a
  autoplan:
    when_modified: [*.hcl] # won't match
    when_modified: [b/c/*.hcl] # will match
    when_modified: [**/*.hcl] # will match

[milpog]

@lkysow it looks like when_modified: ["**/*.hcl"] is not working as I stated in issue description. At least for me.

atlantis is recognizing that dev/_global/aiven-project-secrets/terragrunt.hcl was modified...

2019/08/06 19:07:10+0000 [DBUG] project_finder.go:106 milpog/infra-live#129: File "dev/_global/aiven-project-secrets/terragrunt.hcl" matched pattern

but instead of running atlantis plan -d dev/_global/aiven-project-secrets it is running atlantis plan -d dev.

2019/08/06 19:07:11+0000 [DBUG] run_step_runner.go:54 milpog/infra-live#129: Error: exit status 1: running "terragrunt plan -no-color -out \"${PLANFILE}\"" in "/home/atlantis/.atlantis/repos/milpog/infra-live/129/default/dev":

[lkysow]

but instead of running atlantis plan -d dev/_global/aiven-project-secrets it is running atlantis plan -d dev.

That's expected behaviour. You need to configure the root of each Terraform project as a directory. It sounds like you need:

projects:
- dir: dev/_global/aiven-project-secrets/

Terraform projects are not multi-directory, that's why it's built this way.

[andrewmackett]

@lkysow Thanks for the clarification. I've been having the same issue when trying to setup Atlantis with Terragrunt for the first time.

Really wish we weren't using Terragrunt right now. I have a lot of dir to configure!

[milpog]

@lkysow would you consider implementing solution I proposed in https://github.com/runatlantis/atlantis/issues/728#issuecomment-519121825 or something similar but in line with your vision of developing atlantis. I love atlantis but it would be huge pain for us to define each directory for our terragrunt setup.

We will probably fork atlantis and build our own binaries after each release to work with terragrunt seamlessly but I would like to avoid that if possible.

[mmclane]

I am looking at the same issue. I would like it to pick up that .hcl files changed and not just tf files without have to specify per directory. I don't see a way to set this in the server side config

[kipkoan]

@milpog - maybe you guys can consider making this configurable? Have the default be how it is now, thus changing nothing, but allow you to configure it to pick up .hcl files (or any other type of files that someone may want to configure it to autoplan on).

[lkysow]

@lkysow would you consider implementing solution I proposed in #728 (comment) or something similar but in line with your vision of developing atlantis. I love atlantis but it would be huge pain for us to define each directory for our terragrunt setup.

We will probably fork atlantis and build our own binaries after each release to work with terragrunt seamlessly but I would like to avoid that if possible.

I think a top-level autoplan.when_modified might make sense.

[kipkoan]

Just abstract the current autoplan logic. Maybe that's what the autoplan.when_modified could be.

[lkysow]

This was closed by #748