Terragrunt: .terragrunt-cache fails to update with new source "...?ref=..."

[tomalok]

I'm updating my terragrunt.hcl's terraform { source = "..." } } value with new ...?ref=<commit> values as I make adjustments, but terragrunt isn't successfully pulling the new commit from the git repo. Instead, it complains...

[terragrunt] [/Users/jake/git/codecommit.us-west-2/jn-tf/jakesys/regions/us-west-2/vpcs/dev] 2019/06/25 18:04:19 Running command: terraform --version
[terragrunt] 2019/06/25 18:04:19 Reading Terragrunt config file at /Users/jake/git/codecommit.us-west-2/jn-tf/jakesys/regions/us-west-2/vpcs/dev/terragrunt.hcl
[terragrunt] 2019/06/25 18:04:19 Cleaning up contents of download folder /Users/jake/git/codecommit.us-west-2/jn-tf/jakesys/regions/us-west-2/vpcs/dev/.terragrunt-cache/QM1-63WWHsRnztH6ooxLgsdRMGg/tieM-yGmC7c1rfHpXzt3T9uRVsA
[terragrunt] 2019/06/25 18:04:19 Downloading Terraform configurations from git::ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/jn-tf-mod?ref=86e6f62 into /Users/jake/git/codecommit.us-west-2/jn-tf/jakesys/regions/us-west-2/vpcs/dev/.terragrunt-cache/QM1-63WWHsRnztH6ooxLgsdRMGg/tieM-yGmC7c1rfHpXzt3T9uRVsA
[terragrunt] 2019/06/25 18:04:19 Hit multiple errors:
error downloading 'ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/jn-tf-mod?ref=86e6f62': /usr/local/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git

[terragrunt] 2019/06/25 18:04:19 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

The hash subdirectories under .terraform-cache are always the same:

git::ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/jn-tf-mod//vpc?ref=6e30e8a
.terragrunt-cache
└── QM1-63WWHsRnztH6ooxLgsdRMGg
    └── tieM-yGmC7c1rfHpXzt3T9uRVsA
        └── vpc
            ├── _data.tf
            ├── _init.tf
            ├── _vars.tf
            ├── main.tf
            └── terragrunt.hcl
git::ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/jn-tf-mod//vpc?ref=c7f82af
.terragrunt-cache
└── QM1-63WWHsRnztH6ooxLgsdRMGg
    └── tieM-yGmC7c1rfHpXzt3T9uRVsA
        └── vpc
            ├── _data.tf
            ├── _init.tf
            ├── _vars.tf
            ├── main.tf
            └── terragrunt.hcl

If I delete .terragrunt-cache entirely, then it's able to pull the new commit reference. I've also tested to make sure it's actually pulling the specified commit and not just HEAD.

terraform 0.12.2 (via brew)
terragrunt 0.19.4 (via brew)
git 1.22.0 (via brew)
macOS mojave

[brikis98]

Just to make sure I understand:

1. The very first time you run terragrunt <some-command> it successfully downloads the code specified in source.
2. If you change nothing and run terragrunt <some-command>, you get the "error downloading" message?

[tomalok]

The first time i run terragrunt <some-command> it successfully downloads the code. Subsequent runs of terragrunt commands also succeed...

Until I update the commit reference in source and then attempt to run a terragrunt command. That's when I get the error.

If I delete the .terragrunt-cache directory, and try again, it will correctly download the code. terragrunt commands continue to work, until I need to modify the commit reference again.

[brikis98]

Ah, OK, thanks for the clarification. Sounds like some sort of bug with the cleanup code. This was changed recently: https://github.com/gruntwork-io/terragrunt/blob/master/cli/download_source.go#L332. Anyone have time to dig in and submit a PR?

[tomalok]

Did a diff on tree -a .terragrunt-cache output "before" and "after" new commit ref updates, hopefully this will help narrow down the issue...

The "after" is missing .../.git, .../.terragrunt-source-version, and .../vpc/*.tf -- otherwise the same files exist in both. The cleanup code did something (perhaps too much?); the error happens when it's trying to re-download (git clone/pull) the module source.

[ekini]

terrragrunt uses go-getter to download sources. On subsequent source get, go-getter library pulls the changes with git pull if the directory already exists.

Since https://github.com/gruntwork-io/terragrunt/blob/master/cli/download_source.go#L332 cleans everything but whitelisted files, there is no .git directory, and therefore git pull fails.

[brikis98]

@ekini Good find! Anyone have time for a PR to fix that issue?

[ekini]

I've had a look and the easiest workaround would be to exclude .git from deleting, but it doesn't solve the actual problem and might break on other version control systems.

So I don't see an easy solution from this side.
However, I couldn't understand the logic behind the second directory sync. So we get the source with go-getter and then copy it somewhere? Is it necessary?

If yes, then maybe instead of cleaning the dir here https://github.com/gruntwork-io/terragrunt/blob/master/cli/download_source.go#L332, it would be possible to do something like rsync --av --delete, deleting the files that are not in the source, excluding .terraform dir, for example.

[brikis98]

We have to clean up the download dir due to iterative development. E.g., You are running terragrunt apply --terragrunt-source /foo/bar over and over again, and making big changes to /foo/bar (including, potentially, deleting files/folders within /foo/bar). We need to make sure to keep the download folder in sync with /foo/bar. I do an ugly version of this by deleting the download folder (_except_ for .terraform, to avoid having to reconfigure remote state and re-download modules/providers), but I guess that had the unfortunate side effect of deleting .git and leading to confusing errors.

An rsync solution would be great, if one could be found that (a) works on every major OS and (b) works not only for syncing local dirs, but also from Git, Mercurial, and other source URLs.

[mattford63]

This is hurting our deploys quite a lot, breaks the live repo/module repo workflow - I'm pretty sure this didn't happen with 0.19.0, not had a chance to bisect fully.

[ekini]

If nobody wants to handle this, I can try in a couple of days :)

My thoughts are: in this function get a list of files in source and destination directories. Check if there are files in destination that don't exist in source and remove them, excluding .terraform folder. Then copy the files.

[brikis98]

@ekini That would be great!

[ekini]

Created a PR https://github.com/gruntwork-io/terragrunt/pull/774

[tomaszdudek7]

We are having the same issue when pointing the source at specific tag(lets say source=...?ref=1.0), doing stuff with terragrunt(thus having non empty .terragrunt-cache) and then trying to run apply on newer version (for example source=...?ref=2.0) of the source.

Thank you for working on it. Hopefully it will fix our case too. So far we have been just wiping the cache folder when the issue occurs, but that is a bit cumbersome as it requires redownloading providers which takes a minute or two.

[mattford63]

I built terragrunt from the PR #774 to try and repair our workflow, however the same problem as described at the top of the PR seems to persist. I will double check the build.

I'm also a bit puzzled as to how this is not effecting everyone who uses Terragrunt - this seems a bad break to a key feature (immutable version modules). Am I missing something?

[tomaszdudek7]

@mattford63 What are you referencing in ref= - commits' SHAs or git tags?

[mattford63]

Hi @spaszek I'm referencing tags.

[ekini]

@mattford63 could you please ensure you are using the PR#774 terragrunt version? Because that version removes only the files that were copied from the terragrunt module folder.

[mattford63]

Hi @ekini yes just double checked, I'm using my own build of PR #774

[ekini]

@mattford63 then could you please give more details? What files you have in the terragrunt module dir, in the source dir and in the working dir? Like in the starting comment.

Maybe after patching terragrunt you need to delete the working directory once, because .git has been deleted already.