Terragrunt: Destination path already exists and is not an empty directory using --terragrunt-source-update

[matiasbavauala]

Hi everyone,

We have experienced the following error while trying to run terragrunt plan-all --terragrunt-source-update --terragrunt-non-interactive --terragrunt-include-external-dependencies

error downloading 'ssh://[email protected]/project-terragrunt/secrets': /usr/bin/git exited with 128: fatal: destination path '..project-terragrunt/secrets/.terragrunt-cache/M062klXZFj2B2dKtXGow5FueqdY/ZS8XBy97Un6SDm3LMf87-Dx8xEY' already exists and is not an empty directory.

Our project has the following structure:

...
├── lambda
│   ├── LambdaA
│   │   └── terragrunt.hcl
│   ├── LambdaB
│       └── terragrunt.hcl
├── secrets
│   └── terragrunt.hcl
...

And we were trying to execute the command in the lambda folder.

LambdaA and LambdaB have:

terraform {
  source = "git::[email protected]:project-terraform//lambda/LambdaA?ref=branch"
}
dependency "secret" {
  config_path = "../../secrets"
}

We are currently avoiding this issue changing the reference to a certain commit and removing the flag --terragrunt-source-update, so we think the problem should be in the cache folder cleanup.

[brikis98]

Could you share more of the log output? Seeing the surrounding context for your error would be helpful.

[matiasbavauala]

Hi Yevgeniy!

I attach an output of the log where we see the error:

output-terragrunt-test.txt

This is a smaller version than the original, which is too long. If you want, we can attach it too.

[matiasbavauala]

Hi everyone!

We still have this issue. Were able to conduct any research on it? @brikis98

[lgallard]

In this case, when running a plan-all, Terragrunt runs concurrent plans for each lambda, and both lambdas has the same dependencies, thus using --terragrunt-source-update will be trying to clone the source twice and at the same time.

A workaround can be to iterate with a loop using plan instead of plan-all. For example:

for D in lambda/*/;
  do 
    cd "${D}"
    terragrunt plan --terragrunt-source-update --terragrunt-non-interactive -no-color
    cd ../..
  done

Another approach can be to delete the .terragrunt-cache directory to force downloading the sources for each dependency and module. For example, using a before_hook like this one:

terraform {
  before_hook "interpolation_hook" {
    commands     = ["plan"]
    execute      = ["rm", "-rf", "${get_terragrunt_dir()}/.terragrunt-cache/" ]
 }

But these are just workarounds. Terragrunt should be smart enough to resolve the dependencies, and download them just one time and then run a plan for each lambda.

[zelig81]

Hi!
I have the same issue. I'm trying to use the same terraform module in multiple terragrunt modules.
The second workaround throws me the following panic on runtime error (Terraform 0.12.12, Terragrunt 0.20.4):

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x199a682]

goroutine 8 [running]:
github.com/gruntwork-io/terragrunt/cli.runTerraformWithRetry(0xc0002cc2c0, 0x3, 0x0)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:475 +0x142
github.com/gruntwork-io/terragrunt/cli.runTerragruntWithConfig.func1(0xc00048d400, 0x1)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:428 +0x2a
github.com/gruntwork-io/terragrunt/cli.runActionWithHooks(0x1c060d9, 0x9, 0xc0002cc2c0, 0xc000111000, 0xc00050bd30, 0xc00024c960, 0xc000496010)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:439 +0x28e
github.com/gruntwork-io/terragrunt/cli.runTerragruntWithConfig(0xc0002cc2c0, 0xc000111000, 0x0, 0x0, 0x0)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:427 +0x2c7
github.com/gruntwork-io/terragrunt/cli.runTerragrunt(0xc0002cc2c0, 0x1c1076d, 0x15)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:315 +0x4fb
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runNow(0xc000305540, 0x0, 0x0)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:238 +0x16f
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runModuleWhenReady(0xc000305540)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:201 +0x6a
github.com/gruntwork-io/terragrunt/configstack.runModules.func1(0xc000584ef0, 0xc000305540)
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:171 +0x5b
created by github.com/gruntwork-io/terragrunt/configstack.runModules
    /private/tmp/terragrunt-20191014-59879-vdrekm/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:169 +0xe1

[gcrespi]

Hi @zelig81, yes there were some problems with that workarround so we finally decide to analyse when the problem was first created and we found that it was on this PR.
What has worked for us so far is to make a fork and revert that PR then we compile the code from there and the command we are running (terragrunt plan-all --terragrunt-source-update --terragrunt-non-interactive --terragrunt-include-external-dependencies) worked as expected.

@brikis98 We know that this is not a final solution because on reverting that PR we are breaking the use case of the person that proposed it originally, but I think that our use case is one more commonly needed than the one addressing that PR. Would you consider reverting that PR on the main project?

[yorinasub17]

I'll take a detailed look at this today since I implemented the fix.

[yorinasub17]

Hi, I decided to revert the problematic change for the time being. Can you see if that fixes the issue? I created an integration testing release for convenience: https://github.com/gruntwork-io/terragrunt/releases/tag/v0.21.2-alpha.1

[gcrespi]

I've just tried with that release and it worked :+1:

[zelig81]

I've tried it now - worked as charm :)

[yorinasub17]

Hi @gcrespi and @zelig81 , thanks for confirming that fix! I would also appreciate it if you can give https://github.com/gruntwork-io/terragrunt/releases/tag/v0.21.2-alpha.2 (https://github.com/gruntwork-io/terragrunt/pull/937) a try as well. Unlike the previous fix which rolls back the change, this will do proper synchronization of the terragrunt output call (and cache the results) so that we can keep the original change that is causing problems. If this works, we will most likely go with this version of the fix.

[yorinasub17]

Released the latest alpha fix as v0.21.2 (https://github.com/gruntwork-io/terragrunt/releases/tag/v0.21.2). Binaries will show up shortly. The original alpha releases will be deleted once the binaries for v0.21.2 show up.

I will keep this open until I have confirmation that this fixes the root issue.

[zelig81]

After upgrading to 0.21.2 the problem still persists :(
In order to work around the problem I've remove the cache files using find . ..... solution from the read.me but it did not help...
When once more I used the alpha version that is still on stored on my computer, the problem continues to be solved, so meanwhile I continue to use it :)

Directory structure:

.
├── production
│   └── environment.tfvars.json
├── staging
│   ├── infrastructure
│   │   ├── sg
│   │   │   ├── sg-db-access-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-https-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-ssh-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-vpc-traffic
│   │   │   │   └── terragrunt.hcl
│   │   │   └── sg-web-traffic
│   │   │       └── terragrunt.hcl
│   │   └── vpc
│   │       └── terragrunt.hcl
│   └── environment.tfvars.json
├── README.md
├── commons.tfvars.json
└── terragrunt.hcl

The log with errors:

[terragrunt] 2019/11/03 09:51:00 Encountered the following errors:
Hit multiple errors:
error downloading 'ssh://git@*****/tf-modules.git': /usr/bin/git exited with 128: Cloning into '/Users/ilyag/code/infra-deployment/staging/infrastructure/vpc/.terragrunt-cache/Ef-6lmvbZXkKDLUhRJi2mrsUqtQ/e--jc6WhBhtnKuyeyTkOxcY52Do'...
fatal: cannot copy '/Library/Developer/CommandLineTools/usr/share/git-core/templates/description' to '/Users/ilyag/code/infra-deployment/staging/infrastructure/vpc/.terragrunt-cache/Ef-6lmvbZXkKDLUhRJi2mrsUqtQ/e--jc6WhBhtnKuyeyTkOxcY52Do/.git/description': File exists

Hit multiple errors:
error downloading 'ssh://git@********/tf-modules.git': /usr/bin/git exited with 128: fatal: destination path '/Users/ilyag/code/infra-deployment/staging/infrastructure/vpc/.terragrunt-cache/Ef-6lmvbZXkKDLUhRJi2mrsUqtQ/e--jc6WhBhtnKuyeyTkOxcY52Do' already exists and is not an empty directory.

unlinkat /Users/ilyag/code/infra-deployment/staging/infrastructure/vpc/.terragrunt-cache/Ef-6lmvbZXkKDLUhRJi2mrsUqtQ/e--jc6WhBhtnKuyeyTkOxcY52Do: directory not empty
Hit multiple errors:
error downloading 'ssh://git@*******/tf-modules.git': /usr/bin/git exited with 128: fatal: destination path '/Users/ilyag/code/infra-deployment/staging/infrastructure/vpc/.terragrunt-cache/Ef-6lmvbZXkKDLUhRJi2mrsUqtQ/e--jc6WhBhtnKuyeyTkOxcY52Do' already exists and is not an empty directory.

[terragrunt] 2019/11/03 09:51:00 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[yorinasub17]

@zelig81 Thanks for trying the new release. Can you share the entire log output, not just the error section, for a run with the environment variable TG_LOG=debug? This will help see other possibilities for the issue. It is unlikely that we will merge in the rolled back version, so I would much rather roll forward with the proper fix.

FWIW, I have been able to reproduce the issue in test and successfully validated that the issue is fixed in https://github.com/gruntwork-io/terragrunt/pull/937, but it is possible that you have a different setup that is causing this to happen that the fix in 0.21.2 isn't handling. The extended debug logging will help in identifying that potential problem.

[zelig81]

Sorry - took time to relaunch. Checked with terragrunt 0.21.4, terraform 0.12.13

folders structure of infra-deployment:

.
├── production
│   └── environment.tfvars.json
├── shared
│   └── route53
│       └── public-hosted-zone
│           └── terragrunt.hcl
├── staging
│   ├── infrastructure
│   │   ├── rds
│   │   │   └── aurora-online
│   │   │       └── terragrunt.hcl
│   │   ├── route53
│   │   │   └── private-hosted-zone
│   │   │       └── terragrunt.hcl
│   │   ├── sg
│   │   │   ├── sg-db-access-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-https-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-ssh-from-vpn
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── sg-vpc-traffic
│   │   │   │   └── terragrunt.hcl
│   │   │   └── sg-web-traffic
│   │   │       └── terragrunt.hcl
│   │   └── vpc
│   │       └── terragrunt.hcl
│   └── environment.tfvars.json
├── README.md
├── commons.tfvars.json
├── output.txt
└── terragrunt.hcl

folders structure of modules:

../tf-modules
├── rds
│   ├── README.md
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
├── route53
│   └── hosted-zone
│       ├── README.md
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── sg-generic
│   ├── README.md
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
├── vpc
│   ├── README.md
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
└── README.md

attached output of export TG_LOG=debug; terragrunt plan-all --terragrunt-source-update --terragrunt-source ~/code/tf-modules/ --terragrunt-non-interactive
debug.log

./terragrunt.hcl:
terragrunt.txt

staging/infrastructure/rds/aurora-online/terragrunt.hcl:
terragrunt.txt

[yorinasub17]

Hi thanks for sharing the logs. For some reason, it looks like the synchronization locking mechanism introduced in v0.21.2 isn't working for your environment. This is possible if the dependency config path isn't properly overlapping, but the logs seem to indicate that the path is cleaned correctly.

I just pushed a new release (https://github.com/gruntwork-io/terragrunt/releases/tag/v0.21.5) with additional debug logging which will give me a definitive answer for this. If you don't mind, can you run export TG_LOG=debug; terragrunt plan-all --terragrunt-source-update --terragrunt-source ~/code/tf-modules/ --terragrunt-non-interactive again for me using that version?

And I really appreciate your patience in helping out here!

[zelig81]

Of course, hope I'll do it today or tomorrow

[zelig81]

after upgrading to the new version of terragrunt it is suddenly started to work :)
terragrunt version v0.21.5
Terraform v0.12.13

I can confirm that the issue is solved. Thank you for your patience and time!!!!

[yorinasub17]

Thank you for your patience and time!!!!

And thank you for your patience helping me with debugging this!

Going to go ahead and close this now that it is confirmed fixed. If others continue to have issues, we can open a new ticket.