Terragrunt: Hooks init-from-module runs differently in 0.18 and 0.19

[antonbabenko]

Trying to convert my scripts to 0.19 and I have a problem with understanding how hooks work.

In 0.18 it worked like this. I have a hook which copied a file which contains backend "s3" {} before running terraform init -backend-config=... in Terragrunt working directory (.terragrunt-cache/foo/bar/):

after_hook "copy_common_main_providers" {
  commands = ["init-from-module"]
  execute  = ["cp", "${get_parent_tfvars_dir()}/../common/main_providers.tf", "."]
}

I run terragrunt init and main_providers.tf was copied just into terragrunt-cache working directory.

In 0.19 the similar code (shown below) copies file main_providers.tf in the wrong directory (current directory) and into the working directory:

after_hook "copy_common_main_providers" {
  commands = ["init-from-module"]
  execute  = ["cp", "${get_parent_terragrunt_dir()}/../../common/main_providers.tf", "."]
}

To remove unnecessary main_providers.tf from the wrong directory I added another hook (aka "hack"):

after_hook "remove_useless_copy_of_main_providers" {
  commands = ["init"]
  execute  = ["rm", "-f", "${get_parent_terragrunt_dir()}/${path_relative_to_include()}/main_providers.tf"]
}

Any ideas how to instruct Terragrunt to copy files just once into a working directory?

[brikis98]

In 0.19 the similar code (shown below) copies file main_providers.tf in the wrong directory (current directory) and into the working directory:

Not sure I follow. What directory is it copying to? Which directory do you expect it to copy to?

Also, the path in the updated code is different (../../ instead of ../). Is that intentional?

[antonbabenko]

Sorry for the delay.

The difference with ../../ and ../ is intentional, because it does not work otherwise in 0.19 while worked in 0.18.

This is the tree structure:

.
├── terragrunt.hcl
└── vpc
    ├── .terragrunt-cache
    │   └── 8tw6nJm13wWcoe79ee6cpyG1tvk
    │       └── BoKn_3NNiOJEg4PmxaP5wYFi3M8
    └── terragrunt.hcl

Parent terragrunt.hcl contains the hook:

after_hook "copy_common_main_providers" {
  commands = ["init-from-module"]
  execute  = ["cp", "${get_parent_terragrunt_dir()}/../../common/main_providers.tf", "."]
}

terragrunt init is running this hook before it changes directory to the working directory (.terragrunt-cache/8tw6nJm13wWcoe79ee6cpyG1tvk/BoKn_3NNiOJEg4PmxaP5wYFi3M8):

[terragrunt] [........./vpc] 2019/06/22 21:35:43 Executing hook: copy_common_main_providers
[terragrunt] [........./vpc] 2019/06/22 21:35:43 Running command: cp ....../../../common/main_providers.tf .
[terragrunt] 2019/06/22 21:35:45 Copying files from ........./vpc into ........./vpc/.terragrunt-cache/8tw6nJm13wWcoe79ee6cpyG1tvk/BoKn_3NNiOJEg4PmxaP5wYFi3M8
[terragrunt] 2019/06/22 21:35:45 Setting working directory to ........./vpc/.terragrunt-cache/8tw6nJm13wWcoe79ee6cpyG1tvk/BoKn_3NNiOJEg4PmxaP5wYFi3M8
[terragrunt] 2019/06/22 21:35:46 Initializing remote state for the s3 backend

I expect it to run the hook after changing directory to a working directory as it was in 0.18.

[skluck]

This also affects us.

It seems like in 0.18, hooks ran from the .terragrunt-cache as the current working directory, but in 0.19 the cwd is the source directory (not the cache).

Is it possible to know the cache directory from terragrunt hooks (A function for this would also work)? It is useful so we can modify the terraform project/code before terraform runs.

Sample config:

after_hook "add_backend" {
  commands = ["init", "init-from-module"]
  execute = ["cp", "/shared/backend.tf", "."]
}

[antonbabenko]

@skluck There is no function to get working dir at the moment, but there is an open issue https://github.com/gruntwork-io/terragrunt/issues/750 if you have time to work on it.

[brikis98]

Which version of Terragrunt? https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.5 fixed some issues with paths and include.

[antonbabenko]

Yes, I use the latest - 0.19.5. Problem is still there.

[brikis98]

Roger, thx for confirming. I'm a bit buried at the moment. Does anyone have a minute to dig into this and submit a PR?

[rgarrigue]

Same issue here, on the same use case as @skluck

[seppi91]

just want to confirm the issue even with terragrunt version v0.19.24

[lanmalkieri]

This is still an issue in v0.19.27 as well.

[geota]

Even this workaround doesnt seem to work for modules that directly reference a remote git module and pass in all the inputs directly - i.e. modules that dont reference a local filesystem module. The common-provider.tf does not get copied over. This might be a separate bug entirely, because even providing a local backend.tf in the child terragrunt.hcl folder doesnt seem to get copied over in the destroy phase - however it does get copied over in create phase.

Steps to reproduce...

1. Have a parent and child terragrunt.hcl
2. Use the snippet @antonbabenko posted in the parent terragrunt.hcl
3. In the child terragrunt.hcl provide a remote source (git url) and pass in any inputs directly in the terragrunt.hcl block. Make sure your remote git source doesnt include a backend.tf
4. Call apply-all at the parent terragrunt.hcl level and it should copy over the file fine
5. Remove any terragrunt-cache files.
6. Call destroy-all at the parent terragrunt.hcl level.
7. Confirm that the common-provider.tf does not get copied into the child modules cache directory.

[yorinasub17]

Hi folks! If you've been tracking this, the main use case for this functionality was the ability to copy a .tf file that contains the empty remote state backend configuration for use with modules that don't have that defined. We've discussed this use case internally and came up with the solution implemented in: https://github.com/gruntwork-io/terragrunt/pull/1050

The basic gist is as follows:

Terragrunt remote_state blocks now have a new attribute generate which can be used to output the remote state config as a terraform block. That is, if you had the following terragrunt config:

remote_state {
  backend = "s3"
  generate = {
    path = "provider.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "my-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

Instead of using -backend-config to manage the backend, terragrunt will output a file provider.tf in the working directory (where terraform is called) with the following contents:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "relpath/from/parent/to/child/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

before calling out to terraform.

In addition, this implementation introduces a new terragrunt block called generate. This block can be used to generate arbitrary files in the working directory. E.g:

generate "random_file" {
  path = "random_file.txt"
  if_exists = "overwrite"
  contents = "Hello, world"
}

will generate the file random_file.txt with the contents Hello, world in the working directory.

Hopefully, one of these two features would be a replacement for the before hook to copy a .tf file into the working directory. Let us know what you think on the PR!

Why didn't we implement a get_terragrunt_working_directory function?

Ideally we would implement a get_terragrunt_working_directory which seems like a more straightforward solution than code generation. However, the problem with this function is that the implementation is not trivial.

That is, the terragrunt working directory is a function of the source configuration defined in the terragrunt config, and the routine for determining that happens AFTER the config is parsed. This means that you need context of a partial parsing of the file in order to resolve that directory. While we could do the partial parsing in the function, this gets complicated when you have include blocks and need to resolve which source to use. It gets even more complicated when --terragrunt-source is passed in. Not to mention refactoring the routine so that it can be accessed by the config package.

Ultimately, it turned out to be much easier to maintain and implement a code generation approach, since the codegen routine can be injected after the terragrunt working directory is computed.

[lorengordon]

@yorinasub17 I'm willing to try it when it's released, seems promising! Our provider.tf file generally includes the terraform block, several provider blocks, and variable blocks used in the provider blocks. Long as the generation approach supports all that, including terraform interpolation and functions, I'd think it would work...

[yorinasub17]

The codegen feature is released! https://github.com/gruntwork-io/terragrunt/releases/tag/v0.22.0

Long as the generation approach supports all that, including terraform interpolation and functions, I'd think it would work...

The generate block injects the file into the terraform working directory before calling terraform, which means that it is the same as having that file in the module in the first place. So as long as you are referencing variables that are defined in the module (or if it isn't, you can include the variable blocks in the generated code), then this should work. Try it out, and please open issues for any bugs/problems you run into!

[antonbabenko]

Yes, this now work as expected. I got the current issue resolved. Closing this issue.

Thanks a lot, @yorinasub17 !