Terragrunt: terraform init : backend re initialization is failing

[irfanjs]

Hi,
i am using terraform + terragrunt to deploy the infra.

here is the directory structure i have .

resource-group/
service-bus/
terragrunt.hcl

i am using remote backend to store the state files . here is the configuration.

remote_state {
    backend = "azurerm"
    config = {
        key = "${path_relative_to_include()}/terraform.tfstate"
        resource_group_name = "ABC-QA-RG"
        storage_account_name = "strgacct"
        container_name = "tfstate"
        subscription_id = "sadsadsads45435c59"
    }
}

inputs = {
  location = "East US"
  subs_id  = "fsdfdsfdsfdsf-sdfdsfdsf-6-4250b797bc59"
  tenant_id = "sdfdsfdsf67657657-546546"

}

the issue is , i have just moved the development environment to some other machine and copy/paste the above folder structure as it is .
now, in the parent folder , when i run the command terragrunt plan-all it fails , saying that , you need to initialize the terraform.
as per doc : https://www.terraform.io/docs/backends/config.html , it says, just run the commandterraform initand everything will be initialized.

As per my understanding , i expect terraform / terragrunt to get the state of infra from the remote backend and start the execution without error . is that correct ?

please suggest / help what needs to be done to resolve the error .

[irfanjs]

please suggest . this is blocker .

[yorinasub17]

terragrunt should run init automatically on a completely clean clone. Did you by chance copy over the .terragrunt-cache as well, and if so, does it work if you remove that dir?

[ahlinc]

I found that the v0.23.34 version works fine and doesn't run terraform init every time on every dependency when I call terragrunt apply-all. All newer versions starting from v0.23.35 seems are broken.

[yorinasub17]

doesn't run terraform init every time on every dependency

This is part of the dependency optimization feature and it is init-ing in a temporary folder so that terragrunt can pull down the remote state information as opposed to running through the standard path of parsing the entire config (including recursive dependencies) and redownloading sources, etc.

Note that this is only init-ing the remote state. The target module source is an empty terraform module that only has the backend block configured, so no providers or modules should be downloaded.

If this is not desirable, you can disable this functionality starting with v0.23.37 by setting disable_dependency_optimization = true on the remote_state block. You can read more about this in the updated docs on the dependency block.

[ahlinc]

I've checked how much time takes a deployment cycle in a folder on second and third attempts:
v0.23.34 ~ 32 sec.
v0.23.37 ~ 1 min. 2 sec.
It seems that this optimization doesn't do any speed up in my case.
Also here is an example of output on third deployment attemt:

Initializing the backend...

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding latest version of hashicorp/null...

Initializing provider plugins...
- Finding latest version of hashicorp/external...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/local...
- Finding latest version of hashicorp/external...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/local...
- Finding latest version of hashicorp/external...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/local...
- Installing hashicorp/local v1.4.0...
- Installing hashicorp/aws v3.4.0...
- Installing hashicorp/external v1.2.0...
- Installed hashicorp/local v1.4.0 (signed by HashiCorp)
- Installed hashicorp/external v1.2.0 (signed by HashiCorp)
- Installing hashicorp/null v2.1.2...
- Installing hashicorp/local v1.4.0...
- Installed hashicorp/null v2.1.2 (signed by HashiCorp)
- Installed hashicorp/local v1.4.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.4.0...
- Installed hashicorp/aws v3.4.0 (signed by HashiCorp)
- Installing hashicorp/null v2.1.2...
- Installing hashicorp/external v1.2.0...
- Installed hashicorp/null v2.1.2 (signed by HashiCorp)
- Installed hashicorp/external v1.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.4.0...
- Installing hashicorp/local v1.4.0...
- Installed hashicorp/aws v3.4.0 (signed by HashiCorp)
- Installing hashicorp/external v1.2.0...
- Installed hashicorp/local v1.4.0 (signed by HashiCorp)
- Installing hashicorp/null v2.1.2...
- Installed hashicorp/external v1.2.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/aws: version = "~> 3.4.0"
* hashicorp/external: version = "~> 1.2.0"
* hashicorp/local: version = "~> 1.4.0"
* hashicorp/null: version = "~> 2.1.2"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Where we can see that terraform downloads many external providers every time.

[yorinasub17]

Thanks for the report. I'm investigating this now.

[yorinasub17]

@ahlinc Ok I think I have a fix for this in #1321 . Can you try https://github.com/gruntwork-io/terragrunt/releases/tag/v0.23.38-alpha.1 to see if that has similar speeds to v0.23.34?

[ahlinc]

I don't see any terraform init calls and outputs with the v0.23.38-alpha.1 version.
But the running time looks not well:

real    0m52.974s
user    0m51.535s
sys 0m7.645s

For v0.23.34:

real    0m29.304s
user    0m40.001s
sys 0m3.894s

Here is how I tested it:

                                                     # Install a terragrunt version for the test
cd ...                                               # To some subfolder with terragrunt.hcl that has many dependencies
git -C "$(git rev-parse --git-dir)/.." clean -ffdx   # To cleanup all .terragrunt-cache dirs in a whole git repo
time bash -c "yes y | terragrunt apply-all"          # Run 5 time to find mean running time

[ahlinc]

Setting disable_dependency_optimization = true for v0.23.38-alpha.1 version fixes the running time and it is the same as for v0.23.34 version.

[yorinasub17]

Thanks for testing it! It is still running init, but with get-plugins=false now, with the difference in runtime coming from the delay to setup the backend configuration.

It looks like there is further optimization we can do, but probably will keep it as is for now, and you can opt out of it since the latest version. I filed https://github.com/gruntwork-io/terragrunt/issues/1322 for revisiting later if you want to follow that ticket.