Terragrunt: mock-outputs in plan for lambda with sqs dependency

[matiasbavauala]

Hi everyone!

We have experienced the following error while trying to run terragrunt plan-all --terragrunt-source-update --terragrunt-non-interactive --terragrunt-include-external-dependencies in a lambda folder, which have dependency with sqs.

Unsupported attribute; This object does not have an attribute named “Sqs-Two_name”., and 1 other diagnostic(s)
[terragrunt] 2019/11/01 17:53:59 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

In the tfstate, we have the Sqs-One, but the plan is going to create a new resource (Sqs-Two).

Could there be a flag that always force you to use the mock_outputs in the plan?
Something like this
mock_outputs_forced_terraform_commands = [“plan”]

Our project has the following structure:

├── live
│   ├── lambda
│   │   ├── Lambda_Sqs-One_Dependent
│   │   │   └── terragrunt.hcl
│   │   └── Lambda_Sqs-Two_Dependent
│   │       └── terragrunt.hcl
│   └── sqs
│       └── terragrunt.hcl
└── terraform
    ├── lambda
    │   ├── Lambda_Sqs-One_Dependent
    │   │   └── main.tf
    │   └── Lambda_Sqs-Two_Dependent
    │       └── main.tf
    └── sqs
        ├── Sqs-One.tf
        └── Sqs-Two.tf

Sqs-One and Sqs-Two have:

module "Sqs-One" {
  source = "[email protected]:Enterprise/terraform-aws-sqs.git?ref=tags/2.1"
}

output "Sqs-One_name" {
  value       = module.Sqs-One.sqs_name
}

module "Sqs-Two" {
  source = "[email protected]:Enterprise/terraform-aws-sqs.git?ref=tags/2.1"
}

output "Sqs-Two_name" {
  value       = module.Sqs-Two.sqs_name
}

For each lambda terragrunt.hcl, we have:

terraform {
  source = "path_to_source"
}

dependency "sqs" {
  config_path = "../../sqs"

  mock_outputs = {
    Sqs-Two_name = "mock_name"
  }
}

inputs = {
  dlq_name  = dependency.sqs.outputs.Sqs-Two_name
}

[yorinasub17]

That seems reasonable. Will be happy to review a PR!

[barryib]

Doing something like this won't help ?

dependency "sqs" {
  config_path = "../../sqs"

  mock_outputs = {
    Sqs-Two_name = "mock_name"
  }

  skip_outputs = true
  mock_outputs_allowed_terraform_commands = ["plan"]
}

With skip_outputs seted to true, Terragrunt won't pull outputs from states.

[yorinasub17]

IIRC, setting skip_outputs = true will skip pulling outputs for terraform apply as well.

[thomvaill]

We have the same issue! When we add a new output, consumed by another dependent one, it breaks our CI/CD pipeline because mock_outputs are not taken into account (because the state already exists).
We were thinking of another solution: create a partial_mock_outputs = true option. It would default to false to be compatible with the current behavior. When true, it would merge the actual outputs with the mocked ones.
What do you think?

This problem should really be fixed because we have to do manual operations to fix our pipeline, which is not sustainable!

[yorinasub17]

When we add a new output, consumed by another dependent one, it breaks our CI/CD pipeline because mock_outputs are not taken into account (because the state already exists).
We were thinking of another solution: create a partial_mock_outputs = true option. It would default to false to be compatible with the current behavior. When true, it would merge the actual outputs with the mocked ones.
What do you think?

I think this has the element of surprise. It feels like a smell to me to be depending on mock_outputs to workaround that issue, and it won't be immediately obvious when things are coming from mock_outputs and when things are coming from the actual outputs.

In general, plan-all is simply broken for day to day use for various reasons including this one and we generally discourage it's usage now, especially in a CI/CD pipeline. We are actively investigating an alternative solution for the use case, but that is not going to come any time soon. It is simply not an easy feat to have plans on multiple state files because terraform doesn't give us a lot of functionality to work with that (since that is not really the main use case they are focusing on).

Nevertheless, if you need this kind of functionality right now, there are two alternative workarounds to this that doesn't require any changes to terragrunt:

1.) Use read_terragrunt_config to read in the dependency, and "pack" it using merge in the inputs block. E.g., if you had the following:

dependency.hcl

locals {
  default_outputs = {
    Sqs-Two_name = "mock_name"
  }
}

dependency "sqs" {
  config_path = "../../sqs"
}

inputs = {
  final_out = merge(dependency.sqs.outputs, local.default_outputs)
}

You can read this in in your other configs as follows:

locals {
  sqs_dep = read_terragrunt_config("/path/to/dependency.hcl").inputs.final_out
}

2.) Use the lookup terraform function to provide a default on use. E.g.,

inputs = {
  dlq_name  = lookup(dependency.sqs.outputs, "Sqs-Two_name", "mock_name")
}