Terragrunt: The apply-all and destroy-all commands should save log output to file

[brikis98]

When you run terragrunt apply-all or terragrunt destroy-all, they can make changes across a ton of different modules, in parallel. This is great, but as mentioned in #71, the amount of log output is overwhelming, and it's easy to miss something important. One small step in helping with this is to store the log output for each module somewhere. Perhaps within the module or a tmp directory will do.

[justyns]

Posting this here in case it helps someone else. You can save the output of the terraform commands by making a wrapper around terraform and having terragrunt call that.

An example wrapper script:

#!/bin/bash

terraform $@ 2>&1 | tee ./terraform.log

And then using it with terragrunt:

terragrunt apply-all --terragrunt-non-interactive --terragrunt-tfpath terraform-wrapper

For me, this creates a terraform.log file in each directory where terragrunt plan/applied something.

Ideally terragrunt could do something like this by itself, but this workaround helps until that happens.

[falmotlag]

Solution

Delay Terraform output for all modules and display it formatted at the end of Terragrunt execution.

CLI option

Add a --delay-terraform-output cli option to turn on this feature.

Code Change

1. Add a --delay-terraform-output cli option to turn on this feature.
2. in stack.Plan and stack.Apply assign a buffer to Writer
3. defer to function that print buffer of each module

Snippet of change to stack.Plan(terragruntOptions *options.TerragruntOptions) error:

if terragruntOptions.DelayOutput == true {
    outputStreams := make([]bytes.Buffer, len(stack.Modules))
    for n, module := range stack.Modules {
        module.TerragruntOptions.Writer = &outputStreams[n]
    }
    defer stack.printDelayedOutput(terragruntOptions, outputStreams)
}

Code to create cli flag and to printDelayedOutput is excluded from the snippet.

I am not too familiar with the codebase so I could be missing something here.
Can someone familiar with the codebase review this?

[brikis98]

Thx for the proposal!

Unfortunately, I've often found that buffering log output for a long period of time is not a great solution for a few reasons:

1. With no output, the user has no idea what's going on. Is apply still running? Is there an error? What's being deployed? This leads to hitting CTRL+C, which is not a good combination with Terraform.
2. Some CI systems will kill a build if it goes more than N minutes without log output (e.g., in CircleCi, the limit is 10m).

Therefore, I think we'd have to stream the log output to stdout/stderr, and if there's any sort of buffer, it's a file on disk.

[yorinasub17]

I wonder if we could just buffer the stdout and print out one module at a time in a stream, but continue to stream stderr logs? Terraform is fairly consistent in outputting the useful information to stdout in a single timestep so I don't think that would be too long.

[dudicoco]

@yorinasub17 would it be possible to implement this? I think most of the terragrunt community have been waiting for years for such a feature :)

[geota]

Logging in terrgrunt is one of the reasons it has been harder than I had anticipated getting wider adoption of TG within my current company.