Terragrunt: Can you pipe terragrunt plan/apply to tfmask?

[arbreezy]

Hello,

This is more a question than an issue.
Is it possible to pipe terragrunt commands to tfmask in order to hide any sensitive date from terraform outputs?

e.g:
terragrunt plan --terragrunt-non-interactive --terragrunt-source-update -lock=true -lock-timeout=70s -out=./terragrunt.plan | tfmask

[brikis98]

Not sure. Have you tried it?

[arbreezy]

I was testing with a previous version of tfmask which wasn't supporting Terraform v12.
It seems to work fine when adding terragrunt apply -no-color | tfmask

I also tried to include tfmask in terragrunt hcl ( without -no-color arg) as an extra argument which failed with the below error.

e.g
Terrgrunt configuration

terraform { extra_arguments "mask_apply" { commands = [ "apply" ] arguments = [ "-input=false", "-var-file=${path_relative_from_include()}/terraform.tfvars", "-lock=true", "-lock-timeout=60s", " terragrunt.plan", " | tfmask" ] }

Error message:
terragrunt] 2020/07/29 16:04:40 Running command: terraform apply -input=false -var-file=../terraform.tfvars -lock=true -lock-timeout=60s ./terragrunt.plan | tfmask Too many command line arguments. Configuration path expected. [terragrunt] 2020/07/29 16:04:40 Hit multiple errors: exit status 1

[brikis98]

I was testing with a previous version of tfmask which wasn't supporting Terraform v12.
It seems to work fine when adding terragrunt apply -no-color | tfmask

Good to know!

terraform { extra_arguments "mask_apply" { commands = [ "apply" ] arguments = [ "-input=false", "-var-file=${path_relative_from_include()}/terraform.tfvars", "-lock=true", "-lock-timeout=60s", " terragrunt.plan", " | tfmask" ] }

This won't work. The contents of the arguments array is passed to terraform as args and NOT executed in a subshell. Therefore, the pipe (|) gets passed as an argument to terraform (which terraform won't recognize or understand) rather than being treated as a redirect by the shell.

[yorinasub17]

This is an interesting use case. It would be nice if terragrunt provided a log postprocessor block that allows you to define a process to stream logs through to before emitting to stdout.

This would be useful for automatically piping outputs through to these kinds of terraform output processors (e.g., tfmask, terraform-landscape, etc).

[arbreezy]

From an implementation point of view, it might be good to add in terragrunt hcl file another stanza like shell_arguments{} to trigger the postprocessor block.

[yorinasub17]

From an implementation point of view, it might be good to add in terragrunt hcl file another stanza like shell_arguments{} to trigger the postprocessor block.

While intuitive, unfortunately this doesn't quite make sense because we don't call terraform in a shell: we directly invoke the binary in a subprocess. This is why I mentioned a "log postprocessor" block, where stdout and stderr is streamed to the stanza.

[id27182]

While intuitive, unfortunately this doesn't quite make sense because we don't call terraform in a shell: we directly invoke the binary in a subprocess. This is why I mentioned a "log postprocessor" block, where stdout and stderr is streamed to the stanza.

I'm interested in this feature a loot, I'd like to try to implement it.

[id27182]

Seems like it's possible to connect stdout of cmd, which is used to run terraform to stdin of any other command.
Config can look like:

terraform {

  output_pipe "test" {
    stream = ["stdout", "stderr"]

    commands = [
       "apply",
       "plan",
       "import",
       "push",
       "refresh"
    ]

    executable = "tfmask"

    arguments = [ ]

    environment = [] 
  }

[yorinasub17]

Thanks for the design suggestion! That looks reasonable. I have a few suggested tweaks:

stream = ["stdout", "stderr"]

I think this should be a string, with 3 options stdout for stdout only, stderr for stderr only, or stdout_and_stderr for both interleaved.

executable = "tfmask"
arguments = [ ]

This should mimic what we currently have for before_hook and after_hook, which is a single attribute execute, which is the list of command + args (e.g., execute = ["echo", "Foo"] for echo Foo).