Terraform: Output Terraform log as json

[gilzellner]

I would like to log all terraform operations in our organisation.
For Example:

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_instance.created_instance_record[0]
      disable_api_termination: "true" => "false"
      tags.V_Recepies:         "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"

  ~ aws_instance.created_instance_record[1]
      disable_api_termination: "true" => "false"
      tags.V_Recepies:         "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"

  ~ aws_instance.created_instance_record[2]
      disable_api_termination: "true" => "false"
      tags.V_Recepies:         "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"


Plan: 0 to add, 3 to change, 0 to destroy.

------------------------------------------------------------------------

Right now we collect the output in a shell script and push to slack. I want to collect this and send to ELK, but in order for this to make sense, I need json output, or something similar, that would look like this:

{
    [ 
        { 
            object: aws_instance, 
            disable_api_termination: "true" => "false", 
            tags: 
                [V_Recepies: "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"]

        },
        { 
            object: aws_instance, 
            disable_api_termination: "true" => "false", 
            tags: 
                [V_Recepies: "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"]

        },
        { 
            object: aws_instance, 
            disable_api_termination: "true" => "false", 
            tags: 
                [V_Recepies: "role[af-base], recipe[af-elasticsearch-6]n" => ""run_list":["role[af-base]","recipe[af-elasticsearch-6]"]"]

        }
    ]

}

I saw Terraform has json output, but only for some of the things.
Is the any way to turn TF logs into json ?
Thanks!

https://stackoverflow.com/questions/54748220/in-terraform-is-it-possible-to-get-the-log-as-json?noredirect=1#comment96307555_54748220

[apparentlymart]

Hi @gilzellner! Thanks for sharing this use-case.

Full JSON-based equivalents of all of Terraform's ongoing output during operations is a big change that we've been considering for some time, but at this time the output of Terraform is changing enough between versions (as we iterate on what is the best user experience) that we don't think it's the right time to commit to a machine-readable representation that would then become an interface contract.

However, we are making small steps towards more JSON output as different parts of Terraform become stable enough that we don't expect to need to make significant changes. In the master branch for the forthcoming v0.12.0 release we've been working on a way to render a _saved plan_ as JSON, which would give you the specific information you described in this issue.

If you'd like to see how it stands right now (subject to change further before final release), you could try the dev snapshots, which are builds from Terraform itself and most of the providers from last week reflecting the latest changes on master. These are not appropriate to use in production, but they can give you a sense of how this new feature will behave in the final release. (The dev snapshots cannot auto-install providers in terraform init the same way a normal release can, so you'll need to also download any providers you use and place them in the same directory as the extracted terraform binary.)

terraform init
terraform plan -out=tfplan
terraform show -json tfplan

This feature gives access only to the plan. To see what actually happened during apply, for now the only option is to continue capturing the human-readable text logs, since the apply output is not currently structured enough to have a machine-readable version. That may change in a later release, but we need to make sure the Terraform workflow is baked enough that there are no further plans to significantly change the apply-time output before we'd be able to do that, and that is (unfortunate) not likely in the very near future.

I think there's already an issue covering other forms of JSON output, but I wasn't able to find it in some quick searching, so for now we'll use this issue to represent having JSON output for ongoing operation progress (logs), which I think is separate from JSON renderings of specific artifacts like the plan file.

[weakcamel]

Hmm.... does the -json flag even work?

$ terraform show -json
Usage: terraform show [options] [path]

  Reads and outputs a Terraform state or plan file in a human-readable
  form. If no path is specified, the current state will be shown.

Options:

  -module-depth=n     Specifies the depth of modules to show in the output.
                      By default this is -1, which will expand all.

  -no-color           If specified, output won't contain any color.
$ terraform version
Terraform v0.11.13
+ provider.null v2.1.0
+ provider.vault v1.5.0
+ provider.vsphere v1.9.1

I'm using remote state, but the same behaviour observed when running against a plan file.

[mildwonkey]

@weakcamel : the -json flag was added in terraform 0.12-beta. Updating our documentation without specifying that -json was a 0.12 feature was my mistake, and I apologize for the confusion.

[weakcamel]

@mildwonkey ah, I see - many thanks for explanation and no worries!

So I guess this means that the online docs are not branched/version but are always referring to the current state of terraform (even for unofficial, alfa/beta releases)?

Or is there some repository of all docs versions one could look up?

[mildwonkey]

The documentation generally refers to the current _general release_ of terraform. We also have one example of versioned documentation for the 0.12 release: if you look at the Configuration Language docs you will see 0.11 and 0.12 syntax have their own sections.

Sometimes our docs will specific which version a feature or flag was included in. In this case, the terraform show docs should have clarified that -json was introduced in v0.12.

If you ever do want to look at our documentation source code, it's all in this repository! For instance, the command pages can be found here.

So if you wanted to see the _accurate_ terraform show command documentation for terraform 0.11: https://github.com/hashicorp/terraform/blob/v0.11.13/website/docs/commands/show.html.markdown

Again, you should not need to do this - the documentation should accurately reflect the current general release of terraform, and should also make it clear when feature is introduced in a specific version.

[pselle]

I believe that at this point, terraform show -json is addressing this request, so I'm going to close this issue.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.