Terraform: Feature Request: terraform fmt applies sort to variables.tf and terraform.tfvars

[sysadmiral]

It would be useful imho for terraform fmt to also do an alphabetical sort on variable definition files, namely variables.tf and terraform.tfvars.

[Gary-Armstrong]

Disagree.

[nateww]

Disagree x 2

[spanktar]

We currently do this manually. While I agree with the dissenters, it would be excellent if this was an OPTION that could be enabled.

[dpgeekzero]

I'm curious, this sounds like a great feature. Why the disagreement?

[armenr]

I'd buy it.

[armenr]

I found a script somewhere on the webs that got me really close to what I needed, so I pretty much lifted that dude's code and made some minor changes - and now have a really, really, naive (and yeah, totally hacky) script to take care of automatically sorting terraform variables files.

See below, and please, if you have suggestions, I'd love for you to change the script and post back with what's more clever. I'll probably parameterize it so you pass the file you want sorted on the command line after calling the script. Tested on my own machine - works fine. Run at your own risk.

WARNING - If you've got ANY commented lines, descriptions inside the variable braces, or empty braces without any content initialized as variables, this will break your stuff! This only works when your variables look exactly like:

variable "testbed_vpc_name" {
  default = "testbed-vpc"
}

#!/bin/bash

# Get the number of lines in the document.
lines=$(cat variables.tf | wc -l)

# This is the starting range and end range. Each section is three lines.
x=1
y=3

until [ "$x" -gt "$lines" ]; do
    # Store the three lines to one line. 
    block=$(awk 'NR=="'"$x"'",NR=="'"$y"'"' variables.tf)
    # Echo each instance into my file. 
    # The $block variable is not double quotes so new lines are not honored. 
    echo $block >> tmp_vars.tf
    # Increment so it goes on to the next block.
    x=$((x+4)) 
    y=$((y+4)) 
done 

# Sort the output file in place by the second column.
sort -k2 tmp_vars.tf -o tmp_vars.tf
terraform fmt -write=true
mv tmp_vars.tf sorted_variables.tf

# Put it back into original formatting.
# while read i; do 
#     (echo "$i" | awk '{ print $1 " " $2 }'; echo "$i" | awk '{ print $3 }'; echo "$i" | awk '{ print $4 }'; echo "") >> final.txt
# done < output.txt

[apparentlymart]

Hi all! Thanks for the discussion here.

This does seem like a reasonable helper-tool, though I'd be nervous about making it the default behavior for terraform fmt since it could make some pretty destructive changes given certain input, such as separating doc comments from what they are commenting, etc.

Sorting a hypothetical variables.tf file is not really straightforward because there's really nothing special about that file from Terraform's perspective... it's just another config file, which some users choose to use only for variable blocks. Some users call this vars.tf, and others mix variable blocks with output blocks in the same file, depending on what they are trying to achieve.

I could see us defining a "canonical sort" for a Terraform configuration file, such as:

• The single terraform block, if present
• The single atlas block, if present
• variable blocks, alphabetized by name
• provider blocks, alphabetized by name and alias
• locals blocks, in _some_ order (there isn't a well-defined sort for these, so not sure; maybe we'd just meld them all together into a single block and sort by key :man_shrugging:)
• data blocks, alphabetized by type and then name
• module blocks, alphabetized by name
• resource blocks, alphabetized by type and then name
• output blocks, alphabetized by name

This would allow us to then have a tool that applies this sort to arbitrary config files, as long as the input complies with certain expectations, such as doc comments appearing immediately before whatever they belong to and having no other "unattached" comments at the top-level of the file.

(Much of the above ordering is arbitrary, just for illustration purposes; let's not bike-shed the specific ordering for the moment.)

The ordering of .tfvars files is easier because their top-level is all just user-supplied keys, which we could sort by name with the same constraints about comments.

Such a tool won't be a priority for us right now because we're in the middle of integrating a revamped config language parser and so this would end up needing to get rewritten against the new parser anyway, but once that's done we could think about how best to design a tool like this, how/whether it should integrate with terraform fmt etc.

In the mean time, having third-party tools to do this for subsets of valid Terraform configurations seems reasonable, with the caveat that they will probably need revision once the new configuration language parser is integrated since it includes new syntax elements that the current parser would choke on.

[Gary-Armstrong]

This is indeed a very good discussion. I will remain in the camp that organizes variables and other resources in more logical groupings by function rather than a lexical sorting. I would rather dev time be spent on making the fmt subcommand more flexible, since I don't like how it does some things but others are great.

It might meet certain requests if terraform state show is given additional display options. Not 100% related to sorting source code, but maybe 65% related in terms of finding things.

Ultimately I will also appreciate a greater amount of time spent on core and critical issues.

[robinbowes]

Here's a slightly less dirty hack than the previously-presented bash script.

#!/usr/bin/env python
# sort terraform variables

import sys
import re

# this regex matches terraform variable definitions
# we capture the variable name so we can sort on it
pattern = r'(variable ")([^"]+)(" {[^{]+})'


def process(content):
    # sort the content (a list of tuples) on the second item of the tuple
    # (which is the variable name)
    matches = sorted(re.findall(pattern, content), key=lambda x: x[1])

    # iterate over the sorted list and output them
    for match in matches:
        print ''.join(map(str, match))

        # don't print the newline on the last item
        if match != matches[-1]:
            print


# check if we're reading from stdin
if not sys.stdin.isatty():
    stdin = sys.stdin.read()
    if stdin:
        process(stdin)

# process any filenames on the command line
for filename in sys.argv[1:]:
    with open(filename) as f:
        process(f.read())

[armenr]

@robinbowes - Well done. Thank you for this. Hell of a lot cleaner than what I'd ham-fistedly mashed together.

[pysysops]

This is old and closed but still found by Google searching. For anyone who stumbles on this, the Python code above works but will destroy any map variables you have. Change the pattern regex to this:

pattern = r'(variable ")([\w\d]+)(" {\n[\w\W]+?\n})'

Example:
https://www.regexpal.com/?fam=107028

Sure, there's probably a nicer / more optimal way but it works...

[robinbowes]

Actually, this is probably the least hacky way to sort variables:

 json2hcl < <(jq . < <(hcltool variables.tf))

Or, if you prefer pipes:

hcltool variables.tf | jq . | json2hcl

[pysysops]

Nice @robinbowes

[smiller171]

I would love to see this as a non-default option in fmt, as well as the ability so sort parameters within resource blocks.

[tzaitsev]

Actually, this is probably the least hacky way to sort variables:

 json2hcl < <(jq . < <(hcltool variables.tf))

Or, if you prefer pipes:

hcltool variables.tf | jq . | json2hcl

At least with the versions of the libraries I tried this technique modifies and quotes the 'variable' so each one reads as "variable" in a .tf @robinbowes

[smiller171]

hcl2json doesn't support hcl2, so the fun workarounds are dead for me :(

[sblack4]

I updated @robinbowes script for python3, https://gist.github.com/sblack4/34d74f6a4a6df65eb8d6e563a5135111

> python sort_terraform_variables.py variables.tf > sorted_variables.tf
> mv sorted_variables.tf variables.tf

[robinbowes]

terraform-config-inspect might also be worth a look.

[sblack4]

terraform-config-inspect might also be worth a look.

It's cool and probably would be useful if I was handy in go but I need a quick fix. I ran it and here's an example output:

>  terraform-config-inspect                                                                                                                                                                                                                

# Module `.`

## Input Variables
* `name` (required): Moniker to apply to all resources in the module
* `tags` (required): User-Defined tags

## Output Values
* `tags_module`: Tags Module in it's entirety

## Child Modules
* `tags` from `rhythmictech/tags/terraform` (`1.0.0`)

[robinbowes]

Try it with the --json switch.