Terraform-provider-aws: Plan output of migrating a policy is unreadable

[gtmtech]

Terraform 0.9.8

I think I raised something like this before and someone suggested a way of formatting the plan output differently, but I cannot find any documentation on it so raising it again.

Basically when I have a policy such as an aws_iam_role_policy, it is basically a huge JSON string. Typically I'll have this in a json file, and I'll reference with the ${file()} method in the aws_iam_role_policy resource. When I change that policy to a different JSON, it would be really nice if the plan output is actually readable. Instead it treats JSON as a string including all the "\n" characters written as "\n" - this means it is very hard to understand whether the policy will do the right thing when applied, especially for big json strings.

It would be much better if in these cases, it could somehow spit out the JSON side-by-side in sdiff format, or even as patch format so you can really see very easily which lines in the JSON have changed.

As it is, I have to write a special utility to paste the output of terraform plan into which will format it in a way where I can validate that the plan will do what I want.

e.g. this is the kind of thing I see: - I'm sure you'll agree its not meant for humans to easily work out whats going on.

~ aws_iam_role_policy.foo_bucket_access
    policy: "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        
{\n            \"Sid\": \"BucketRead\",\n            \"Effect\": \"Allow\",\n   
         \"Action\": [\n                \"s3:GetBucketLocation\",\n             
   \"s3:List*\"\n            ],\n            \"Resource\": [\n                \"
arn:aws:s3:::foo.libs\",\n                \"arn:aws:s3:::foo.raw.data\",\n      
          \"arn:aws:s3:::foo2.raw.data\"\n            ]\n        },\n        {\n
            \"Sid\": \"BucketObjectRead\",\n            \"Effect\": \"Allow\",\n
            \"Action\": [\n                \"s3:Get*\",\n                \"s3:He
ad*\",\n                \"s3:List*\"\n            ],\n            \"Resource\": 
[\n                \"arn:aws:s3:::foo.libs/*\"\n            ]\n        },\n     
   {\n            \"Sid\": \"BucketObjectWrite\",\n            \"Effect\": \"All
ow\",\n            \"Action\": [\n                \"s3:Get*\",\n                
\"s3:List*\",\n                \"s3:Head*\",\n                \"s3:Create*\",\n 
               \"s3:Put*\",\n                \"s3:Abort*\"\n            ],\n    
        \"Resource\": [\n                \"arn:aws:s3:::foo.raw.data/*\",\n     
           \"arn:aws:s3:::foo2.raw.data/*\"\n            ]\n        }\n    ]\n}\
n" => "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n   
         \"Sid\": \"BucketRead\",\n            \"Effect\": \"Allow\",\n         
   \"Action\": [\n                \"s3:GetBucketLocation\",\n                \"s
3:List*\"\n            ],\n            \"Resource\": [\n                \"arn:aw
s:s3:::data.foo\",\n                \"arn:aws:s3:::foo.libs\",\n                
\"arn:aws:s3:::foo.raw.data\",\n                \"arn:aws:s3:::foo2.raw.data\"\n
            ]\n        },\n        {\n            \"Sid\": \"BucketObjectRead\",
\n            \"Effect\": \"Allow\",\n            \"Action\": [\n               
 \"s3:Get*\",\n                \"s3:Head*\",\n                \"s3:List*\"\n    
        ],\n            \"Resource\": [\n                \"arn:aws:s3:::foo.libs
/*\"\n            ]\n        },\n        {\n            \"Sid\": \"BucketObjectW
rite\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n        
        \"s3:Get*\",\n                \"s3:List*\",\n                \"s3:Head*\
",\n                \"s3:Create*\",\n                \"s3:Put*\",\n             
   \"s3:Abort*\"\n            ],\n            \"Resource\": [\n                \
"arn:aws:s3:::data.foo/*\",\n                \"arn:aws:s3:::foo.raw.data/*\",\n 
               \"arn:aws:s3:::foo2.raw.data/*\"\n            ]\n        }\n    ]
\n}\n"

[apparentlymart]

Hi @gtmtech! Sorry (again) for this unreadable diff.

We are planning to make improvements to the diff output once we've completed some refactoring of how Terraform represents the diff internally. Currently, for historical reasons, everything is just represented as the same kind of string and so gets treated identically for diffing purposes. We are intending to make the diff structure support more complex types, at which point the diff _renderer_ will probably be rewritten entirely to produce a structural diff rather than a field-by-field diff, and at that point I expect we would also incorporate special handling for multi-line strings too.

In principle a more targeted change could be made here to address _just_ the multi-line string case. Given that in the long run the current diff code will likely be mostly discarded as we move to the new internal model, we've been hesitant to continue making incremental tweaks to it, but if someone had the time and motivation to implement a smaller, temporary fix here we'd be happy to discuss approaches (including mocking what the UI might look like in this case) and then, hopefully, eventually review a PR. But this feels like a non-trivial amount of work and so I'm reluctant to ask someone to embark on this knowing that it'll eventually be -- at least in part -- rewritten.

In the mean time, I can only ask for your patience as we work through the internal changes that will eventually enable a more useful diff rendering in general.

[akumria]

Do you have a pointer / link to the internal core work being undertaken? It would allow interested parties to track the upstream feature.

[marco-m]

@apparentlymart any news ?

[Jonnymcc]

Something like this would be nice https://github.com/yudai/gojsondiff. Possible to make use of this lib?

[apparentlymart]

Terraform v0.12.0 will have the new diff renderer I mentioned, which now includes not only a line-oriented diff for multi-line strings but also a special mode when it detects that the before and after strings are both JSON arrays or objects.

This is something Terraform Core itself is handling, so the AWS provider needs to take no special action to support it, and so I'm going to close this out. You can try the behavior out in experimental contexts using v0.12.0-beta1.

[marco-m]

Yes! Very good news!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!