Terragrunt: Access Logging setup causes logging loop.

[tony-harverson-moonpig]

After #645 being merged, Terragrunt is setting up Access_Logging on terraform state buckets, logging access to the prefix TFStateLogs/ in the same bucket. This causes an access logging loop causing there to be lots of logs. It's also against Amazon's recommendations:

https://aws.amazon.com/premiumsupport/knowledge-center/s3-server-access-logs-same-bucket/
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html#server-access-logging-overview

I'd suggest the "skip_bucket_accesslogging" option should default to false on bucket creation.

If I PR'ed that change, would it be likely to be merged?

[brikis98]

So you're saying that if you send access logs for bucket X to bucket X, AWS gets in an infinite loop where it seems the access logs themselves as a new "access" and logs that too, and so on, infinitely?

[tony-harverson-moonpig]

According to Amazon, That's what happens.

Replication recipe:
Spin up a new bucket by running a terragrunt apply on any minimal terraform stack.
list the bucket in the aws console. No TFStateLogs will be present initially (not sure why this is, maybe there's a setup delay or something).

Inspect the bucket in an hour's time. We're seeing logs being continually written to that bucket, despite no additional access.

[brikis98]

Yikes 😮 Thanks for reporting.

I think then we'd want to change the logic to use a separate bucket for access logging. Perhaps we pick some name pattern by default (xxx-access-logs where xxx is the original bucket name?) and allow it to be overridden via config. A PR to fix this is VERY welcome.

[tony-harverson-moonpig]

With derived bucket names, there's an edge case around the lengths of bucket names (63 characters max), so long bucket names could result in illegally long log bucket names (which would assumedly fail creation), or would need to be truncated before adding the logging suffix (which would result in possibly non-unique names).

It might be better to have a single logs bucket with a predictable name, and a unique prefix (say /TFStateLogs/{tfstate-bucket-name}/) for each state bucket.

[brikis98]

Ah, that makes sense. In that case, let's make access logging off by default, and allow it to be turned with one config parameter, that also requires you to set the access logs bucket name in another param.

[tony-harverson-moonpig]

I'l try to get a PR in for that in the next couple of days.

[brikis98]

Thank you!

[michelzanini]

It would be good if access log is disabled by default 👍

[ina-stoyanova]

Hey @tony-harverson-moonpig @michelzanini Thought I'd give you a heads up that I'll be picking this up. I've read through the comments, so will be able to share some more details as I dive into this a bit more. 🙂

[dsecik]

Hello, first of all thanks for this improvement to the access logging setup. It would be handy to have an ability to specify also the target prefix to override targetPrefix := "TFStateLogs/" for example to be able to send access logs from multiple terraform state buckets to single logging bucket and not mix them together. Is there any chance to add such option?

[brikis98]

@dsecik Please file a separate issue for that! PR to support it is welcome too.

[dsecik]

Hi @brikis98, I created #1437. Sadly can't help with submitting PR as I have no experience with Go.