Terraform-provider-aws: S3 bucket policy invalid principal for cloudfront

[Dgadavin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.14

• provider.aws v2.28.1

Affected Resource(s)

aws_cloudfront_origin_access_identity

Terraform Configuration Files

resource "aws_cloudfront_origin_access_identity" "default" {
  comment = "some comment"
}

Expected Behavior

I want to get correct iam_arn output. In docs I see it should be arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity E2QWRUHAPOMQZ but AWS wait something like this arn:aws:iam::cloudfront:user/CloudFront_Origin_Access_Identity_E2QWRUHAPOMQZ

Actual Behavior

* aws_s3_bucket_policy.default: Error putting S3 policy: MalformedPolicy: Invalid principal in policy status code: 400, request id: 91F717DA11D3AD4C, host id: neJZv3+m697Cym14SQnkBaUmDyYWrP7pg/sNyPk7T1PQmaosp8ZqNUytSTPvpxUJKHoXhr4v1oI=
When I try to add bucket policy.

[dirgapeter]

@Dgadavin getting same error, have you found solution? why close?

[Dgadavin]

@dirgapeter Sorry. I thought I find a solution with cloudfront_access_identity_path but it don't work. Reopen

[dirgapeter]

Using:

identifiers = [replace("${aws_cloudfront_origin_access_identity.website_bucket.iam_arn}", " ", "_")]

for now.

[Dgadavin]

Good workaround. But I think this should be fix by terraform.

[nodomain]

Thanks for posting this. And thanks to Google for indexing so fast. Just saved me - I was doubting everything since I did not change anything in my code.

[IronCore864]

I think there's probably something going on in AWS S3 side.
For some of my buckets, I must use underscore, while for others they still must use spaces.

[nodomain]

That sucks. I have a huge list of environments that are deployed via batch. I'll open a support ticket.

[bbrunod]

You must provide the canonical user id :

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "CanonicalUser": "CANONICAL_ID"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my_bucket/*"
        }
    ]
}

You can find it in the attribute of the aws_cloudfront_origin_access_identity ressource:

s3_canonical_user_id

[bbrunod]

Answer from aws :
_ The reason for this is that CloudFront updated to a new IAM auth system which does not allow spaces in user names. This means that old OAIs like "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EXXXXXXXXXXXX" will now look like this "arn:aws:iam::cloudfront:user/CloudFrontOriginAccessIdentityEXXXXXXXXXXXX". Users who try to hardcode either underscores or spaces into their bucket policy updates (not using CanonicalIds) will result in malformed principal, like the "invalid policy" error you’re getting. This due to some CloudFront OAI ARN uses spaces in the format and some uses underscores (_) in the ARN format.
We recommend using the CanonicalId when updating bucket policy to avoid this error message. To get the canonical ID, you can list the bucket policy or CF OAI and it will return the the AWS user name with the encrypted canonicalId "EXXXXXXXX" back in the output.

So, for example, you can run this:
$ aws cloudfront get-cloud-front-origin-access-identity --id

And it will return your canonical ID which you can add to your bucket policy and hit Save.
_

[nodomain]

Thanks for clarification here!

[davetapley]

Edit: Disregard this, see below 🎉

~FYI if you are using the iam_policy_document data source approach (i.e. not the Multiple Line Heredoc Syntax approach), then:~

~The CanonicalUser solution presented by bbrunod won't work because iam_policy_document doesn't allow CanonicalUser in the principal (it requires (https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html#type) 😞~

~But! The replace("${....}", " ", "_")] solution from dirgapeter does work 🙏~

[nodomain]

@dukedave this is not correct. This worked for me:

data "aws_iam_policy_document" "s3_policy" {
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.static.arn}/*"]

    principals {
      type        = "CanonicalUser"
      identifiers = [aws_cloudfront_origin_access_identity.origin_access_identity.s3_canonical_user_id]
    }
  }

  statement {
    actions   = ["s3:ListBucket"]
    resources = [aws_s3_bucket.static.arn]

    principals {
      type        = "CanonicalUser"
      identifiers = [aws_cloudfront_origin_access_identity.origin_access_identity.s3_canonical_user_id]
    }
  }
}

[bbrunod]

@dukedave

Maybe this regreplace works now, but, as I posted before, for how much time... :

_This means that old OAIs like "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EXXXXXXXXXXXX" will now look like this "arn:aws:iam::cloudfront:user/CloudFrontOriginAccessIdentityEXXXXXXXXXXXX". Users who try to hardcode either underscores or spaces into their bucket policy updates (not using CanonicalIds) will result in malformed principal, like the "invalid policy" error you’re getting_

[dirgapeter]

type = "CanonicalUser" works for me too.

with latest terraform and aws provider, just it also causes endless loop:

- AWS           = "arn:aws:iam::cloudfront:user/CloudFront_Origin_Access_Identity_..." -> null..
+ CanonicalUser = "..."

sometimes it is "CloudFront Origin Access Identity ", sometimes "CloudFront_Origin Access_Identity_"

[bbrunod]

And the funny thing is, even if you provide a CanonicalId, aws transforms with... a cloudfront origin access identity with underscores!

[parabolic]

principals {
      type        = "CanonicalUser"
      identifiers = [aws_cloudfront_origin_access_identity.origin_access_identity.s3_canonical_user_id]
    }

Works like a charm, I guess the docs should be updated. :1st_place_medal:

Or maybe not...
Now terraform always has a diff from the principal with underscores. I guess I am going with the

identifiers = [replace("${aws_cloudfront_origin_access_identity.website_bucket.iam_arn}", " ", "_")]

solution.

[IronCore864]

It seems after using type = "CanonicalUser", the policy created in the bucket is still the old format, but can be with either spaces or underscores. But at least the policy adding is successful now.

[simontabor]

Using CanonicalUser seems to work but results in a change for every deployment, which makes no sense (although there is a warning about this in the Terraform docs)

Using the replace method seems to work on some distributions, but not all.

[bgpat]

I had faced this problem, and workaround by https://github.com/terraform-providers/terraform-provider-aws/issues/10158#issuecomment-533486189.
Although I tried to create a new CloudFront distribution today, the API server returns the error MalformedPolicy: Invalid principal.... It looks only to accept the identifier contains not underscores but spaces now.

[DJAlPee]

We are using multiple AWS accounts (20+) and it seems to be a problem only for "newer" AWS accounts (at least for me).

On old accounts, it only works with aws_cloudfront_origin_access_identity.website_bucket.iam_arn. Replacing the blanks with underscores causes here an invalid principal error.

On new accounts it works only with replace(aws_cloudfront_origin_access_identity.website_bucket.iam_arn, " ", "_"), which is very annoying, if you try to apply the same config on multiple accounts

[saschakohnke]

Hey guys, any chance that there has something changed again?
I tried all the described workarounds here but still always receive the same error message.
using Terraform 0.12.21

Error putting S3 policy: MalformedPolicy: Invalid principal in policy