Terraform-provider-aws: Terraform considers my elasticsearch domain policy to be modified every time it runs.

Created on 13 Jun 2017 · 28Comments · Source: hashicorp/terraform-provider-aws

_This issue was originally opened by @yeehaa123 as hashicorp/terraform#12390. It was migrated here as part of the provider split. The original body of the issue is below._

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.
8.8

Affected Resource(s)

Please list the resources as a list, for example:

aws_elasticsearch_domain_policy"
If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files


data "aws_iam_policy_document" "es" {
  statement {
    sid = "admin"
    principals = [
      {
        type        = "AWS"
        identifiers = ["*"]
      },
    ]
    actions = [
      "es:*"
    ]
    condition {
      test = "IpAddress"
      variable = "aws:SourceIp"
      values = [
        "95.97.17.4"
      ]
    }
  }

  statement {
    sid = "client"
    principals = [
      {
        type        = "AWS"
        identifiers = ["*"]
      },
    ]
    actions = [
      "es:ESHttpGet"
    ]
  }
}

resource "aws_elasticsearch_domain_policy" "main" {
  domain_name = "${aws_elasticsearch_domain.domain.domain_name}"
  access_policies = "${data.aws_iam_policy_document.es.json}"
}

Debug Output

Please provider a link to a GitHub Gist containing the complete debug output: https://www.terraform.io/docs/internals/debugging.html. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

What should have happened?

Actual Behavior

What actually happened?

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform apply

Important Factoids

Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? Custom version of OpenStack? Tight ACLs?

This behavior is especially problematic, because this resource takes a long time to modify. Min 15 min, up to an hour.
even though it's said to be fixed with pr 8648

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:
-5067
-8648

bug servicelasticsearch

Source

hashibot

👍7

All 28 comments

experiencing the same issue:

Terraform Version

$ terraform -v
Terraform v0.9.8

Affected Resource

aws_elasticsearch_domain

Terraform Configuration File

resource "aws_elasticsearch_domain" "sandbox" {
    domain_name                     = "sandbox"
    elasticsearch_version           = "5.3"

    cluster_config {
        instance_type               = "t2.medium.elasticsearch"
        instance_count              = 3
    }

    ebs_options {
        ebs_enabled                 = true
        volume_size                 = 20
        volume_type                 = "gp2"
    }

    advanced_options {
        "rest.action.multi.allow_explicit_index" = "true"
    }

    access_policies                 = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": { "aws:SourceIp": "${ aws_eip.nat.public_ip }/32" }
            }
        }
    ]
}
POLICY

    tags {
        Domain =  "sandbox"
    }
}

Debug Output

https://gist.github.com/sirlancealittle/0142c73aa7ace8ad6ea3f9db1bd6ab28

Expected Behavior

I would expect that this not force a change

Actual Behavior

A change is forced on every run

Steps to Reproduce

terraform plan || terraform apply

Interesting Factoids

It appears that a reference to the arn of the elasticsearch domain is being added by the provider during the apply, however when state is read back and compared to the inline access policy it is finding that the arn reference is missing and registering it as a change.

The first attempt to work around this was to add the arn reference to the policy but it appears inline policy does not allow self references.

Terraform Configuration

resource "aws_elasticsearch_domain" "sandbox" {
    domain_name                     = "sandbox"
    elasticsearch_version           = "5.3"

    cluster_config {
        instance_type               = "t2.medium.elasticsearch"
        instance_count              = 3
    }

    ebs_options {
        ebs_enabled                 = true
        volume_size                 = 20
        volume_type                 = "gp2"
    }

    advanced_options {
        "rest.action.multi.allow_explicit_index" = "true"
    }

    access_policies                 = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "${ self.arn }",     # << ADDED THIS
            "Condition": {
                "IpAddress": { "aws:SourceIp": "${ aws_eip.nat.public_ip }/32" }
            }
        }
    ]
}
POLICY

    tags {
        Domain = "Sandbox"
    }
}

Debug Output

$ tf plan

1 error(s) occurred:

* module root: 1 error(s) occurred:

* resource 'aws_elasticsearch_domain.sandbox' config: cannot contain self-reference self.arn

sirlancealittle on 14 Jun 2017

I, too, am experiencing this. its pretty harsh since changes to es domains take at least 10 minutes to apply... Is there any discussion somewhere as to a fix? Any workarounds?

brendanjerwin on 1 Sep 2017

duplicate of #205 ?

brendanjerwin on 1 Sep 2017

Here is an example of the policy changes its applying:

Before

 {
        "Statement": [
            {
                "Action": [
                    "es:RemoveTags",
                    "es:ListTags",
                    "es:ListDomainNames",
                    "es:ESHttpPut",
                    "es:ESHttpPost",
                    "es:ESHttpHead",
                    "es:ESHttpGet",
                    "es:ESHttpDelete",
                    "es:DescribeElasticsearchDomains",
                    "es:DescribeElasticsearchDomainConfig",
                    "es:DescribeElasticsearchDomain",
                    "es:AddTags"
                ],
                "Condition": {
                    "IpAddress": {
                        "aws:SourceIp": "10.0.0.0/16"
                    }
                },
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::11111111111111:root",
                        "arn:aws:iam::111111111111111:user/user1",
                        "arn:aws:iam::111111111111111:user/user2t"
                    ]
                },
                "Resource": "arn:aws:es:us-east-1:1111111111111111:domain/es-domain-here/*",
                "Sid": "system-access"
            }
        ],
        "Version": "2012-10-17"
    }

After

 {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "system-access",
                "Effect": "Allow",
                "Action": [
                    "es:RemoveTags",
                    "es:ListTags",
                    "es:ListDomainNames",
                    "es:ESHttpPut",
                    "es:ESHttpPost",
                    "es:ESHttpHead",
                    "es:ESHttpGet",
                    "es:ESHttpDelete",
                    "es:DescribeElasticsearchDomains",
                    "es:DescribeElasticsearchDomainConfig",
                    "es:DescribeElasticsearchDomain",
                    "es:AddTags"
                ],
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::11111111111111:root",
                        "arn:aws:iam::111111111111111:user/user1",
                        "arn:aws:iam::111111111111111:user/user2t"
                    ]
                },
                "Condition": {
                    "IpAddress": {
                        "aws:SourceIp": "10.0.0.0/16"
                    }
                }
            }
        ]
    }
}

Diff

screen shot 2017-09-01 at 7 54 04 am

brendanjerwin on 1 Sep 2017

Ah, I switched to using https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain_policy.html and it seems to have made it happy. (why is the other method of applying policies even allowed?)

brendanjerwin on 1 Sep 2017

I confirm using the ES domain resource policy fixed the same issue for me, thanks @brendanjerwin

sjourdan on 15 Sep 2017

I am also having this issue, see #2636. @brendanjerwin what is the workaround? Thanks.

nodesocket on 12 Dec 2017

@nodesocket "I switched to using https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain_policy.html"

brendanjerwin on 12 Dec 2017

@brendanjerwin darn, switching to a separate aws_elasticsearch_domain_policy resource did not fix it. Still rebuilding my ES cluster every time I invoke apply. I think it is because I am using an interpolation function jsonencode() which dynamically builds up JSON?

Any ideas or tricks to try?

variable "allowed_ips" {
    type = "list"
    description = "A list of allowed IPs that can connect to the Elasticsearch cluster"
}

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG
}

nodesocket on 13 Dec 2017

Wanted to add the plan to show what Terraform is saying is different. The two JSON's appears to be identical, excluding whitespace. Any ideas?

  ~ module.elasticsearch.aws_elasticsearch_domain.elasticsearch
      access_policies:                      "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev/*\"}],\"Version\":\"2012-10-17\"}" => "    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Action\": [\n                    \"es:*\"\n                ],\n                \"Principal\": {\n                    \"AWS\": \"*\"\n                },\n                \"Effect\": \"Allow\",\n                \"Condition\": {\n                    \"IpAddress\": {\"aws:SourceIp\": [\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}\n                }\n            }\n        ]\n    }\n    "

nodesocket on 15 Dec 2017

the issue is Aws reformats your policy and therefore terraform reports it as a change. use the policy that aws reformats after you terraform and update your policy in terraform. it’s an old known issue.

Puneeth-n on 17 Dec 2017

@Puneeth-n can you point to the issue that describes this? We should try and get a fix. Just FYI, I tried using ignore_changes on the aws_elasticsearch_domain_policy but still having problems. See https://github.com/hashicorp/terraform/issues/16930#issuecomment-352135009

nodesocket on 17 Dec 2017

https://github.com/hashicorp/terraform/issues/3634#issuecomment-312816843

Puneeth-n on 17 Dec 2017

Update

I was able to work around this issue by using:

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }

But even breaking the policy into a separate aws_elasticsearch_domain_policy resource still causes a rebuild of the Elasticsearch cluster each apply:

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG
}

nodesocket on 23 Jan 2018

👍1

i confirm using aws_elasticsearch_domain_policy solved that issue for me

ebarault on 8 Feb 2018

It's the order of the elements ("version" moved to the end) that causes the (apparent) change. Using an inline-policy you might be able to work around this, but when using aws_iam_policy_document data to create a policy you are lost -> this is a bug.

frittentheke on 7 Mar 2018

aws_elasticsearch_domain_policy did not prevent a modify on every plan for me on 1.18.0. I tried inlining, loading from a file, and rendering from a template. All approaches appear to be adding line breaks and white space which is causing the diff:

  ~ module.elasticsearch.aws_elasticsearch_domain_policy.es
      access_policies:                                    "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":[\"es:ESHttpPut\",\"es:ESHttpPost\",\"es:ESHttpHead\",\"es:ESHttpGet\"],\"Resource\":\"arn:aws:es:us-east-1:615190663081:domain/staging-knotch/*\"}]}" => "  {\n      \"Version\": \"2012-10-17\",\n      \"Statement\": [\n          {\n              \"Action\": [\n                \"es:ESHttpPut\",\n                \"es:ESHttpPost\",\n                \"es:ESHttpHead\",\n                \"es:ESHttpGet\"\n              ],\n              \"Principal\": \"*\",\n              \"Effect\": \"Allow\"\n          }\n      ]\n  }\n  "

adamgotterer on 16 May 2018

@adamgotterer the plan looks like the AWS API is returning a Resource field ("Resource": "arn:aws:es:us-east-1:615190663081:domain/staging-knotch/*") and your configuration is trying to remove it. What happens if you add this to your policy configuration?

bflad on 16 May 2018

@bflad adding resource fixed it for me. Thanks!

adamgotterer on 16 May 2018

@adamgotterer I still encounter this problem, every time Terraform modifies the aws_elasticsearch_domain_policy. Where did you add resource to fix it?

nodesocket on 16 May 2018

@nodesocket my resources look like:

resource "aws_elasticsearch_domain" "es" {
 ...
}

resource "aws_elasticsearch_domain_policy" "es" {
  domain_name = "${aws_elasticsearch_domain.es.domain_name}"
  access_policies = <<POLICIES
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                "es:ESHttpPut",
                "es:ESHttpPost",
                "es:ESHttpHead",
                "es:ESHttpGet"
              ],
              "Principal": "*",
              "Effect": "Allow",
              "Resource": "${aws_elasticsearch_domain.es.arn}/*"
          }
      ]
  }
  POLICIES
}

I removed access_policies from aws_elasticsearch_domain and migrated to aws_elasticsearch_domain_policy. When looking at the plan diff I noticed "/*" was being added to the resource path, so double check for that.

adamgotterer on 16 May 2018

@adamgotterer yeah, still seeing aws_elasticsearch_domain_policy being updated every apply. Ideas?

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Principal": {
                "AWS": "*"
            },
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
            }
        }
    ]
}
CONFIG
}

nodesocket on 16 May 2018

@nodesocket you are missing the resource key.

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Principal": {
                "AWS": "*"
            },
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
            },
            "Resource": "${aws_elasticsearch_domain. elasticsearch.arn}" #<==========
        }
    ]
}
CONFIG
}

adamgotterer on 16 May 2018

Adding resource stopped Terrform from changing the policy each apply, but it also broke my access to Elasticsearch. Note, my Terraform definition I tried was:

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                },
                "Resource": "${aws_elasticsearch_domain.elasticsearch.arn}"
            }
        ]
    }
CONFIG

nodesocket on 17 May 2018

The trick is:

"Resource": "${aws_elasticsearch_domain.elasticsearch.arn}/*"

Adding /*. Sigh. Thanks for the help @adamgotterer

nodesocket on 17 May 2018

I think this can be closed because people seem to be happy with either using the aws_elasticsearch_domain_policy resource or you can build the resource ARN for the access_policy parameter in the aws_elasticsearch_domain resource manually with something like this:

resource "aws_elasticsearch_domain" "es" {
  domain_name           = "${var.domain}"
  ...

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ["66.193.100.22/32"]}
            },
            "Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.domain}/*"
        }
    ]
}
CONFIG
}

I'll raise a PR for the docs to add the resource element to make it a bit easier for future users.

tomelliff on 22 Jun 2018

🎉1

Documentation updated via #4942 and will release with version 1.25.0 of the AWS provider, likely middle of next week. If there are further issues here, please open a new issue. Thanks!

bflad on 22 Jun 2018

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] on 5 Apr 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

providers/aws aws_instance not importing network interfaces

ghost · 3Comments

Provide releases more often

dvishniakov · 3Comments

Feature: S3 bucket-wide default encryption

reedloden · 3Comments

Using security_groups instead of the correct vpc_security_group_ids on an instance within VPC results in instance recreation on each apply

andywirv · 3Comments

aws_elasticsearch_domain: vpc_options - subnet_ids asks for list but can only be a single value.

ccslamstack · 3Comments

Terraform-provider-aws: Terraform considers my elasticsearch domain policy to be modified every time it runs.

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

All 28 comments

Terraform Version

Affected Resource

Terraform Configuration File

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Interesting Factoids

Terraform Configuration

Debug Output

Before

After

Diff

Related issues