Terraform-provider-aws: Changes to aws_elasticsearch_domain not applying

Created on 13 Jun 2017  ·  10Comments  ·  Source: hashicorp/terraform-provider-aws

_This issue was originally opened by @rafaelmagu as hashicorp/terraform#3539. It was migrated here as part of the provider split. The original body of the issue is below._

Now that #3381 has been released, I've tried adding a couple of ES domains to our plans. We already had ES domains created manually via the AWS GUI, so I expected TF to warn me about a name conflict. Instead, it just added the two ARNs to its state.

I've then proceeded to change a snapshot setting, and TF hangs while trying to apply modifications:

2015/10/19 11:23:03 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:03 [TRACE] Waiting 1.6s before next try
2015/10/19 11:23:03 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:03 [TRACE] Waiting 1.6s before next try
2015/10/19 11:23:04 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:05 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:05 [TRACE] Waiting 3.2s before next try
2015/10/19 11:23:05 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:05 [TRACE] Waiting 3.2s before next try
2015/10/19 11:23:08 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:09 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:09 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:09 [TRACE] Waiting 6.4s before next try
2015/10/19 11:23:09 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:09 [TRACE] Waiting 6.4s before next try
2015/10/19 11:23:13 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:14 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:15 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:15 [TRACE] Waiting 10s before next try
2015/10/19 11:23:15 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:15 [TRACE] Waiting 10s before next try
2015/10/19 11:23:18 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:19 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:23 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:24 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:26 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:26 [TRACE] Waiting 10s before next try
2015/10/19 11:23:26 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:26 [TRACE] Waiting 10s before next try
2015/10/19 11:23:28 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:29 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:33 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:34 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:36 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:36 [TRACE] Waiting 10s before next try
2015/10/19 11:23:36 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:36 [TRACE] Waiting 10s before next try
2015/10/19 11:23:38 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:39 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:43 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:44 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:46 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:46 [TRACE] Waiting 10s before next try
2015/10/19 11:23:46 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:46 [TRACE] Waiting 10s before next try
2015/10/19 11:23:48 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:49 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:53 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:54 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:23:57 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:57 [TRACE] Waiting 10s before next try
2015/10/19 11:23:57 [DEBUG] terraform-provider-aws: 2015/10/19 11:23:57 [TRACE] Waiting 10s before next try
2015/10/19 11:23:58 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:23:59 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:03 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:04 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:07 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:07 [TRACE] Waiting 10s before next try
2015/10/19 11:24:07 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:07 [TRACE] Waiting 10s before next try
2015/10/19 11:24:08 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:09 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:13 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:14 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:17 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:17 [TRACE] Waiting 10s before next try
2015/10/19 11:24:17 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:17 [TRACE] Waiting 10s before next try
2015/10/19 11:24:18 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:19 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:23 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:24 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:27 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:27 [TRACE] Waiting 10s before next try
2015/10/19 11:24:28 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:28 [TRACE] Waiting 10s before next try
2015/10/19 11:24:28 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:29 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:33 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:34 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:38 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:38 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:38 [TRACE] Waiting 10s before next try
2015/10/19 11:24:38 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:38 [TRACE] Waiting 10s before next try
2015/10/19 11:24:39 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:43 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:44 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:48 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:48 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:48 [TRACE] Waiting 10s before next try
2015/10/19 11:24:48 [DEBUG] terraform-provider-aws: 2015/10/19 11:24:48 [TRACE] Waiting 10s before next try
2015/10/19 11:24:49 [DEBUG] vertex root, waiting for: provider.aws (close)
2015/10/19 11:24:53 [DEBUG] vertex provider.aws (close), waiting for: aws_elasticsearch_domain.elastic-log
2015/10/19 11:24:54 [DEBUG] vertex root, waiting for: provider.aws (close)
(...)

It is also detecting a change in access policies applied to the domains even though the policies are the same. I am assuming the latter is because it cannot apply the changes correctly, so it isn't saving the policies either.

bug upstream waiting-response
Source
picture hashibot

All 10 comments

_This comment was originally opened by @radeksimko as https://github.com/hashicorp/terraform/issues/3539#issuecomment-149595400. It was migrated here as part of the provider split. The original comment is below._

Hi @rafaelmagu
thanks for the report. Would you mind sharing any terraform code that will help us to reproduce this (minus any secrets)?

I'm specifically interested in the part where you've been adding existing ES domains into terraform and also some details about those ES domains - e.g. are those existing domains using the same region as your AWS provider defined in terraform code?

From the AWS docs:

Domain names are unique across the domains owned by an account within an AWS region.

hashibot picture hashibot on 13 Jun 2017

_This comment was originally opened by @rafaelmagu as https://github.com/hashicorp/terraform/issues/3539#issuecomment-149713525. It was migrated here as part of the provider split. The original comment is below._

I didn't plan to add the existing domain. I was expecting TF to fail when applying the plan, at which point I was going to remove the existing domains and let TF create new ones. They had the same ARN, so I assume that's why TF "imported" them into the state file.

Resource:

resource "aws_elasticsearch_domain" "elastic-log" {
  domain_name = "elastic-log"

  ebs_options {
    ebs_enabled = true
    volume_type = "gp2"
    volume_size = 512
  }

  cluster_config {
    instance_type = "r3.2xlarge.elasticsearch"
    instance_count = 3
    dedicated_master_enabled = false
    zone_awareness_enabled = true
  }

  snapshot_options {
    automated_snapshot_start_hour = 23
  }

  access_policies = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:${var.vpc_region}:1234567890:*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "1.2.3.4/32"
          ]
        }
      }
    }
  ]
}
EOF
}

Domains already exist in the same region as the provider defined.

hashibot picture hashibot on 13 Jun 2017

_This comment was originally opened by @dwradcliffe as https://github.com/hashicorp/terraform/issues/3539#issuecomment-166669832. It was migrated here as part of the provider split. The original comment is below._

I'm also having trouble making changes. TF just hangs forever.

hashibot picture hashibot on 13 Jun 2017

_This comment was originally opened by @gposton as https://github.com/hashicorp/terraform/issues/3539#issuecomment-186355089. It was migrated here as part of the provider split. The original comment is below._

I'm not having any issues updating the ES domains, but I can confirm the following:

It is also detecting a change in access policies applied to the domains even though the policies are the same. I am assuming the latter is because it cannot apply the changes correctly, so it isn't saving the policies either.

I'm seeing the same issue.... Looking at the output from the terraform apply, the access_policy is updated despite the fact that it is logically equivalent (no update should be required).

Although the access policy is logically equivalent, it is not syntactically equivalent (the order of the elements in the access policy json are different). I suspect this is why the update is being triggered.

I can reproduce this on each terraform apply.

Here is the output from the terraform apply:

[ats_dependencies] access_policies: "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-west-2:763429161784:domain/ats-prod/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"52.36.237.75/32\",\"52.32.171.250/32\",\"52.34.208.134/32\",\"50.112.131.3/32\",\"54.201.98.238/32\",\"54.69.54.64/32\",\"52.24.90.172/32\"]}}}]}" => "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"52.36.237.75/32\",\"52.32.171.250/32\",\"52.34.208.134/32\",\"50.112.131.3/32\",\"54.201.98.238/32\",\"54.69.54.64/32\",\"52.24.90.172/32\"]}},\"Effect\":\"Allow\",\"Principal\":\"*\"}],\"Version\":\"2012-10-17\"}"

hashibot picture hashibot on 13 Jun 2017

_This comment was originally opened by @gposton as https://github.com/hashicorp/terraform/issues/3539#issuecomment-186493824. It was migrated here as part of the provider split. The original comment is below._

Looking a bit closer at the output in the previous comment... it looks like the Resource ARN is being automatically injected into the access_policy. That explains why terraform see's the diff and triggers the modify. It would probably work to add the resource arn to the policy in the tf template after the initial creation of the ES domain (can't add it at the initial creation because you don't know what it is until after the resource is created).

hashibot picture hashibot on 13 Jun 2017

_This comment was originally opened by @radeksimko as https://github.com/hashicorp/terraform/issues/3539#issuecomment-186781732. It was migrated here as part of the provider split. The original comment is below._

@gposton Can we keep this thread focused on the original problem described by @rafaelmagu and not widen it to _any_ aws_elasticsearch_domain bug, please?

See https://github.com/hashicorp/terraform/issues/5067 which is describing the bug you mentioned. Thanks.

hashibot picture hashibot on 13 Jun 2017

Hi folks,
I believe the policy drift was resolved in https://github.com/terraform-providers/terraform-provider-aws/commit/ccd8d3e63fe50e3a5f364d756d6979c95b6f80a9 and https://github.com/terraform-providers/terraform-provider-aws/commit/5f190120eb9d0ca351afc899a00ce7a6950ede34 respectively.

In regards to hanging - ES domain will always be one of few types of those resources which take time to create/update/delete, simply because Terraform waits until the operation (creation/update/deletion) is completely finished. The time this takes may vary, but we've observed 40min+ isn't too unusual.

The timeout is set to 60 minutes https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_elasticsearch_domain.go#L249 and we'd be willing to either bump it or introduce custom timeouts if it times out for anyone. In such case having a debug log would be incredibly helpful - so we can verify the process genuinely took longer than 60mins and it's not just slow network or CPU or any other API-unrelated edge case.

Thirdly es/CreateElasticsearchDomain API method does not detect duplicate domain names, which is why @rafaelmagu experienced the odd behaviour. I have raised a question to AWS support about this to see if it's (undocumented) intention or a bug (that needs fixing on API's side).

radeksimko picture radeksimko on 15 Aug 2017
👍1

Regarding the third problem - AWS support folks came back to me with the following:

The ES team have confirmed that it is the documentation that is incorrect. They have made changes to the documentation which will be pushed to the public facing documentation with the next release. I am not able to comment on when exactly this will be.

So the easiest fix would be to ask API whether the domain exist prior to creating it. I'll see if I can find time to do this.

radeksimko picture radeksimko on 31 Aug 2017
👍1

Thank you for your diligent research into this issue.

rafaelmagu picture rafaelmagu on 31 Aug 2017

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] picture hashibot[bot] on 11 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

carmas picture carmas  ·  3Comments
dvishniakov picture dvishniakov  ·  3Comments
EmmN picture EmmN  ·  3Comments
hazmeister picture hazmeister  ·  3Comments
hashibot picture hashibot  ·  3Comments