Terraform-provider-aws: aws_elasticsearch_domain cognito_options cause Cycle Error

[ovomason]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

• provider.aws v1.31.0

Affected Resource(s)

aws_elasticsearch_domain
aws_cognito_user_pool
aws_cognito_user_pool_client
aws_cognito_identity_pool

Terraform Configuration Files

resource "aws_elasticsearch_domain" "elasticsearch-cognito" {
  domain_name = "test-ovowebsite-${var.environment}"
  elasticsearch_version = "6.2"
  cluster_config {

    instance_count = "${var.elasticsearch_instance_count}"
    instance_type = "${var.elasticsearch_instance_type}"
    zone_awareness_enabled = "${var.elasticsearch_zone_awareness}"
    dedicated_master_enabled = "${var.elasticsearch_dedicated_master_enabled}"
    dedicated_master_count = "${var.elasticsearch_master_count}"
    dedicated_master_type = "${var.elasticsearch_master_type}"
  }

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
              "AWS": ["*"]
            },
            "Action": ["es:*"],
            "Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/test-ovowebsite-${var.environment}/*"
        }
    ]
}
  CONFIG

  ebs_options {
    ebs_enabled = true
    volume_type = "gp2"
    volume_size = "10"
  }
  encrypt_at_rest {
    enabled = true
    kms_key_id = "${aws_kms_key.es_key.id}"
  }
  snapshot_options {
    automated_snapshot_start_hour = "${var.elasticsearch_snapshot_time}"
  }
  vpc_options {
    subnet_ids = ["${aws_subnet.protected_subnet1.id}","${aws_subnet.protected_subnet2.id}"]
    security_group_ids = ["${aws_security_group.elasticsearch.id}"]
  }
  cognito_options {
    enabled = true
    user_pool_id = "${aws_cognito_user_pool.kibana.id}"
    identity_pool_id = "${aws_cognito_identity_pool.kibana.id}"
    role_arn = "${aws_iam_role.CustomESCognitoAccess.arn}"
  }
  tags {
    Name = "ovowebsite-${var.environment}"
    Team = "${var.team}"
  }
}

// cognito setup
// create user pool
resource "aws_cognito_user_pool" "kibana" {
  name = "kibana user pool"
  auto_verified_attributes = ["email"]
  admin_create_user_config = {
    allow_admin_create_user_only = false
  }
  schema {
    attribute_data_type = "String"
    name = "email"
    required = true
  }
  alias_attributes = ["email"]
}
// set user pool domain
resource "aws_cognito_user_pool_domain" "kibana" {
  domain = "ovo-kibana-login"
  user_pool_id = "${aws_cognito_user_pool.kibana.id}"
}

// set user pool client
resource "aws_cognito_user_pool_client" "kibana" {
  name = "elasticsearch"
  user_pool_id = "${aws_cognito_user_pool.kibana.id}"
  generate_secret = true
  callback_urls = ["${aws_elasticsearch_domain.elasticsearch-cognito.kibana_endpoint}"]
  logout_urls = ["${aws_elasticsearch_domain.elasticsearch-cognito.kibana_endpoint}"]
  allowed_oauth_flows = ["code", "implicit"]
  allowed_oauth_scopes = ["email", "openid"]
}

//create identity pool
resource "aws_cognito_identity_pool" "kibana" {
  identity_pool_name               = "kibana pool"
  allow_unauthenticated_identities = false

  cognito_identity_providers {
    client_id               = "${aws_cognito_user_pool_client.kibana.id}"
    provider_name           = "${aws_cognito_user_pool.kibana.endpoint}"
    server_side_token_check = false
  }
}

Debug Output

Cycle: module.common.aws_cognito_user_pool_client.kibana, module.common.aws_cognito_identity_pool.kibana, module.common.aws_elasticsearch_domain.elasticsearch-cognito

Expected Behavior

I need to create a user pool, identity pool and user client (with the Elasticsearch Kibana Endpoint as the callback_url) and create the elasticsearch resource. Then enable Elasticsearch Cognito with the user pool id and identity pool id.

Actual Behavior

Creates an Error:
Cycle: module.common.aws_cognito_user_pool_client.kibana, module.common.aws_cognito_identity_pool.kibana, module.common.aws_elasticsearch_domain.elasticsearch-cognito

Steps to Reproduce

Create an 'aws_cognito_user_pool' resource
Create an 'aws_cognito_identity_pool' resource with 'cognito_identity_providers' attribute
Create an 'aws_elasticsearch_domain' resource with 'cognito_options' attribute
Create an 'aws_cognito_user_pool_client' resource with the elasticsearch kibana endpoint

Important Factoids

We require the kibana endpoint to redirect users to kibana after login.

I propose abstracting the cognito options for elasticsearch into its own resource. This would be the minimum required to get this working i believe.

It may also be work creating a new resource that can attach a new cognito identity provider?

[jleskovar-tyro]

Hi there. Putting aside the cyclic dependency issue you mention, my experience with ElasticSearch and cognito so far is that it simply isn't possible to have Terraform manage cognito with ElasticSearch. The problem I've been having is that enabling Cognito auth for ES results in AWS going rogue and creating its own application client on the user pool you specify, as well as adding this client as an auth provider on your identity pool. My current workaround is to use the AWS CLI to configure cognito, after using Terraform to enable cognito on the ElasticSearch domain. Doing this, you'll need to use "ignore_changes" on your identity pool to prevent TF from blowing away the AWS-managed app client

[Geartrixy]

Hello,

as @jleskovar-tyro mentioned, when you enable cognito authentication on ES, AWS will automatically create a new cognito app client in the background.

This renders the app client you want created via Terraform to be used via ES useless, as the configuration on ES references the AWS created app client I think?

The only work around I can think of is to data source the app client that gets created via AWS.

But then this would require two deployments, as the app client cannot be data sourced until it is created, so you would have to comment out the "cognito_identity_providers" block in the "aws_cognito_identity_pool" resource first (and any roles attachment resources). Deploy to enable cognito authentication for kibana on ES. Then remove the commented out code (adding the data sourced app client to the configuration) and deploy again?

[Geartrixy]

ah, this is not possible: https://www.terraform.io/docs/providers/aws/d/cognito_user_pool_client.html

[bnr242003]

I am facing the similar issues. Two user pools are being created - one by Terraform and another by ES while creating the cluster and unfortunately the callback and signout URL's getting assigned to the user pool created by ES and that user pool is got getting attached to the Cognito Identity provider.

[bnr242003]

Some quick solution will be appreciated. Please let me know if you need anymore clarifications.

[jleskovar-tyro]

@bnr242003
What we ended up doing was using terraform to still manage our aws_elasticsearch_domain with Cognito enabled, together with our Cognito related resources (i.e. aws_cognito_user_pool, aws_cognito_user_pool_domain, aws_cognito_identity_provider, and aws_cognito_identity_pool).

After doing a tf plan/apply, our pipeline will run a post-apply.sh script, which uses the AWS CLI to directly modify the UserPoolClient AWS automatically creates for you, in order to support our SAML identity provider.

See this gist for an example of how that works. Obviously, your use case may be different:
https://gist.github.com/jleskovar-tyro/051317370190394d551f8861fb760f68

If you use this approach, you'll need to be sure to add the appropriate lifecycle.ignore_changes to any TF resources that are modified by the post-apply.sh script, or AWS itself. In our case, we had to ignore changes to cognito_identity_providers on our TF-managed aws_cognito_identity_pool resource, as otherwise the next tf plan/apply would revert the UserPoolClient AWS automatically creates and attaches for you. Hope that makes sense

[phr3nzii]

Any updates on this issue? We're experiencing this, I'd rather not ignore changes on resources or use a script to make changes after tf apply as part of CI.

[umeat]

I tried to be cheeky and work around the behaviour of AWS creating those resources.

If you only give the role (aws_elasticsearch_domain.cognito_options.role_arn) access to cognito-idp:Describe* and cognito-identity:Describe*, the AWS API accepts the configuration without issue and doesn't go on to create any resources in the background.

However, now my Domain Status is stuck in Processing state and it hasn't actually updated the Cognito settings (I assume it's stuck retrying the calls to create the Client and Identity Provider).

I guess this half of the issue is with AWS.

You would still have the circular dependency issue in terraform.

[denibertovic]

@jleskovar-tyro Still using the post install approach? Do you happen to have an example of the terraform code as well?

[lapkritinis]

Looks like I was able to work around it. Only downside, that you need to hardcode domain name:

data "aws_elasticsearch_domain" "es-domain" {
domain_name = "your-elasticsearch-domain"
}

callback_urls = [
"https://${data.aws_elasticsearch_domain.es-domain.endpoint}app/kibana"
]

[jleskovar-tyro]

@denibertovic still using the post-apply script. Sorry for late reply. Let me distill a working sample for you in a gist

@lapkritinis the main issue though is when you want to have Kibana enabled with cognito auth, that’s when you start running into problems

[lapkritinis]

Thats what I have - elastic search with cognito auth.

On 2020-05-17, Sun at 22:02, jleskovar-tyro [e-mail]
wrote:

@denibertovic https://github.com/denibertovic still using the
post-apply script. Sorry for late reply. Let me distill a working sample
for you in a gist

@lapkritinis https://github.com/lapkritinis the main issue though is
when you want to have Kibana enabled with cognito auth, that’s when you
start running into problems

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/terraform-providers/terraform-provider-aws/issues/5557#issuecomment-629844738,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADVYNANZNT7IXTWVI57K3ZLRSAYEZANCNFSM4FPYTJGQ
.

[jleskovar-tyro]

Hmm, were you using a custom user pool client, or just relying on the AWS managed user pool? The problem occurs due to AWS creating and attaching a user pool for you, which is a problem if you want a user pool backed by a custom identity provider (e.g. SAML provider)

[msneller]

I think this issue could be solved if terraform had a data source for Cognito client app (with support for regex filter on name) and an independent resource for adding Cognito identity providers to an existing identity pool.

[KarstenSiemer]

We also do have this problem. Most logical solution for me would be to be able to reference the the client at the elastic resource. Shall we open an issue at aws? They would have to patch this on aws side also

[dcepelik]

@radeksimko Hello again, I can confirm we're also affected by this issue. Is there anything we can do to move forward with this?

[leoskyrocker]

@msneller @KarstenSiemer even in that case you'd end up with having to applying twice right? Since data source cannot be "not found" it will complain the very first time you're enabling Cognito.
Refer to https://github.com/hashicorp/terraform-provider-aws/issues/5557#issuecomment-478562921