Terraform-provider-aws: ERROR when creating an Aurora RDS global cluster and snapshot_identifier is defined

[jamengual]

Hi.

I have been working with terraform to create RDS global cluster without many issues until now.

I'm using the same code I use to create my prod global cluster to create another cluster base on the original prod cluster snapshot but when snapshot_identifier is provided the cluster gets created as a regional cluster and it is not attached to the newly created global cluster BUT if I use exactly the same code without specifing the snapshot_identifier the global cluster is created and the new regional rds cluster gets atached inmediatly to the global cluster.

Exactly the sam behavior happens when using the console but in the console I can successfully create the global cluster from the snapshot.

Keep in mind that I replaced some text to hide personal information

the sample code :

# Global mydata RDS cluster

resource "aws_rds_global_cluster" "mydata_clone" {
  count                     = var.create_clone ? 1 : 0
  engine_version            = "5.6.10a"
  global_cluster_appntifier = "clone-test-mydata-global"
  storage_encrypted         = true
  deletion_protection       = false
  provider                  = aws.primary
}


module "test_mydata_us_east_2_clone_cluster" {
  source         = "git::https://github.com/cloudposse/terraform-aws-rds-cluster.git?ref=0.17.0"
  enabled        = var.create_clone
  engine         = "aurora"
  engine_version = "5.6.10a"
  cluster_family = "aurora5.6"
  cluster_size   = 1
  namespace      = var.namespace
  stage          = var.environment
  name           = "us-east-2-${var.mydata_name}-clone"
#   admin_user     = var.mydata_db_user
#   admin_password = random_string.db_password.result
  db_name        = var.mydata_db_name
  instance_type  = "db.r5.2xlarge"
  vpc_id         = local.vpc_id
  security_groups = [
    local.sg-web-server-us-east-2,
    local.sg-app-scan-us-east-2,
    local.sg-app-scan-us-east-2,
    local.sg-app-scan-us-east-2
  ]
  allowed_cidr_blocks                 = var.mydata_allowed_cidr_blocks
  subnets                             = local.private_subnet_ids
  engine_mode                         = "global"
  global_cluster_appntifier           = join("", aws_rds_global_cluster.mydata_clone.*.id)
  iam_database_authentication_enabled = true
  storage_encrypted                   = true
  deletion_protection                 = false
  iam_roles                           = ["${aws_iam_role.AuroraAccessToDataBuckets.arn}"]
  ##enabled_cloudwatch_logs_exports     = ["audit", "error", "general", "slowquery"]
  tags                                = local.complete_tags
  snapshot_appntifier                 = var.snapshot_appntifier
  skip_final_snapshot                 = true

  # DNS setting
  cluster_dns_name = "test-${var.environment}-mydata-writer-clone-us-east-2"
  reader_dns_name  = "test-${var.environment}-mydata-reader-clone-us-east-2"
  zone_id          = data.aws_route53_zone.ds_example_com.zone_id

  # enable monitoring every 30 seconds
  ##rds_monitoring_interval = 15

  # reference iam role created above
  ##rds_monitoring_role_arn      = aws_iam_role.mydata_enhanced_monitoring.arn
  ##performance_insights_enabled = true

  cluster_parameters = [
    {
      name         = "binlog_format"
      value        = "row"
      apply_method = "pending-reboot"
    },
    {
      apply_method = "immediate"
      name         = "max_allowed_packet"
      value        = "16777216"
    },
    {
      apply_method = "pending-reboot"
      name         = "performance_schema"
      value        = "1"
    },
    {
      apply_method = "immediate"
      name         = "server_audit_logging"
      value        = "0"
    }
  ]
  providers = {
    aws = aws.primary
  }
}

Plan output :

```# aws_rds_global_cluster.mydata_clone[0] will be created
+ resource "aws_rds_global_cluster" "mydata_clone" {
+ arn = (known after apply)
+ deletion_protection = false
+ engine = "aurora"
+ engine_version = "5.6.10a"
+ global_cluster_identifier = "clone-test-mydata-global"
+ global_cluster_resource_id = (known after apply)
+ id = (known after apply)
+ storage_encrypted = true
}

# module.test_mydata_us_east_2_clone_cluster.aws_db_parameter_group.default[0] will be created
+ resource "aws_db_parameter_group" "default" {
    + arn         = (known after apply)
    + description = "DB instance parameter group"
    + family      = "aurora5.6"
    + id          = (known after apply)
    + name        = "test-staging-us-east-2-mydata-clone"
    + name_prefix = (known after apply)
    + tags        = {
        + "Name"           = "test-staging-us-east-2-mydata-clone"
        + "Namespace"      = "test"
        + "Stage"          = "staging"
        + "environment"    = "staging"
        + "expiration"     = "never"
      }
  }

# module.test_mydata_us_east_2_clone_cluster.aws_db_subnet_group.default[0] will be created
+ resource "aws_db_subnet_group" "default" {
    + arn         = (known after apply)
    + description = "Allowed subnets for DB cluster instances"
    + id          = (known after apply)
    + name        = "test-staging-us-east-2-mydata-clone"
    + name_prefix = (known after apply)
    + subnet_ids  = [
        + "subnet-1111111111111
        + "subnet-1111111111111
        + "subnet-1111111111111
      ]
    + tags        = {
        + "Name"           = "test-staging-us-east-2-mydata-clone"
        + "Namespace"      = "test"
        + "Stage"          = "staging"
        + "environment"    = "staging"
        + "expiration"     = "never"
      }
  }

# module.test_mydata_us_east_2_clone_cluster.aws_rds_cluster.default[0] will be created
+ resource "aws_rds_cluster" "default" {
    + apply_immediately                   = true
    + arn                                 = (known after apply)
    + availability_zones                  = (known after apply)
    + backup_retention_period             = 5
    + cluster_identifier                  = "test-staging-us-east-2-mydata-clone"
    + cluster_identifier_prefix           = (known after apply)
    + cluster_members                     = (known after apply)
    + cluster_resource_id                 = (known after apply)
    + copy_tags_to_snapshot               = false
    + database_name                       = "testdb"
    + db_cluster_parameter_group_name     = "test-staging-us-east-2-mydata-clone"
    + db_subnet_group_name                = "test-staging-us-east-2-mydata-clone"
    + deletion_protection                 = false
    + enabled_cloudwatch_logs_exports     = []
    + endpoint                            = (known after apply)
    + engine                              = "aurora"
    + engine_mode                         = "global"
    + engine_version                      = "5.6.10a"
    + final_snapshot_identifier           = "test-staging-us-east-2-mydata-clone"
    + global_cluster_identifier           = (known after apply)
    + hosted_zone_id                      = (known after apply)
    + iam_database_authentication_enabled = true
    + iam_roles                           = [
        + "arn:aws:iam::1111111111:role/AuroraAccessToDataBuckets",
      ]
    + id                                  = (known after apply)
    + kms_key_id                          = (known after apply)
    + master_username                     = "admin"
    + port                                = (known after apply)
    + preferred_backup_window             = "07:00-09:00"
    + preferred_maintenance_window        = "wed:03:00-wed:04:00"
    + reader_endpoint                     = (known after apply)
    + skip_final_snapshot                 = true
    + snapshot_identifier                 = "snapshot-prep-for-data-load"
    + storage_encrypted                   = true
    + tags                                = {
        + "Name"           = "test-staging-us-east-2-mydata-clone"
        + "Namespace"      = "test"
        + "Stage"          = "staging"
        + "environment"    = "staging"
        + "expiration"     = "never"
      }
    + vpc_security_group_ids              = (known after apply)
  }

# module.test_mydata_us_east_2_clone_cluster.aws_rds_cluster_instance.default[0] will be created
+ resource "aws_rds_cluster_instance" "default" {
    + apply_immediately               = (known after apply)
    + arn                             = (known after apply)
    + auto_minor_version_upgrade      = true
    + availability_zone               = (known after apply)
    + cluster_identifier              = (known after apply)
    + copy_tags_to_snapshot           = false
    + db_parameter_group_name         = "test-staging-us-east-2-mydata-clone"
    + db_subnet_group_name            = "test-staging-us-east-2-mydata-clone"
    + dbi_resource_id                 = (known after apply)
    + endpoint                        = (known after apply)
    + engine                          = "aurora"
    + engine_version                  = "5.6.10a"
    + id                              = (known after apply)
    + identifier                      = "test-staging-us-east-2-mydata-clone-1"
    + identifier_prefix               = (known after apply)
    + instance_class                  = "db.r5.2xlarge"
    + kms_key_id                      = (known after apply)
    + monitoring_interval             = 0
    + monitoring_role_arn             = (known after apply)
    + performance_insights_enabled    = false
    + performance_insights_kms_key_id = (known after apply)
    + port                            = (known after apply)
    + preferred_backup_window         = (known after apply)
    + preferred_maintenance_window    = (known after apply)
    + promotion_tier                  = 0
    + publicly_accessible             = false
    + storage_encrypted               = (known after apply)
    + tags                            = {
        + "Name"           = "test-staging-us-east-2-mydata-clone"
        + "Namespace"      = "test"
        + "Stage"          = "staging"
        + "environment"    = "staging"
        + "expiration"     = "never"
      }
    + writer                          = (known after apply)
  }

```

Version :
terraform_0.12.16
provider "local" (hashicorp/local) 1.4.0
provider "aws" (hashicorp/aws) 2.38.0...
provider "null" (hashicorp/null) 2.1.2...
provider "template" (hashicorp/template) 2.1.2
provider "mysql" (terraform-providers/mysql) 1.9.0
provider "random" (hashicorp/random) 2.2.1

Expected Behavior

new global cluster should be created and a new rds cluster should have been attached to the global cluster after being created from the snapshot.

Actual Behavior

A global cluster is created and a standalone RDS cluster is created from the snapshot but the RDS cluster is not attached to the Global cluster

When created without using snapshot_identifier the global cluster an RDS clusters are created correctly.

the error when trying to re-apply the terraform is :

Error: Existing RDS Clusters cannot be added to an existing RDS Global Cluster

[jamengual]

Hi, anything I can help for someone to take a look at this ?

[marinsalinas]

Hi, anything I can help for someone to take a look at this ?

Maybe @bflad since he did last commit on RDS resources.

[bflad]

Hi folks 👋 My focus is elsewhere at the moment, but in brief searching around this topic:

• The RestoreDBClusterFromSnapshot API call does not accept a GlobalClusterIdentifier
• In the RDS User Guide, it suggests that data must be restored to the primary cluster of a global database. The closest equivalent during creation would be adding the SourceDBClusterIdentifier parameter of the CreateGlobalCluster API call

This represents a little bit of a problem in Terraform though as it'll introduce a circular reference (to do it properly):

# Potential implementation
# The source_db_cluster_identifier argument does not currently exist
resource "aws_rds_global_cluster" "example" {
  # ... other configuration ...
  source_db_cluster_identifier = aws_rds_cluster.example.id
}

resource "aws_rds_cluster" "example" {
  snapshot_identifier = aws_db_cluster_snapshot.example.id

  # NOTE: Due to restoring this Cluster from Cluster Snapshot
  # and using this Cluster to create a Global Cluster, the
  # global_cluster_identifier attribute will become populated and
  # Terraform will begin showing it as a difference. We cannot do:
  # global_cluster_identifier = aws_rds_global_cluster.example.id
  # as it introduces a circular reference. As a potential workaround:
  lifecycle {
    ignore_changes = [global_cluster_identifier]
  }
}

This workaround would need to be documented to reduce potential confusion around the subject. Another option would be to mark the global_cluster_identifier argument in the aws_rds_cluster resource as Computed: true and annotating the documentation with:

* global_cluster_identifier - (Optional) The global cluster identifier specified on [`aws_rds_global_cluster`](/docs/providers/aws/r/rds_global_cluster.html). Terraform will only perform drift detection if this argument has a value. If using this cluster to restore a snapshot to create a global cluster, this should be omitted to prevent a circular reference.

Hope this helps.

EDIT: Please note that this is marked as an enhancement since this is a feature request to support the source_db_cluster_identifier argument on the aws_rds_global_cluster resource, which appears to be how RDS expects this to be implemented.

[marinsalinas]

Hi folks 👋 My focus is elsewhere at the moment, but in brief searching around this topic:

• The RestoreDBClusterFromSnapshot API call does not accept a GlobalClusterIdentifier
• In the RDS User Guide, it suggests that data must be restored to the primary cluster of a global database. The closest equivalent during creation would be adding the SourceDBClusterIdentifier parameter of the CreateGlobalCluster API call

This represents a little bit of a problem in Terraform though as it'll introduce a circular reference (to do it properly):

# Potential implementation
# The source_db_cluster_identifier argument does not currently exist
resource "aws_rds_global_cluster" "example" {
  # ... other configuration ...
  source_db_cluster_identifier = aws_rds_cluster.example.id
}

resource "aws_rds_cluster" "example" {
  snapshot_identifier = aws_db_cluster_snapshot.example.id

  # NOTE: Due to restoring this Cluster from Cluster Snapshot
  # and using this Cluster to create a Global Cluster, the
  # global_cluster_identifier attribute will become populated and
  # Terraform will begin showing it as a difference. We cannot do:
  # global_cluster_identifier = aws_rds_global_cluster.example.id
  # as it introduces a circular reference. As a potential workaround:
  lifecycle {
    ignore_changes = [global_cluster_identifier]
  }
}

This workaround would need to be documented to reduce potential confusion around the subject. Another option would be to mark the global_cluster_identifier argument in the aws_rds_cluster resource as Computed: true and annotating the documentation with:

* global_cluster_identifier - (Optional) The global cluster identifier specified on [`aws_rds_global_cluster`](/docs/providers/aws/r/rds_global_cluster.html). Terraform will only perform drift detection if this argument has a value. If using this cluster to restore a snapshot to create a global cluster, this should be omitted to prevent a circular reference.

Hope this helps.

EDIT: Please note that this is marked as an enhancement since this is a feature request to support the source_db_cluster_identifier argument on the aws_rds_global_cluster resource, which appears to be how RDS expects this to be implemented.

Thanks @bflad, I have another question, how this will work with multiple secondary clusters?

[bflad]

Hi @marinsalinas 👋 I'm not sure I understand the question, could you elaborate?

[marinsalinas]

Hey, @bflad, I can provide more explanation on this:

Since the potential implementation is:

# Potential implementation
# The source_db_cluster_identifier argument does not currently exist
resource "aws_rds_global_cluster" "example" {
  # ... other configuration ...
  source_db_cluster_identifier = aws_rds_cluster.example.id
}

resource "aws_rds_cluster" "example" {
  snapshot_identifier = aws_db_cluster_snapshot.example.id

  # NOTE: Due to restoring this Cluster from Cluster Snapshot
  # and using this Cluster to create a Global Cluster, the
  # global_cluster_identifier attribute will become populated and
  # Terraform will begin showing it as a difference. We cannot do:
  # global_cluster_identifier = aws_rds_global_cluster.example.id
  # as it introduces a circular reference. As a potential workaround:
  lifecycle {
    ignore_changes = [global_cluster_identifier]
  }
}

How will we add secondary clusters into the configuration?

I'm thinking something like:

# Potential implementation
# The source_db_cluster_identifier argument does not currently exist
resource "aws_rds_global_cluster" "example" {
  # ... other configuration ...
  source_db_cluster_identifier = aws_rds_cluster.example.id
}

resource "aws_rds_cluster" "example" {
  snapshot_identifier = aws_db_cluster_snapshot.example.id

  # NOTE: Due to restoring this Cluster from Cluster Snapshot
  # and using this Cluster to create a Global Cluster, the
  # global_cluster_identifier attribute will become populated and
  # Terraform will begin showing it as a difference. We cannot do:
  # global_cluster_identifier = aws_rds_global_cluster.example.id
  # as it introduces a circular reference. As a potential workaround:
  lifecycle {
    ignore_changes = [global_cluster_identifier]
  }
}

provider "aws" {
  region = "us-west-2"
  alas   = "uw2"
}

//My point of view and how to add secondary clusters.
resource "aws_rds_cluster" "secondary" {
  provider                  = aws.uw2
  global_cluster_identifier = aws_rds_global_cluster.example.id

  depends_on = [aws_rds_cluster.example]
}

is that correct?

[bflad]

@marinsalinas that would be the expected configuration, potentially without the ignore_changes depending on the final implementation 👍 (you could also drop the depends_on, its transitively aws_rds_cluster.secondary -> aws_rds_global_cluster.example -> aws_rds_cluster.example already).

[chartierf]

I have the exact same problem.

When i create a global cluster from scratch everythings works but this is not what i need to do.

I have an existing DB which need to be replicated in another region.
With Terraform after many tests no way to do this.
Last error : Existing RDS Clusters cannot be added to an existing RDS Global Cluster

I see at least 2 ways:

• Create a new global cluster and attach existing DB to it
• Create a new global cluster and a new DB from existing Snapshot

First way is what AWS Console do via 'add region'.

I think @bflad has pointed the main problem : Terraform AWS provider actually miss SourceDBClusterIdentifier parameter to connect the global cluster with an existing cluster.
You can create a global database that is initially empty, and then add a primary cluster and a secondary cluster to it. Or you can specify an existing Aurora cluster during the create operation, and this cluster becomes the primary cluster of the global database.

[phani308]

@bflad Hello ,

I am facing a similar issue when trying to attach the existing RDS cluster to the Global cluster. terraform errors out as follow.
Error: Existing RDS Clusters cannot be added to an existing RDS Global Cluster

AWS does offer API/CLI/SDK ways to attach and existing regional cluster to Global while creating the Global with "--source-db-cluster-identifier". It would be great if terraform can release this enhancement/bug to support attaching existing DB regional clusters to Global.

[bflad]

Enhancement submitted: https://github.com/terraform-providers/terraform-provider-aws/pull/14487

[bflad]

Hi folks 👋 Support for new force_destroy and source_db_cluster_identifier arguments in the aws_rds_global_cluster resource have been merged and will release with version 3.1.0 of the Terraform AWS Provider, likely later today. The 3.1.0 and later resource documentation will include a quick example of how to utilize these to create a Global Cluster from an existing DB Cluster. 👍

[hashibot[bot]]

This has been released in version 3.1.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!