Terraform-provider-aws: Cannot create global RDS Aurora cluster

[orlvn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.9
+ provider.aws v2.28.1

Affected Resource(s)

• aws_rds_cluster
• aws_rds_global_cluster
• aws_rds_cluster_instance

Terraform Configuration Files

provider "aws" {
  version = "~> 2.22"
  profile = var.profile
  region  = var.region
}

provider "aws" {
  profile = var.profile
  region  = var.primary_region
  alias   = "primary"
}

provider "aws" {
  profile = var.profile
  region  = var.secondary_region
  alias   = "secondary"
}

### Global

resource "aws_rds_global_cluster" "example" {
  global_cluster_identifier   = "global-test"
  deletion_protection         = false
  engine                      = "aurora"
  engine_version              = "5.6.10a"
  storage_encrypted           = "false"
}

### Primary

resource "aws_rds_cluster" "primary" {
  provider                  = aws.primary
  cluster_identifier        = "test-primary-cluster"
  global_cluster_identifier = aws_rds_global_cluster.example.id
  engine_mode               = "global"
  skip_final_snapshot       = true
  db_subnet_group_name      = "default"
}

resource "aws_rds_cluster_instance" "primary" {
  provider                  = aws.primary
  identifier                = "test-primary-cluster-instance"
  cluster_identifier        = aws_rds_cluster.primary.id
  instance_class            = "db.r4.large"
  db_subnet_group_name      = "default"
}

### Secondary

resource "aws_rds_cluster" "secondary" {
  provider                  = aws.secondary
  cluster_identifier        = "test-secondary-cluster"
  global_cluster_identifier = aws_rds_global_cluster.example.id
  engine_mode               = "global"
  skip_final_snapshot       = true
  db_subnet_group_name      = "default"
}

resource "aws_rds_cluster_instance" "secondary" {
  provider                  = aws.secondary
  identifier                = "test-secondary-cluster-instance"
  cluster_identifier        = aws_rds_cluster.secondary.id
  instance_class            = "db.r4.large"
  db_subnet_group_name      = "default"
}

Debug Output

n/a. (The error is a standard API error)

Panic Output

n/a

Expected Behavior

Provision a global Aurora RDS cluster

Actual Behavior

Error: error creating RDS cluster: InvalidParameterValue: The parameter MasterUsername must be provided and must not be blank.
    status code: 400, request id: b62c8f6b-fd62-4664-af8a-35ff30d25b98

  on aurora-global.tf line 27, in resource "aws_rds_cluster" "primary":
  27: resource "aws_rds_cluster" "primary" {



Error: error creating RDS cluster: InvalidParameterValue: The parameter MasterUsername must be provided and must not be blank.
    status code: 400, request id: e647a811-9bbf-4b01-b1fb-60d2f1eb7ec1

  on aurora-global.tf line 46, in resource "aws_rds_cluster" "secondary":
  46: resource "aws_rds_cluster" "secondary" {

Steps to Reproduce

1. terraform apply

Important Factoids

If the Username and Password is provided for the primary RDS cluster, the primary cluster can be created, but the secondary fails.

If both primary and secondary clusters are provided with a Username and Password, the error is:

Error: error creating RDS cluster: InvalidParameterCombination: Cannot specify user name for cross region replication cluster
    status code: 400, request id: 781b5f20-3f92-46c0-9e2d-3c3e27e97733

  on aurora-global.tf line 27, in resource "aws_rds_cluster" "primary":
  27: resource "aws_rds_cluster" "primary" {

References

This behavior is exactly the same as reported in

• https://github.com/terraform-providers/terraform-provider-aws/issues/7020

It was supposed to be fixed in AWS provider version 1.60, but even after installing that old version I got the same errors/behavior.

Note that Username and Password should be optional when global_cluster_identifier is provided in aws_rds_cluster as stated in https://www.terraform.io/docs/providers/aws/r/rds_cluster.html#master_password

[typez]

Hi there,

Could somebody from AWS provider contributors please take a look? Since it doesn't work via TF we had to implement the same functionality via AWS CLI, but hope to get it working via Terraform some day. Appreciate your help.

[diprochatterjee]

Hi there, this is still an issue - I have the same issue with creating a rds global cluster. The rds global cluster gets created but when I try to create an rds cluster as the primary cluster for the global cluster, it fails with error - * aws_rds_cluster.primary: error creating RDS cluster: InvalidParameterValue: The engine mode global you requested is currently unavailable. I am using Terraform version 0.11.11

[mrchief]

Using provider v2.57.0 and terraform 0.12.24, this issue is still happening.

If I provide a username for secondary cluster, terraform fails with

Error: error creating RDS cluster: InvalidParameterCombination: Cannot specify user name for cross region replication cluster
    status code: 400, request id: d0f8c5be-15a2-4414-b41c-b349027b88e1
  on rds.tf line 93, in resource "aws_rds_cluster" "secondary":
  93: resource "aws_rds_cluster" "secondary" {

If I don't provide one, it fails with

Error: error creating RDS cluster: InvalidParameterValue: The parameter MasterUsername must be provided and must not be blank.
    status code: 400, request id: 8a62a802-85b1-4463-97bb-33cb8ef6bcfc

  on rds.tf line 94, in resource "aws_rds_cluster" "secondary":
  94: resource "aws_rds_cluster" "secondary" {

[mrchief]

Just for the sake of experimentation, I switched to mysql engine from postgres and that went through. Another colleague of mine also was able to successfully stand up a mysql cluster. So it seems the problem is with postgres sql engine

This is what I had before

  engine              = "aurora-postgresql"
  engine_version      = "10.11"
  db_parameter_family = "aurora-postgresql10"

and now:

  engine              = "aurora-mysql"
  engine_version      = "5.7.mysql_aurora.2.07.1"
  db_parameter_family = "aurora-mysql5.7"

[mrchief]

I think I figured out the magic combo to make it work with Postgres:

resource "aws_rds_global_cluster" "primary" {
  provider                  = aws.primary
  engine                    = "aurora-postgresql"
  global_cluster_identifier = "my-global-cluster"
  engine_version            = "10.11"
  database_name             = "my-db"
}

Specifying database_name under aws_rds_global_cluster is what made it work.

And then you don't specify it for any of the cluster but make sure you provide username/password for primary (but not database_name)

resource "aws_rds_cluster" "primary" {
  provider                        = aws.primary
  global_cluster_identifier       = aws_rds_global_cluster.primary.id
  master_password                 = var.admin_password
  master_username                 = var.admin_username
  ...
}

resource "aws_rds_cluster" "secondary" {
  provider                        = aws.secondary
  ...
}

So in summary:

• specify database_name for aws_rds_global_cluster
• specify master_username, master_password for primary aws_rds_cluster
• omit database_name, master_username, master_password for secondary aws_rds_cluster

[mrchief]

Note that although it now succeeds, it doesn't create the initial database as specified by database_name. At this point, it seems there are several bugs:

• no initial DB created when database_name is specified on aws_rds_global_cluster
• buggy validation if database_name is specified on aws_rds_cluster
• InvalidParameterCombination issue
• InvalidParameterValue issue

[mrchief]

I managed to create the initial database, it required setting up the database name both at the global resources as well as the primary cluster resource level. So using the same setup as mentioned in my earlier comment, just add database_name to primary cluster.

resource "aws_rds_cluster" "primary" {
  ...
  database_name = "my-db"
}

It seems redundant to set it up for both global and primary but that's what did the trick.

[chartierf]

This thing drives me crazy... but after many tests i've finally something up and running

From my testings and values to be set :

• engine, engine_verion must be set on all resources to avoid Value for engine should match setting for global cluster (default is 'aurora') or Cannot find version 10.12 for aurora
  
  
  
  • aws_rds_global_cluster
  
  
  • aws_rds_cluster
  
  
  • aws_rds_cluster_instance
  
  
• storage_encrypted
  
  
  
  • aws_rds_global_cluster : MUST absolutly be set to false or we get source_version predefinied URL that cannot be set
  
  
• database_name
  
  
  
  • aws_rds_global_cluster
  
  
  • aws_rds_cluster
  
  

So to resume here is (for me) the terraform missing points

• engine, engine_version propagation from aws_rds_global_cluster to aws_rds_cluster and aws_rds_cluster to aws_rds_cluster_instance
• database_name propagation from aws_global_cluster to aws_rds_cluster

And to finalize be able to use storage_encrypted with true value.

I now to test if everythings works as intended

[suji22ss]

I tried creating Global Cluster for PostgreSQL and got the below error using terraform . Works absolutely fine for Aurora MySQL.

λ terraform apply --var-file=env/dev.tfvars

Error: expected engine to be one of [aurora aurora-mysql], got aurora-postgresql

on main.tf line 32, in resource "aws_rds_global_cluster" "rds_global_cluster":
32: resource "aws_rds_global_cluster" "rds_global_cluster" {

terraform {
required_version = "~> 0.12.13"

required_providers {
aws = "~> 2"
}
}