Terraform-provider-aws: aws_db_instance Importing via ARN Sets Incorrect Identifier

Created on 28 Jun 2017  ·  10Comments  ·  Source: hashicorp/terraform-provider-aws

Hi,

I'm trying to change the instance type of an RDS instance managed by terraform, but am getting errors when applying the change. This is the only change I have ever made to the rds instance, and thus the only time it has show up in a plan/apply.

Terraform Version

Terraform v0.9.9

Affected Resource(s)

  • aws_db_instance

Terraform Configuration Files

resource "aws_db_instance" "rds" {
    identifier              = "my-rds-instance"
    allocated_storage       = 100
    engine                  = "postgres"
    engine_version          = "9.6.2"
    instance_class          = "db.m4.large"
    name                    = "${data.external.secure.result.db_name}"
    username                = "${data.external.secure.result.db_user}"
    password                = "${data.external.secure.result.db_password}"
    backup_retention_period = 1
    skip_final_snapshot     = true
    apply_immediately       = true

    vpc_security_group_ids  = ["${aws_security_group.rds.id}"]
    db_subnet_group_name    = "${aws_db_subnet_group.rds.name}"
}

Debug Output

Error applying plan:

1 error(s) occurred:

* aws_db_instance.rds: 1 error(s) occurred:

* aws_db_instance.rds: Error modifying DB Instance arn:aws:rds:us-west-2:<masked>:my-rds-instance: InvalidParameterValue: The parameter DBInstanceIdentifier is not a valid identifier. Identifiers must begin with a letter; must contain only ASCII letters, digits, and hyphens; and must not end with a hyphen or contain two consecutive hyphens.
    status code: 400, request id: <masked>

And with TF_LOG=DEBUG the relevant section shows:

2017/06/28 10:59:13 [DEBUG] root: eval: *terraform.EvalApply
2017/06/28 10:59:13 [DEBUG] apply: aws_db_instance.rds: executing Apply
aws_db_instance.rds: Modifying... (ID: arn:aws:rds:us-west-2:<masked>:db:my-rds-instance)
  instance_class: "db.t2.large" => "db.m4.large"
2017/06/28 10:59:13 [DEBUG] plugin: terraform: aws-provider (internal) 2017/06/28 10:59:13 [DEBUG] Send DB Instance Modification request: true
2017/06/28 10:59:13 [DEBUG] plugin: terraform: aws-provider (internal) 2017/06/28 10:59:13 [DEBUG] DB Instance Modification request: {
2017/06/28 10:59:13 [DEBUG] plugin: terraform:   ApplyImmediately: true,
2017/06/28 10:59:13 [DEBUG] plugin: terraform:   DBInstanceClass: "db.m4.large",
2017/06/28 10:59:13 [DEBUG] plugin: terraform:   DBInstanceIdentifier: "arn:aws:rds:us-west-2:<masked>:db:my-rds-instance"
2017/06/28 10:59:13 [DEBUG] plugin: terraform: }

There is a 400 return from aws shortly after this, but I think the above shows the bug pretty clear: terraform is using the aws_db_instance's arn instead of identifier in the DBInstanceIdentifier field when making a request to aws.

Expected Behavior

The instance type should have been changed.

Actual Behavior

The apply failed.

Steps to Reproduce

Happens every time I try to apply this change.

bug servicrds stale
Source
picture Lemmons
👍10

Most helpful comment

I've worked it around by removing the aws_db_instance resource from the state (terraform state rm ) and re-importing it by name, rather than by ARN.

picture vigasin on 19 Jun 2018
👍8

All 10 comments

For now I have worked around this issue by making the change manually in the aws console and then refreshing the terraform state, but it's by no means ideal.

Lemmons picture Lemmons on 28 Jun 2017

Hi @Lemmons

Apologies for this - by any chance are you using a data source in your terraform config as part of this? In the aws_db_instance resource, no where do we set the identifier to actually be an ARN - we are always using the value from the schema.

The reason I ask, I cannot reproduce this at all:

% terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_db_instance.bar: Refreshing state... (ID: terraform-0031a3ab586bb79eb31c7cf92f)
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

~ aws_db_instance.bar
    instance_class: "db.t1.micro" => "db.m4.large"


Plan: 0 to add, 1 to change, 0 to destroy.

Then the apply does the following:

% terraform apply                                                                                                                                                             1 ↵
aws_db_instance.bar: Refreshing state... (ID: terraform-0031a3ab586bb79eb31c7cf92f)
aws_db_instance.bar: Modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f)
  apply_immediately: "" => "true"
  instance_class:    "db.t1.micro" => "db.m4.large"
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 10s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 20s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 30s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 40s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 50s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m0s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m10s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m20s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m30s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m40s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 1m50s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m0s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m10s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m20s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m30s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m40s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 2m50s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m0s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m10s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m20s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m30s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m40s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 3m50s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m0s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m10s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m20s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m30s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m40s elapsed)
aws_db_instance.bar: Still modifying... (ID: terraform-0031a3ab586bb79eb31c7cf92f, 4m50s elapsed)
aws_db_instance.bar: Modifications complete (ID: terraform-0031a3ab586bb79eb31c7cf92f)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:

Thoughts?

Paul

stack72 picture stack72 on 6 Jul 2017

So thinking about it, I'm pretty sure I broke this state out from a larger state a while ago. I'm using shared state with s3 as the backend, so to "move" the state I actually had to delete it from one state, and the import it into another. Perhaps this actually indicates a breakage of the importer for aws_db_instances?

Looking at the terraform.state (I'm trimming out data which seems unnecessary):

                "aws_db_instance.rds": {
                    "primary": {
                        "id": "arn:aws:rds:us-west-2:<masked>:db:my-rds-instance",
                        "attributes": {
                            "address": "my-rds-instance.<masked>.us-west-2.rds.amazonaws.com",
                            "allocated_storage": "100",
                            "apply_immediately": "true",
                            "arn": "arn:aws:rds:us-west-2:<masked>:db:arn:aws:rds:us-west-2:<masked>:db:my-rds-instance",
                            "engine": "postgres",
                            "engine_version": "9.6.2",
                            "id": "arn:aws:rds:us-west-2:<masked>:db:my-rds-instance",
                            "identifier": "my-rds-instance",
                            "instance_class": "db.m4.large",
                        },
                    },
                },

Particularlly, look at the arn, it's definitely off, repeating it's self twice for most of it. I think one or both of the two id fields should be my-rds-instance instead of the arn, and the arn should not have the repeated arn:aws:rds:us-west-2:<masked>:db, but I'm not confident in any of that.

Lemmons picture Lemmons on 6 Jul 2017

I have just encountered this issue myself and I think I might be able to shed some light on it.

I have an RDS instance that was created manually in the AWS Console. At a later date, I used terraform import to bring this RDS instance into a statefile with corresponding Terraform.

However, upon examination of the resource within the statefile I see the following (for the sake of brevity I've removed all but the interesting fields):

"aws_db_instance.rds_instance": {
  "type": "aws_db_instance",
  ...
  "primary": {
    "id": "arn:aws:rds:eu-west-2:000000000000:db:my-rds-instance",
    "attributes": {
      ...
      "arn": "arn:aws:rds:eu-west-2:000000000000:db:arn:aws:rds:eu-west-2:000000000000:db:my-rds-instance",
      "id": "arn:aws:rds:eu-west-2:000000000000:db:my-rds-instance",
      "identifier": "my-rds-instance",
    },
    ...
  }
}

When I compare this with an RDS instance in a statefile where the Terraform created the RDS instance from scratch, I see the following:

"aws_db_instance.rds_instance": {
  "type": "aws_db_instance",
  ...
  "primary": {
    "id": "my-rds-instance",
    "attributes": {
      ...
      "arn": "arn:aws:rds:eu-west-2:000000000000:db:my-rds-instance",
      "id": "my-rds-instance",
      "identifier": "my-rds-instance",
    },
    ...
  }
}

It appears that what's happening here is that terraform import is erroneously treating the instance ARN as the instance ID during the import process, and therefore:

  • The primary/id field is set to the ARN, when it should be the instance ID.
  • The primary/attributes/arn field has the entire ARN appended to arn:aws:rds:eu-west-2:000000000000:db: string, giving what looks like an ARN that repeats itself Presumably there is some kind of string concatenation going on behind the scenes here.
  • The primary/attributes/id field has the entire ARN when it should be the instance ID only.

I'm currently using version 0.9.11 so this might have been fixed in 0.10.x but I haven't had the opportunity to test it yet. A scan of the most recent changelog however does not suggest that it has been fixed.

danielgrant picture danielgrant on 31 Oct 2017
👍8

@danielgrant definitely looks to be the same issue and the same cause. I think we can pretty confidently say that terraform import has an issue for rds instances.

Lemmons picture Lemmons on 31 Oct 2017

My setup is the same as @danielgrant and I have consistently reproduced this issue. Here is the debug trace (sensitive information redacted):

State:

id                                  = arn:aws:rds:us-east-1:000000000000:db:application-db
address                             = application-db.abcdabcd1234.us-east-1.rds.amazonaws.com
allocated_storage                   = 100
apply_immediately                   = true
arn                                 = arn:aws:rds:us-east-1:000000000000:db:arn:aws:rds:us-east-1:000000000000:db:application-db
auto_minor_version_upgrade          = true
availability_zone                   = us-east-1a
backup_retention_period             = 7
backup_window                       = 09:36-10:06
ca_cert_identifier                  = rds-ca-2015
copy_tags_to_snapshot               = false
db_subnet_group_name                = database-subnet-group
endpoint                            = application-db.abcdabcdabcd.us-east-1.rds.amazonaws.com:3306
engine                              = mysql
engine_version                      = 5.7.16
hosted_zone_id                      = ZZZZZZZZZZZZZZZ
iam_database_authentication_enabled = false
identifier                          = application-db
instance_class                      = db.t2.medium
iops                                = 0
kms_key_id                          =
license_model                       = general-public-license
maintenance_window                  = mon:08:10-mon:08:40
monitoring_interval                 = 0
multi_az                            = false
name                                = application_database
option_group_name                   = default:mysql-5-7
parameter_group_name                = default.mysql5.7
port                                = 3306
publicly_accessible                 = false
replicas.#                          = 0
replicate_source_db                 =
resource_id                         = db-EZEZEEZEZEZEZEZEZEZEZEZ
security_group_names.#              = 0
skip_final_snapshot                 = true
status                              = available
storage_encrypted                   = false
storage_type                        = gp2
tags.%                              = 0
timezone                            =
username                            = root
vpc_security_group_ids.#            = 2
vpc_security_group_ids.1856740275   = sg-ffffffff
vpc_security_group_ids.2545511893   = sg-bbbbbbbb

Trace log:

[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: 2017/11/24 08:35:06 [DEBUG] Send DB Instance Modification request: true
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: 2017/11/24 08:35:06 [DEBUG] DB Instance Modification request: {
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   AllocatedStorage: 100,
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   ApplyImmediately: true,
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   DBInstanceIdentifier: "arn:aws:rds:us-east-1:0000000000000:db:application-database",
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   Iops: 0
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: }
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: 2017/11/24 08:35:06 [DEBUG] [aws-sdk-go] DEBUG: Request rds/ModifyDBInstance Details:
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: POST / HTTP/1.1
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Host: rds.us-east-1.amazonaws.com
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: User-Agent: aws-sdk-go/1.12.19 (go1.9; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.10.9-dev
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Content-Length: 189
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXX/20171124/us-east-1/rds/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=2087bdc5d06f660daa25xxxxxxxxxxxxxxx9999999999999999999999
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: X-Amz-Date: 20171124T133506Z
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Accept-Encoding: gzip
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Action=ModifyDBInstance&AllocatedStorage=100&ApplyImmediately=true&DBInstanceIdentifier=arn%3Aaws%3Ards%3Aus-east-1%3A438340096590%3Adb%3Aapplication-database&Iops=0&Version=2014-10-31
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: -----------------------------------------------------
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: 2017/11/24 08:35:07 [DEBUG] [aws-sdk-go] DEBUG: Response rds/ModifyDBInstance Details:
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: ---[ RESPONSE ]--------------------------------------
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: HTTP/1.1 400 Bad Request
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Connection: close
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Content-Length: 472
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Content-Type: text/xml
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: Date: Fri, 24 Nov 2017 13:35:07 GMT
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: X-Amzn-Requestid: 9b6444d7-aa1c-447a-a28f-93a64330a8f1
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: -----------------------------------------------------
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4: 2017/11/24 08:35:07 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   <Error>
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:     <Type>Sender</Type>
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:     <Code>InvalidParameterValue</Code>
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:     <Message>The parameter DBInstanceIdentifier is not a valid identifier. Identifiers must begin with a letter; must contain only ASCII letters, digits, and hyphens; and must not end with a hyphen or contain two consecutive hyphens.</Message>
[DEBUG] plugin.terraform-provider-aws_v1.2.0_x4:   </Error>

I was able to work around this issue by modifying the state by hand and then running plan / apply.

tquach picture tquach on 24 Nov 2017

I just hit very similar thing:
updating RDS instance I got arn in the DBInstanceIdentifier field:
2018-06-07T20:49:26.611+0200 [DEBUG] plugin.terraform-provider-aws_v1.21.0_x4: DBInstanceIdentifier: "arn:aws:rds:eu-central-1:123456789123:db:tf-production-rds",

kwach picture kwach on 7 Jun 2018

I've worked it around by removing the aws_db_instance resource from the state (terraform state rm ) and re-importing it by name, rather than by ARN.

vigasin picture vigasin on 19 Jun 2018
👍8

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

github-actions[bot] picture github-actions[bot] on 9 Jun 2020

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] picture hashibot[bot] on 9 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hashibot picture hashibot  ·  3Comments
joelittlejohn picture joelittlejohn  ·  3Comments
reedloden picture reedloden  ·  3Comments
hashibot picture hashibot  ·  3Comments
carmas picture carmas  ·  3Comments