Terraform-provider-aws: Unable to ignore changes to RDS minor engine version

[joshuaspence]

We are using the aws_db_instance resource for managing RDS instances and I just came across an issue that arises from the use of auto_minor_version_upgrade. Basically, we have some Terraform configuration that looks like this:

resource "aws_db_instance" "master" {
  allocated_storage = 128
  engine            = "mysql"
  engine_version    = "5.7.10"
  instance_class    = "db.r3.large"
  storage_type      = "gp2"

  apply_immediately           = true
  auto_minor_version_upgrade  = true
  allow_major_version_upgrade = true

  # The other parameters are not relevant.
}

During one of our scheduled maintenance windows, Amazon have kindly upgrade this RDS instance from MySQL 5.7.10 to 5.7.17 (which is expected due to auto_minor_version_upgrade). As such, terraform plan now shows the following change to be applied:

~ aws_db_instance.master
    engine_version: "5.7.17" => "5.7.10"

I can workaround this by adding engine_version to ignore_changes, but doing so would prevent me from updating the major engine version. Ideally, I would be able to ignore changes only to the minor/patch version.

[Ninir]

Hey @joshuaspence,

There is nothing Terraform can help on this specific topic: either you want AWS to upgrade your minor version and then you just need to update the engine_version to the one set by AWS, or you control everything on your own.

This is based on your needs here :-/

Why not setting auto_minor_version_upgrade to false?

[joshuaspence]

I think my main problem here is that I wasn't expecting it to happen (although I'll admit that I wasn't sure what my expectation was when using auto_minor_version_upgrade). Generally I would expect that (assuming no changes to the resources had been applied out-of-band, such as manually through the AWS console) running terraform plan after terraform apply should never show changes to be applied (excluding potential issues with Terraform itself). It seems that this assumption doesn't hold valid when using auto_minor_version_upgrade.

One way that Terraform could better handle this is to allow functions to be used in ignore_changes, so that I could ignore changes to the patch version but not the major-minor vengine version. For example, ignore_changes = ["${join(".", slice(split(".", self.engine_version), 0, 2))}"]

[radeksimko]

I'm more inclined to @Ninir here in the sense that Terraform kind of works as expected here (without context of the resource/field). You have specified engine_version and because "something/someone else" has changed that (for better or worse) Terraform wants to keep it in line with what's in the config.

That said it's a common enough problem (AFAIK) that it's worth thinking about solution.
In theory the easiest solution would be to use DiffSuppressFunc which suppresses the diff of engine_version only if:

1. the difference only involves minor versions, and
2. the new version is higher (i.e. we should still display downgrades as anomaly in the diff), and
3. auto_minor_version_upgrade is set to true

Let me know what you think - and sorry for the re-labelling spam as I was thinking through the problem 🙈

[joshuaspence]

I think that the solution that you proposed seems reasonable to me.

[simonvanderveldt]

Would it be possible to support defining the versions as x.y and don't care about the patch (here called minor for some reason) version when defining it?
I think this should only be possible when auto_minor_version_upgrade = true.

Terraform would then pick the highest available version for the initial apply and after that it no longer matters as longs as the first or second digit don't change.

I.e. one would have

resource "aws_db_instance" "somedb" {
  ...
  engine            = "postgres"
  engine_version    = "9.6" (or "9.6.*)
  auto_minor_version_upgrade = true
  ...

[bflad]

Maybe this would be a good use case for a CustomizeDiff function? In psuedo-code:

if d.HasChange("engine_version") && d.Get("auto_minor_version_upgrade").(bool) {
  oldVersion, NewVersion := d.GetChange("engine_version")
  if /* logic to correctly determine non-major version changes */ {
    d.Clear("engine_version")
  }
}

[JohannesEbke]

This is now documented in the published docs, as per the pull request, but does not seem to work for me in aws-provider 1.46.0. I'm seeing the DEBUG statement from
2018-11-27T18:13:46.027+0100 [DEBUG] plugin.terraform-provider-aws_v1.46.0_x4: 2018/11/27 18:13:46 [DEBUG] Ignoring minor version diff
2018/11/27 18:13:46 [DEBUG] Removing 'id' diff and setting Destroy to false because after ignore_changes, this diff no longer requires replacement
...but the diff is still shown on apply.

[JohannesEbke]

It's still there in 1.51:

2018/12/18 10:16:37 [TRACE] root.x: eval: *terraform.EvalDiff
2018-12-18T10:16:37.217+0100 [DEBUG] plugin.terraform-provider-aws_v1.51.0_x4: 2018/12/18 10:16:37 [DEBUG] Ignoring minor version diff
2018/12/18 10:16:37 [DEBUG] Removing 'id' diff and setting Destroy to false because after ignore_changes, this diff no longer requires replacement
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: tags.%
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: snapshot_identifier
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: identifier_prefix
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: availability_zone
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: db_subnet_group_name
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: vpc_security_group_ids.2404141389
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: engine
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: storage_encrypted
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: skip_final_snapshot
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: backup_window
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: monitoring_interval
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: arn
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: final_snapshot_identifier
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: monitoring_role_arn
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: status
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: password
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: instance_class
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: storage_type
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: id
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: publicly_accessible
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: port
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: username
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: character_set_name
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: license_model
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: identifier
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: resource_id
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: auto_minor_version_upgrade
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: copy_tags_to_snapshot
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: ca_cert_identifier
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: allocated_storage
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: apply_immediately
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: endpoint
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: hosted_zone_id
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: backup_retention_period
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: multi_az
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: kms_key_id
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: replicas.#
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: name
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: option_group_name
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: parameter_group_name
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: maintenance_window
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: timezone
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: address
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: vpc_security_group_ids.#
2018/12/18 10:16:37 [DEBUG] [EvalIgnoreChanges] aws_db_instance.x - Ignoring diff attribute: tags.Name

[lowjoel]

@bflad was DiffSuppressFunc supposed to prevent this from occurring? I'm experiencing the same issue as @JohannesEbke with 1.59. If it was meant to be a CustomizeDiff change instead, I can try submitting a PR to fix it.

[kivagant-ba]

https://www.terraform.io/docs/providers/aws/r/db_instance.html#engine_version

engine_version - (Optional) The engine version to use. If auto_minor_version_upgrade is enabled, you can provide a prefix of the version such as 5.7 (for 5.7.10) and this attribute will ignore differences in the patch version automatically (e.g. 5.7.17). For supported values, see the EngineVersion parameter in API action CreateDBInstance. Note that for Amazon Aurora instances the engine version must match the DB cluster's engine version'.

Looks like the problem doesn't exist anymore and the solution is documented.

[vovandodev]

It still shows the difference.