Terraform-provider-google: Error updating secondary IP ranges in Google_compute_subnetwork

[mrjfox]

Terraform plugin version: terraform-provider-google_v1.19.1_x4

So i would like to replace an existing secondary ip range but this fails to work.

Terraform change

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ google_compute_subnetwork.network-with-private-secondary-ip-ranges
      secondary_ip_range.4.ip_cidr_range: "192.168.50.0/24" => "192.168.60.0/24"
      secondary_ip_range.4.range_name:    "secondary-range-50" => "secondary-range-60"


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Error produced

googleapi: Error 400: Invalid value for field 'resource.secondaryIpRanges': ''. Cannot add and remove secondary IP ranges in the same request. Added ranges: [192.168.60.0/24]. 
Removed ranges: [192.168.50.0/24]., invalid

As this error is from the Google Api, can you please change the way terraform applies this change by deleting first and then adding the change.

[dijitali]

I'm seeing similar symptoms but with a different error: Existing secondary range cannot be modified.

Terraform Version

Terraform v0.11.13
+ provider.google v2.3.0
+ provider.google-beta v2.3.0

Affected Resource(s)

• google_compute_subnetwork

Terraform Configuration Files

provider "google" {
  version     = "~>2.2"
  credentials = "${file("~/key.json")}"
  project     = "my-project-123"
  region      = "europe-west1"
  zone        = "europe-west1-a"
}

resource "google_compute_network" "vpc-network" {
  name = "custom-network1"
  auto_create_subnetworks = false
}


resource "google_compute_subnetwork" "vpc-subnet" {
  name          = "subnet-eu-west-192"
  network       = "${google_compute_network.vpc-network.self_link}"
  ip_cidr_range = "192.168.0.0/16"
  region        = "europe-west1"

  secondary_ip_range {
    range_name    = "europe-west1-01"
    ip_cidr_range = "10.0.1.0/24"
  }
  secondary_ip_range {
    range_name    = "europe-west1-02"
    ip_cidr_range = "10.0.2.0/24"
  }
}

Expected Behavior

Subnet ranges are either updated in-place or recreated.

Actual Behavior

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.network.google_compute_subnetwork.vpc-subnet
      secondary_ip_range.0.ip_cidr_range: "10.0.0.0/24" => "10.0.1.0/24"
      secondary_ip_range.1.ip_cidr_range: "10.1.0.0/24" => "10.0.2.0/24"


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Error: Error applying plan:

1 error(s) occurred:

* module.network.google_compute_subnetwork.vpc-subnet: 1 error(s) occurred:

* google_compute_subnetwork-vpc-subnet: Error updating Subnetwork "europe-west1/subnet-eu-west-192": googleapi: Error 400: Invalid value for field 'resource.secondaryIpRanges[0].ipCidrRange': '10.0.1.0/24'. Existing secondary range cannot be modified: 10.0.1.0/24., invalid

[andor44]

To my understanding (which the web console seems to confirm) secondary ranges cannot be changed after creation, so shouldn't changes to them force new? When I try to edit a subnetwork in the console the secondary subnets are grayed out and they cannot be changed.

[benv666]

Just got hit by this bug, as andor44 says it would make sense to just recreate them.
Additionally when we tried to have TF delete the secondary ranges by commenting them out it simply ignored them! (no changes to apply).

[edwardmedia]

I received the error too when I tried to update the secondary_ip_range. My provider v3.16.0

 ---[ REQUEST ]---------------------------------------
 PATCH /compute/v1/projects/myproject/regions/us-central1/subnetworks/issue-2570-subnetwork?a
lt=json HTTP/1.1
 {
  "fingerprint": "kwjy9zCcwp4=",
  "secondaryIpRanges": [
   {
    "ipCidrRange": "192.168.20.0/24",
    "rangeName": "tf-test-secondary-range-update1-20"
   }
.16.0_x5:  ]
 } 

 {
  "error": {
   "errors": [
    {
     "domain": "global",
     "reason": "invalid",
     "message": "Invalid value for field 'resource.secondaryIpRanges': ''. Cannot add and remove secondary IP ranges in the same request. Added ranges: [192.168.20.0/24]. Removed ranges: [192.168.10.0/24]."
    }
],
   "code": 400,
   "message": "Invalid value for field 'resource.secondaryIpRanges': ''. Cannot add and remove secondary IP ranges in the same request. Added ranges: [192.168.20.0/24]. Removed rang
es: [192.168.10.0/24]."
  }
 }

[rileykarson]

This resource is generated by Magic Modules, and it has a limitation that it expects that each field can be updated with a single update method call. However, the rules imposed by the API mean that it would take 2 or more (addition & removal) to do so.

There isn't anywhere in the MM code generator to add this logic, and the addition would be a substantial change to the templates that I don't believe we want to undertake at this time (there are other fairly substantial changes coming in the medium term this would conflict with). I'm labelling this as an enhancement in addition to bug, since it would be a substantial generator change, and leaving it to get triaged by the team during sprint planning.

[danawillow]

(bugs don't get picked up by the triage tool, so changing it from bug+enhancement into persistent-bug)

[billyfoss]

Are there any recommended workarounds?
I'm seeing this when trying to rename secondary cidr ranges for GKE clusters.

I'm using the following versions

Terraform v0.12.26
+ provider.google v3.27.0
+ provider.google-beta v3.27.0
+ provider.kubernetes v1.11.3
+ provider.null v2.1.2
+ provider.random v2.2.1
+ provider.template v2.1.2

I even tried to manually delete the secondary ranges, but Terraform was seeing the state and still tried to remove the secondary ranges in the same request.

Error: Error updating Subnetwork "projects/my-project/regions/us-east1/subnetworks/my-subnetwork": googleapi: Error 400: Invalid value for field 'resource.secondaryIpRanges': ''. Cannot add and remove secondary IP ranges in the same request. Added ranges: [172.24.0.0/16, 240.0.0.0/13]. Removed ranges: [10.124.0.0/14, 172.30.0.0/16]., invalid

[danawillow]

I think the best workaround at the moment is just to terraform apply multiple times (once for deletions, once for additions, with the assumption that a modification is a deletion followed by an addition).

I filed an upstream issue (b/159828937 for googlers who wish to follow along) to see if that restriction could be removed on the API side.