Terraform-provider-google: missing way to create gcp internal HTTPS load balancer.

[adimania]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

As per the GCP documentation, in order to create internal HTTPS load balancer, following items are needed, but I could not find them:

• regional ssl certificates
• proxy-only subnet

I have tried the following but in absence of the above resources (or due to my lack of understanding) I get a 503 from Google backend:

resource "google_compute_subnetwork" "XYZ-internal-lb-subnet" {
  name          = "XYZ-internal-lb-subnet"
  ip_cidr_range = "10.100.10.0/24"
  region        = "us-east1"
  network       = "XYZ-staging"
}

resource "google_compute_health_check" "vault-health-check" {
  name               = "vault-health-check"
  timeout_sec        = 5
  check_interval_sec = 10
  https_health_check {
    port = 8200
    request_path = "/v1/sys/health"
  }
}

resource "google_compute_region_backend_service" "vault-backend-service" {
  name          = "vault-backend-service"
  region        = "us-east1"
  health_checks = [google_compute_health_check.vault-health-check.self_link]
  protocol      = "HTTPS"
  backend {
    group          = google_compute_instance_group.vault-servers.self_link
    balancing_mode = "RATE"
    max_rate_per_instance = 1000
  }
}

resource "google_compute_forwarding_rule" "vault-forwarding-rule" {
  name   = "vault-forwarding-rule2"
  region = "us-east1"
  ports = ["443"]
  load_balancing_scheme = "INTERNAL_MANAGED"
  backend_service       = google_compute_region_backend_service.vault-backend-service.self_link
  network               = "projects/XYZ-165816/global/networks/XYZ-staging"
  subnetwork            = "staging-app"
}

resource "google_compute_instance_group" "vault-servers" {
  name = "vault-servers"
  named_port {
    name = "https"
    port = 443
  }

  instances = [
    google_compute_instance.vault-1.self_link,
    google_compute_instance.vault-2.self_link,
  ]

  zone = "us-east1-b"
}

resource "google_compute_url_map" "vault-urlmap" {
  name        = "vault-urlmap"
  default_service = google_compute_region_backend_service.vault-backend-service.self_link
}

resource "google_compute_target_https_proxy" "vault-https-proxy" {
  name             = "vault-https-proxy"
  url_map          = google_compute_url_map.vault-urlmap.self_link
  ssl_certificates = ["wc-internal-cm-o"]
}

[c2h5oh]

I've had the same problem until I've set

purpose       = "INTERNAL_HTTPS_LOAD_BALANCER"
role          = "ACTIVE"

for the 2nd subnet (the one used by LB)

[jharshman]

Running into this as well.

Error: Error creating ForwardingRule: googleapi: Error 503: Internal error. Please try again or contact Google Support. (Code: '598BE0C67E164.A300A86.3D083C09'), backendError

  on ilb-sandbox.tf line 84, in resource "google_compute_forwarding_rule" "backend-forward-rule":
  84: resource "google_compute_forwarding_rule" "backend-forward-rule" {

tf:

# instance group
resource "google_compute_instance_group" "ig01" {
  name = "ig01"
  instances = [
    "${module.b01.self_link}",
    "${module.b02.self_link}",
  ]

  zone = "${local.uscentral_zones[0]}"

  named_port {
    name = "backend-http"
    port = "80"
  }
}

# health check
resource "google_compute_region_health_check" "backend-hc" {
  provider           = "google-beta"
  region             = "us-central1"
  name               = "check-backend-eighty"
  check_interval_sec = 5
  timeout_sec        = 3

  tcp_health_check {
    port = "80"
  }
}

# reserved ilb subnet
resource "google_compute_subnetwork" "ilb-subnet" {
  provider      = "google-beta"
  name          = "ilb-subnet"
  ip_cidr_range = "10.0.3.0/24"
  region        = "us-central1"
  purpose       = "INTERNAL_HTTPS_LOAD_BALANCER"
  role          = "ACTIVE"
  network       = "${google_compute_network.sandbox.self_link}"
}

# internal forwarding rule
resource "google_compute_forwarding_rule" "backend-forward-rule" {
  name   = "forwardingrule01"
  region = "us-central1"

  load_balancing_scheme = "INTERNAL_MANAGED"
  backend_service       = "${google_compute_region_backend_service.backend.self_link}"
  ports                 = ["80"]
  network               = "${google_compute_network.sandbox.self_link}"
  subnetwork            = "${google_compute_subnetwork.ilb-subnet.self_link}"
  service_label         = "b0"
}

# target proxy
resource "google_compute_region_target_http_proxy" "default" {
  provider = "google-beta"
  name     = "targetproxy01"
  region   = "us-central1"
  url_map  = "${google_compute_region_url_map.default.self_link}"
}

# url map
resource "google_compute_region_url_map" "default" {
  provider        = "google-beta"
  name            = "urlmap01"
  region          = "us-central1"
  default_service = "${google_compute_region_backend_service.backend.self_link}"

  host_rule {
    hosts        = ["b0.internal"]
    path_matcher = "allpaths"
  }

  path_matcher {
    name            = "allpaths"
    default_service = "${google_compute_region_backend_service.backend.self_link}"
  }
}

# backend service
resource "google_compute_region_backend_service" "backend" {
  name                  = "backend01"
  region                = "us-central1"
  health_checks         = ["${google_compute_region_health_check.backend-hc.self_link}"]
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  backend {
    balancing_mode = "UTILIZATION"
    group          = "${google_compute_instance_group.ig01.self_link}"
  }
}


# ingress allow GCP probe
resource "google_compute_firewall" "allow-ilb" {
  name    = "allow-ilb"
  network = "${google_compute_network.sandbox.self_link}"

  allow {
    protocol = "tcp"
    ports = [
      "80",
    ]
  }

  source_ranges = ["172.16.31.0/24"]
  target_tags   = ["load-balanced"]
}

@c2h5oh can you elaborate on your work-around? The compute_forwarding_rule resource only allows you to set one subnet.

[danawillow]

@tysen, can you check on the configs in here and see whether they seem all right or need to be adjusted? it would also probably be worth having an end-to-end example on the website if there isn't one already.

[tysen]

google_compute_region_ssl_certificate is only available in the beta provider for now because HTTP(S) ILB has not yet reached GA. Same for the purpose and role fields for google_compute_subnetwork.

Here's the doc page for google_compute_region_ssl_certificate. Looks like the sidebar hasn't been updated to include it yet.

[jharshman]

@tysen my use case is HTTP.

[tysen]

I was able to repro your 503. Forwarding rules set to INTERNAL_MANAGED can't point (directly) to a backend service, but must go through a proxy (as shown here). However, the target field in google_compute_forwarding_rule only accepts google_compute_target_pool, but it needs to accept other types. I'll work on resolving this.

[justinholmes]

When's the fix eta? I am also affected.

[tysen]

It's blocked by some changes to RegionUrlMap that are waiting for code review. I'll mention this issue in that PR.

[jharshman]

@tysen
can you give us that PR number so we can track that as well?
It hasn't been cross linked here.

I should also mention that the documentation on the google_compute_forwarding_rule resource probably needs to be updated. The target field currently states it is for external load balancers.

target - (Optional) This field is only used for EXTERNAL load balancing. A reference to a TargetPool resource to receive the matched traffic. This target must live in the same region as the forwarding rule. The forwarded traffic must be of a type appropriate to the target object.

[jharshman]

@tysen any updates or ETA for the fix?

[tysen]

@jharshman I have been working on this but encountered an issue with creating a forwarding rule uses a subnetwork with purpose = INTERNAL_HTTPS_LOAD_BALANCER. I'm talking with the relevant team but haven't found a resolution yet.

Specifically what I mean is that if I attempt to have Terraform provision the resources listed for the example (part 1, part 2), the creation of the forwarding rule fails with a 400. Have you been successful setting up a forwarding rule like this outside of Terraform?

[BenCoughlan15]

I'm getting this too.

When I try to run the following :

module "internal-lb" {                                   
  source = "../../modules/internal-lb"                

  ip_cidr              = "0.0.0.00/24"              
  network_name         = "${var.project}-vpc-network"    
  region               = var.region                      
  project              = var.project                     
  backend_service_name = "test"                       
  lb_name              = "test"                   
}                                                       

data "google_compute_backend_service" google_backend_service {
  name    = var.backend_service_name
  project = var.project
}

resource "google_compute_forwarding_rule" "internal-forwarding-rule" {
  provider              = google-beta
  name                  = "${var.lb_name}-forwarding-rule"
  load_balancing_scheme = "INTERNAL_SELF_MANAGED"
  backend_service       = data.google_compute_backend_service.google_backend_service.self_link
  all_ports             = true
  network               = data.google_compute_network.vpc_network.self_link
  subnetwork            = google_compute_subnetwork.l7lb_subnet.self_link
  project               = var.project
}

data "google_compute_network" "vpc_network" {
  name    = var.network_name
  project = var.project
}
resource "google_compute_subnetwork" "l7lb_subnet" {
  provider      = google-beta
  name          = "${var.network_name}-l7lb-subnet"
  ip_cidr_range = var.ip_cidr
  region        = var.region
  purpose       = "INTERNAL_HTTPS_LOAD_BALANCER"
  role          = "ACTIVE"
  network       = data.google_compute_network.vpc_network.self_link
  project       = var.project
}

provider google-beta {
  project = var.project
  version = ">= 2.5.1"
}

I get :

---

2019/12/17 15:50:37 [WARN] module.internal-lb: eval: *terraform.EvalValidateResource, non-fatal err: expected load_balancing_scheme to be one of [EXTERNAL INTERNAL INTERNAL_MANAGED ], got INTERNAL_SELF_MANAGED
2019/12/17 15:50:37 [ERROR] module.internal-lb: eval: *terraform.EvalSequence, err: expected load_balancing_scheme to be one of [EXTERNAL INTERNAL INTERNAL_MANAGED ], got INTERNAL_SELF_MANAGED

Error: expected load_balancing_scheme to be one of [EXTERNAL INTERNAL INTERNAL_MANAGED ], got INTERNAL_SELF_MANAGED

on ../../modules/internal-l7-lb/lb.tf line 6, in resource "google_compute_forwarding_rule" "internal-forwarding-rule":
6: resource "google_compute_forwarding_rule" "internal-forwarding-rule" {

[jharshman]

@tysen I am able to create a L7 ILB from the UI when starting from scratch. However, having TF provision only part of the resources and then going into the UI to finish the forwarding rule does result in an error.

[sorenhansendk]

@tysen Is there any ETA on this feature - or is there any working workaround with URL maps and all the stuff there is possible to configure via the gcloud CLI on the internal HTTP(S) LB? https://cloud.google.com/load-balancing/docs/l7-internal/set-up-gce-vms

[tysen]

The fix is merged so it should be in the next release.

[BenCoughlan15]

Thanks @tysen

[rayuduc]

Hi @tysen,
Is this fix available in google-provider Beta ?
Am getting same issue when using even though we have dedicated subnetwork with
purpose = "INTERNAL_HTTPS_LOAD_BALANCER" and role = "ACTIVE"

resource "google_compute_forwarding_rule" "nginx-europe-lb" {
provider = "google-beta"
name = "nginx-int-lb-test"
load_balancing_scheme = "INTERNAL_MANAGED"
project = var.tes-project
backend_service = google_compute_region_backend_service.nginx-europe-west1.self_link
ports = ["80"]
region = "europe-west1"
#allow_global_access = true
network = google_compute_network.nginx-network.self_link
subnetwork = google_compute_subnetwork.ilb-nginx-subnetwork.self_link

}
Error: Error creating ForwardingRule: googleapi: Error 503: Internal error. Please try again or contact Google Support. (Code: 'XXXXX.XXXXX.XXXXX'), backendError

[BenCoughlan15]

Hey, looks like it's still not released.

B

[BenCoughlan15]

Hey all, any sign of this being released?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!