Terraform: feature request: parallelism parameter for resources with count

[dkiser]

As a Terraform User, I should be able to specify a parallelism count on resource instances containing a count parameter So That I Can handle creating API resources via providers at a more sane rate and/or deal with mediocre api backends.

Example Terraform Configuration File

resource "providera_resourcea" "many" {
   count = 1000
   parallelism = 10

   attributeA = "a"
   attributeB = "b"
   attributeC = "c"
}

Expected Behavior

GIVEN terraform apply -parallelism = X where X < ${providera_resourcea.many.*.parallelism)
WHEN terraform creates/deletes/refreshes resources
THEN I expect only X concurrent goroutines creating this resource type.

GIVEN terraform apply -parallelism = X where X >= ${providera_resourcea.many.*.parallelism)
WHEN terraform creates/deletes/refreshes resources
THEN I expect only ${providera_resourcea.many.*.parallelism) concurrent goroutines creating this resource type.

[dkiser]

Possibly related to #7388

[kwach]

+1

[Stono]

+1

[pbusquemdf]

I just try to create 3 vsphere_virtual_machine resource.
Because all 3 virtual machines are created at the same time, they take exponentially more time to create, causing the apply operation to time out.

Creating 1 machine take 5 minutes. All 3 machines therefor take 15 minutes with a single thread.
Creating 3 machines timeout after 10 minutes because each machine are now taking longer than 5 minutes each and are creating disk, balancing disk and reconfiguring bottleneck on the vsphere server.

But other resources are working fine. So, I should be able to limit the number of simultaneously job running into either the resource, or the provider (or both)

[invidian]

Another use case I would see for that is rolling update of immutable infrastructure, where you just roll the update one server/resource at a time.

[mkjmdski]

I found this issue when trying to run apply against "pass" provider which needs to communicate with git repository and this should be done one by one, but my infrastructure covers many other resources types (from different providers) so I'd like to run it with high parallelism but limited to 1 only for resources of certain type (or provider as mnetioned @invidian )

[davidquarles]

I'm hitting this today. There is still no known workaround, I take it?

Anecdotally, what I'm trying to do is create many managed instance groups in GCP that are all backends for the same load-balancer (using count) but can't be collapsed into one because of upstream constraints and how we're partitioning outbound traffic. Doing so forfeits rolling update semantics, of course.

I started to hack at having each instance group depend on the prior one after the head of the list until I realized that depends_on is static and the entire resource group is actually a single node in the DAG. Any ideas? As it stands, my only real strategy is to move this stuff out into a dedicated repo run with -parallelism=1 and use the remote state provider to loosely couple back to our primary repository :(

[brendan-sherrin]

I'm getting an error: Deleting CloudWatch Events permission failed: ConcurrentModificationException: EventBus default was modified concurrently

I believe this suggestion would let me work around this issue by applying a parellism limit on permissions affecting the default event bus on the account.

ie. adding a parellism attribute to this resource:
resource "aws_cloudwatch_event_permission" "PerAccountAccess" {
for_each = local.accountslist

[MrSimonEmms]

I've found a similar issue with multiple Google SQL databases on a private IP where this would be incredibly useful (detailed on SO.

[delwaterman]

👍 Have this issue with a custom redshift provider. Need to limit the number of concurrent requests being made.

[hege-aliz]

Same here with AWS task definitions within the same family.

[antanof]

Similar issue with Azure dns and Public IP :
I want to create severals A record for the same public IP

resource "azurerm_dns_a_record" "new" {
  count               = length(var.subdomains)
  name                = coalesce(var.subdomains[count.index])
  zone_name           = "var.zone"
  resource_group_name = "var.dns_rgname"
  ttl                 = 60
  target_resource_id  = azurerm_public_ip.public_ip.id

  depends_on          = [azurerm_public_ip.public_ip]
}

have an issue with terraform apply :

dns.RecordSetsClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" 

No issue with terraform apply --parallelism=1

[clarsonneur]

Like many others, I'm face the same kind of issue for some resources with google cloud. (network peering, firestore indexes, ...)