Terraform-provider-azurerm: azurerm_policy_set_definition update behaviour broken in 2.19 using policy_definition_reference

[bigglesuk69]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.28

• provider.azurerm v2.19.0

Affected Resource(s)

• azure_policy_set_definition

Terraform Configuration Files

I'm unable to provide all the relevant code due to restrictions on releasing code from the end client.

resource "azurerm_policy_set_definition" "diags" {
  name         = "diagnostics-initiative"
  policy_type  = "Custom"
  display_name = "Apply Default Diagnostic Settings"
  management_group_name           = var.definition_management_group_name

  parameters                      = file("${path.module}/json/diagnostics/_globals/initiative_parameters.json")

  dynamic "policy_definition_reference" {
      for_each = [for item in var.diagnostic_settings: {
              name = item.resource_name
          }
          if item.include_in_initiative==true && item.policy_type =="logs"
      ]

      content {
          policy_definition_id = azurerm_policy_definition.definition["${policy_definition_reference.value.name}_logs"].id
          parameters = {
            eventHubAuthorizationRuleId = "[parameters('eventHubAuthorizationRuleId')]"
            eventHubName = "[parameters('eventHubName')]"
            logsEnabled = "[parameters('logsEnabled')]"
            profileName = "[parameters('profileName')]"
            storageAccountId = "[parameters('storageAccountId')]"
            workspace = "[parameters('workspace')]"
            retentionInDays = "[parameters('retentionInDays')]"

          }
       }

    }

  dynamic "policy_definition_reference" {
      for_each = [for item in var.diagnostic_settings: {
              name = item.resource_name
          }
          if item.include_in_initiative==true && item.policy_type =="metrics"
      ]

      content {
          policy_definition_id = azurerm_policy_definition.definition["${policy_definition_reference.value.name}_metrics"].id
          parameters = {
              eventHubAuthorizationRuleId = "[parameters('eventHubAuthorizationRuleId')]"
                eventHubName = "[parameters('eventHubName')]"
                metricsEnabled = "[parameters('metricsEnabled')]"
                profileName = "[parameters('profileName')]"
                workspace = "[parameters('workspace')]"

          }
       }
    }
}

The input consists of input files containing the following which are created using a for_each against the azurerm_policy_definition resource.

variable "diagnostic_settings" {
    default = {
        # Azure Firewall
        firewall_logs = {
            resource_name = "firewall"
            resource_type = "Microsoft.Network/azureFirewalls"
            display_name = "Azure Firewall Logs"
            policy_type = "logs"
            include_in_initiative = true
        }
        firewall_metrics = {
            resource_name = "firewall"
            resource_type = "Microsoft.Network/azureFirewalls"
            display_name = "Azure Firewall Metrics"
            policy_type = "metrics"
            include_in_initiative = true
        }
    }
}

Expected Behavior

Policy Initiative should have been amended with additional or removal of a policy.

Actual Behavior

When adding/removing items it throws errors on the resource_id as it tries to re-map the resource_ids between different policies.

azurerm_policy_set_definition.diags: Modifying... [id=/providers/Microsoft.Management/managementgroups/sandbox/providers/Microsoft.Authorization/policySetDefinitions/diagnostics-initiative]

Error: creating/updating Policy Set Definition "diagnostics-initiative": policy.SetDefinitionsClient#CreateOrUpdateAtManagementGroup: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="DuplicateReferencesInPolicySet" Message="The policy set definition 'diagnostics-initiative' request is invalid. Policy set definition contains duplicate policy references. Duplicate reference Ids are '14127430907021476224'. Please remove any exact duplicates from the policy set definition. Violating policy definition Ids are '/providers/Microsoft.Management/managementgroups/sandbox/providers/Microsoft.Authorization/policyDefinitions/diag-logic_apps-metrics, /providers/Microsoft.Management/managementgroups/sandbox/providers/Microsoft.Authorization/policyDefinitions/diag-virtual_network-metrics'."

on diagnostics_policies_initiative.tf line 34, in resource "azurerm_policy_set_definition" "diags":
34: resource "azurerm_policy_set_definition" "diags" {

Steps to Reproduce

1. Apply an initiative with multiple policies
2. Inject an additional policy into the middle of the list OR remove an item from the list
3. Plan and apply the update

[ArcturusZhang]

Hi @bigglesuk69 thanks for opening this issue!

This should be caused during the expanding of the array of policy_definition_reference, some elements are added multiple times somehow.

Similar with issue #7802

I am working on this to find the root cause.

[ArcturusZhang]

Similar symptom in issue #7751

[caldwell2000]

Having the same issue. Is there a workaround or a fix coming soon?
Using Versions:
Terraform v0.12.21
provider.azurerm v2.25.0

[ezubriski]

@caldwell2000 I have been working around this by recreating the policy initiative. I have been working around this by reordering my policy reference blocks. When that doesn't work, I have recreated the resource so that the update can happen.

This works for dev but for production scenarios with policy attachments its not too useful.

[bigglesuk69]

I'm wondering whether setting a unique value in the optional 'reference_id' will resolve the issue. I believe having that passed in alongside the definition ID may get this to work. Looking at the Security Center initiative, they have the ref ID set to a text string not numeric ID.

I haven't had time to test that yet, but will let you know once I do.

[caldwell2000]

@bigglesuk69 - Looks like your solution works. I set the reference_id to the same value as the policy_definition_id and the "apply" works. Thanks for the idea!

[ArcturusZhang]

Hi @bigglesuk69 and @caldwell2000 thanks for your discussions and I just had some experiments about this.

This is truly caused by the reference_id. When you create the set definition without assigning a reference_id, Azure will automatically create one for you and send it back - therefore this attribute is Computed.
But in the terraform implementation, we do not ensure this kind bind, which is saying that in a case of a reordering of the policy_definition_reference blocks, terraform will treat your change as you have changed the definition_id, but the reference_id remains the same. In this case, Azure will return error like posted in the description complaining about DuplicateReferencesInPolicySet. And the reason for the reordering should be that you are using a set to iterate the policy_definition_reference. In a hash set, the order is deterministic but not guaranteed, every time you add or remove or change the element of it, the iteration may reorder, and therefore comes this issue.

And also, to work this around, explicitly assigning reference_id by yourself works - this manages the bind between the reference_id and the definition_id, and therefore no duplicate error will pop out.

I am considering that we could change the type of policy_definition_reference from List to Set (but this would be considered as a breaking change), but I am not absolutely sure, is this list of policy_definition_reference guaranteed to be distinct and orderless?

[ArcturusZhang]

And just had a test that Azure actually allow two policy_definition_references with same definition_id, therefore set does not make sense on this block.

I am not quite sure ordering matters or not, but since it is not required to be distinct, we cannot make this a set.

[dgcaron]

the solution provided by @bigglesuk69 worked for me, i was looping over the created policies to create a initiative:

resource "azurerm_policy_set_definition" "initiative" {
  name         = "my_initiative"
  policy_type  = "Custom"
  display_name = "MyPolicy Set"

  parameters = <<PARAMETERS
 .. deleted
PARAMETERS

  dynamic "policy_definition_reference" {
    for_each = local.targets
    content {
      policy_definition_id = module.policies[policy_definition_reference.key].definition_id
      --> reference_id         = policy_definition_reference.key <-- adding this fixed it for me
      parameter_values     = <<VALUE
     .. deleted
    VALUE
    }
  }
}