Terraform-provider-azurerm: Terraform wants to delete metrics that intentionally are left out.

[EndureBlackout]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform: 0.12.26

Affected Resource(s)

• azurerm_monitor_diagnostic_setting

Terraform Configuration Files

resource "azurerm_monitor_diagnostic_setting" "api-diagnostics" {
  name               = "${var.global.main-rg.name}-api-diagnostics"
  target_resource_id = azurerm_api_management.api-management.id

  log_analytics_workspace_id     = var.global.log_analytics.id
  log_analytics_destination_type = "Dedicated"

  log {
    category = "GatewayLogs"
    enabled  = true

    retention_policy {
      enabled = true
      days    = var.global.log_analytics.retention_in_days
    }
  }

  metric {
    category = "Gateway Requests"
    enabled  = true

    retention_policy {
      enabled = true
      days    = var.global.log_analytics.retention_in_days
    }
  }
}

Expected Behavior

Only show the included metrics in the terraform plan

 ~ resource "azurerm_monitor_diagnostic_setting" "api-diagnostics" {
        id                             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.ApiManagement/service/theservice"
        log_analytics_destination_type = "Dedicated"
        log_analytics_workspace_id     = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.operationalinsights/workspaces/workspace"
        name                           = "api-diagnostics-name"
        target_resource_id             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.ApiManagement/service/theservice"

        log {
            category = "GatewayLogs"
            enabled  = true

            retention_policy {
                days    = 90
                enabled = true
            }
        }
        metric {
            category = "Gateway Requests"
            enabled  = true

            retention_policy {
                days    = 90
                enabled = true
            }
        }
    }

Actual Behavior

Shows metrics that weren't included in the configuration for destroy

 ~ resource "azurerm_monitor_diagnostic_setting" "api-diagnostics" {
        id                             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.ApiManagement/service/theservice"
        log_analytics_destination_type = "Dedicated"
        log_analytics_workspace_id     = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.operationalinsights/workspaces/workspace"
        name                           = "api-diagnostics-name"
        target_resource_id             = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group/providers/Microsoft.ApiManagement/service/theservice"

        log {
            category = "GatewayLogs"
            enabled  = true

            retention_policy {
                days    = 90
                enabled = true
            }
        }
      - metric {
          - category = "Capacity" -> null
          - enabled  = false -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      - metric {
          - category = "EventHub Events" -> null
          - enabled  = false -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
        metric {
            category = "Gateway Requests"
            enabled  = true

            retention_policy {
                days    = 90
                enabled = true
            }
        }
      - metric {
          - category = "Network Status" -> null
          - enabled  = false -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
    }

Steps to Reproduce

1. Create an azurerm_monitor_diagnostic_setting
2. Include metrics that you want to monitor
3. terraform plan

[davidtom]

I'm seeing this for logs too, and have tried to fix it by explicitly disabling them:

  log {
    category = "SSISIntegrationRuntimeLogs"
    enabled  = false
  }

Not ideal, but better than seeing these in all my plans. However, this just exacerbates the problem, as now my plans looks like this:

      - log {
          - category = "SSISIntegrationRuntimeLogs" -> null
          - enabled  = false -> null

          - retention_policy {
              - days    = 0 -> null
              - enabled = false -> null
            }
        }
      + log {
          + category = "SSISIntegrationRuntimeLogs"
          + enabled  = false
        }

[magodo]

Hi @EndureBlackout Thank you for submitting this!

As you know, terraform config is describing the final state of the representation of the resource. This resource (azurerm_monitor_diagnostic_setting) is a bit special. Different resource (specified by target_resource_id) has a different set of metric categories and log categories. In which case, if you didn't specify the complete available metric/log categories, then the absent ones are regarded as to be absent in the resource representation. But the case is that each resource has a default setting to those categories, which means a refresh will bring these defaults in, and if you didn't specify those in terraform config, you are telling terraform you want to remove them (while actually they can't be removed).

I can tell that is not the intended behavior, users just want to prefer convention over configuration in this case. I think in this case we can leverage the SuppressDiffFunc to suppress terraform from detecting diff if the only diff is the absence of the default setting. I'll submit a PR to fix this.

[EndureBlackout]

@magodo Hey! I appreciate it! Can be closed with PR when merged :)

[jonmaestas]

Just dropping this here for others. I ran into this some time back and it was very annoying when it showed an update every time.

In our case we want logs to go to a Storage Account, and metrics to go to Log Analytics Workspace. This is for cost reasons since Log Analytics is pricey for ingesting logs, but we still want metrics to go here so we can create dashboards and alerts from these.

This is what I did to fix this, which lines up to what @magodo had explained as expected behavior above in his https://github.com/terraform-providers/terraform-provider-azurerm/issues/7235#issuecomment-647974840.

1. Pull a dynamic list of Logs and Metrics available (using the azurerm_monitor_diagnostic_categories data) for the resource. Each resource type has different values so we need to use a dynamic block to iterate through the values.
2. In the first azurerm_monitor_diagnostic_setting resource I set each Metric (using the dynamic data.azurerm_monitor_diagnostic_categories.default.metrics values) to point to a Log Analytics Workspace and set the Logs (using the dynamic data.azurerm_monitor_diagnostic_categories.default.logs values) to the default values when they are disabled.
3. In the second azurerm_monitor_diagnostic_setting resource I am doing the same as the previous, but setting Logs to go to a Storage Account and setting the Metrics to their default values when disabled.

This is what my diagnostic setting module looks like:

variable resource_id {
  description = "ID of Resource to Enable"
}

variable log_analytics_workspace_id {
  description = "Log Analytics Workspace ID to store Diagnostic Metrics"
}

variable storage_account_id {
  description = "Storage Account ID to store Diagnostic Logs"
}

# https://www.terraform.io/docs/providers/azurerm/d/monitor_diagnostic_categories.html
data azurerm_monitor_diagnostic_categories default {
  resource_id = var.resource_id
}

# https://www.terraform.io/docs/providers/azurerm/r/monitor_diagnostic_setting.html
resource azurerm_monitor_diagnostic_setting metrics {
  name               = "diagnostic_metrics"
  target_resource_id = var.resource_id

  log_analytics_workspace_id = var.log_analytics_workspace_id

  dynamic metric {
    for_each = sort(data.azurerm_monitor_diagnostic_categories.default.metrics)
    content {
      category = metric.value
      enabled  = true

      retention_policy {
        enabled = true
        days    = 180
      }
    }
  }

  # this needs to be here with enabled = false to prevent TF from showing changes happening with each plan/apply
  dynamic log {
    for_each = sort(data.azurerm_monitor_diagnostic_categories.default.logs)
    content {
      category = log.value
      enabled  = false

      retention_policy {
        enabled = false
        days    = 0
      }
    }
  }
}

# https://www.terraform.io/docs/providers/azurerm/r/monitor_diagnostic_setting.html
resource azurerm_monitor_diagnostic_setting logs {
  name               = "diagnostic_logs"
  target_resource_id = var.resource_id

  storage_account_id = var.storage_account_id

  dynamic log {
    for_each = sort(data.azurerm_monitor_diagnostic_categories.default.logs)
    content {
      category = log.value
      enabled  = true

      retention_policy {
        enabled = true
        days    = 30
      }
    }
  }

  # this needs to be here with enabled = false to prevent TF from showing changes happening with each plan/apply
  dynamic metric {
    for_each = sort(data.azurerm_monitor_diagnostic_categories.default.metrics)
    content {
      category = metric.value
      enabled  = false

      retention_policy {
        enabled = false
        days    = 0
      }
    }
  }
}

Hope this helps.