Terraform-provider-azurerm: azurerm_windows_virtual_machine_scale_set incorrectly updates storage settings
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform (and AzureRM Provider) Version
terraform -v
Terraform v0.12.24
- provider.azurerm v2.9.0
Affected Resource(s)
azurerm_windows_virtual_machine_scale_set
Terraform Configuration Files
resource "azurerm_windows_virtual_machine_scale_set" "elastic_master" {
name = "elmas"
resource_group_name = azurerm_resource_group.elasticsearch.name
location = azurerm_resource_group.elasticsearch.location
sku = var.master_node_sku
instances = var.master_node_instances
admin_username = "azureuser"
admin_password = "Qwerty12345!"
overprovision = false
provision_vm_agent = true
timezone = "GMT Standard Time"
enable_automatic_updates = false
zones = var.zones
zone_balance = var.zone_balance
source_image_id = "/subscriptions/xxx/resourceGroups/packer-images-uksouth/providers/Microsoft.Compute/galleries/hbtest/images/hbtest-elastic/versions/1.0.5"
health_probe_id = azurerm_lb_probe.elasticsearch_http.id
upgrade_mode = "Rolling"
rolling_upgrade_policy {
max_batch_instance_percent = floor((1 / var.master_node_instances) * 100)
max_unhealthy_instance_percent = 100
max_unhealthy_upgraded_instance_percent = 0
pause_time_between_batches = "PT1M"
}
boot_diagnostics {
storage_account_uri = azurerm_storage_account.diagnostics.primary_blob_endpoint
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadOnly"
diff_disk_settings {
option = "Local"
}
}
data_disk {
storage_account_type = "Standard_LRS"
caching = "ReadOnly"
lun = 1
disk_size_gb = 10
create_option = "Empty"
}
identity {
type = "SystemAssigned, UserAssigned"
identity_ids = [
azurerm_user_assigned_identity.cluster.id
]
}
network_interface {
name = "primary"
primary = true
network_security_group_id = azurerm_network_security_group.cluster["master"].id
ip_configuration {
name = "primary"
primary = true
subnet_id = var.subnet_id
application_security_group_ids = [
azurerm_application_security_group.cluster.id
]
load_balancer_backend_address_pool_ids = [
azurerm_lb_rule.elasticsearch_http["master"].backend_address_pool_id,
azurerm_lb_backend_address_pool.outbound.id,
]
}
}
}
Debug Output
This is the scale set model from the API to begin with
{
"name": "elmas",
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Compute/virtualMachineScaleSets/elmas",
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "uksouth",
"tags": {},
"identity": {
"type": "SystemAssigned, UserAssigned",
"principalId": "b5739d71-6857-4a06-b3cd-1f20536065d0",
"tenantId": "c66e509a-c812-41ee-9c00-ae1517b0bb7e",
"userAssignedIdentities": {
"/subscriptions/xxx/resourcegroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cbuk-core-hbtest-elasticsearch-elastic-uksouth": {
"principalId": "934d4683-c4ea-4959-95c7-2780975bde64",
"clientId": "c2cf571d-327d-481c-a133-58ffb142bafd"
}
}
},
"sku": {
"name": "Standard_D2s_v3",
"tier": "Standard",
"capacity": 3
},
"properties": {
"singlePlacementGroup": true,
"upgradePolicy": {
"mode": "Rolling",
"rollingUpgradePolicy": {
"maxBatchInstancePercent": 33,
"maxUnhealthyInstancePercent": 100,
"maxUnhealthyUpgradedInstancePercent": 0,
"pauseTimeBetweenBatches": "PT1M"
}
},
"scaleInPolicy": {
"rules": [
"Default"
]
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "elmas",
"adminUsername": "azureuser",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": false,
"timeZone": "GMT Standard Time",
"winRM": {
"listeners": []
}
},
"secrets": [],
"allowExtensionOperations": true,
"requireGuestProvisionSignal": true
},
"storageProfile": {
"osDisk": {
"diffDiskSettings": {
"option": "Local"
},
"createOption": "FromImage",
"caching": "ReadOnly",
"writeAcceleratorEnabled": false,
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"diskSizeGB": 30
},
"imageReference": {
"id": "/subscriptions/xxx/resourceGroups/packer-images-uksouth/providers/Microsoft.Compute/galleries/hbtest/images/hbtest-elastic/versions/1.0.5"
},
"dataDisks": [
{
"lun": 1,
"createOption": "Empty",
"caching": "ReadOnly",
"writeAcceleratorEnabled": false,
"managedDisk": {
"storageAccountType": "Standard_LRS"
},
"diskSizeGB": 10
}
]
},
"networkProfile": {"healthProbe":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/probes/elasticsearch_http"},"networkInterfaceConfigurations":[{"name":"primary","properties":{"primary":true,"enableAcceleratedNetworking":false,"networkSecurityGroup":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/networkSecurityGroups/master"},"dnsSettings":{"dnsServers":[]},"enableIPForwarding":false,"ipConfigurations":[{"name":"primary","properties":{"primary":true,"subnet":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-testoce-network-uksouth/providers/Microsoft.Network/virtualNetworks/testoce/subnets/elastic"},"privateIPAddressVersion":"IPv4","applicationSecurityGroups":[{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/applicationSecurityGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth"}],"loadBalancerBackendAddressPools":[{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/backendAddressPools/master"},{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/outbound/backendAddressPools/outbound"}]}}]}}]},
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": true,
"storageUri": "https://36few3807hp3syfyvymaguvr.blob.core.windows.net/"
}
},
"extensionProfile": {
"extensions": [
{
"name": "init-script",
"properties": {
"autoUpgradeMinorVersion": true,
"provisionAfterExtensions": [],
"forceUpdateTag": "368e7889cd30d2b98c6e5b6350c9b667c7145350",
"publisher": "Microsoft.Compute",
"type": "CustomScriptExtension",
"typeHandlerVersion": "1.10",
"settings": {"commandToExecute":"pwsh -noni -nop -file ./elasticsearch.ps1 -config_hash 368e7889cd30d2b98c6e5b6350c9b667c7145350","fileUris":["https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/Privilege.cs","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.ps1","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.psm1","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.yml","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/jvm.options","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/roles.yml","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/secrets.json"]}
}
}
]
},
"priority": "Regular"
},
"provisioningState": "Succeeded",
"overprovision": false,
"doNotRunExtensionsOnOverprovisionedVMs": false,
"uniqueId": "39ed1b28-1fdb-426e-8430-c8f5957c3916"
}
}
This is the PATCH request Terraform sends to the API
{
"identity": {
"type": "SystemAssigned, UserAssigned",
"userAssignedIdentities": {
"/subscriptions/xxx/resourcegroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cbuk-core-hbtest-elasticsearch-elastic-uksouth": {}
}
},
"properties": {
"upgradePolicy": {
"mode": "Rolling",
"rollingUpgradePolicy": {
"maxBatchInstancePercent": 33,
"maxUnhealthyInstancePercent": 100,
"maxUnhealthyUpgradedInstancePercent": 0,
"pauseTimeBetweenBatches": "PT1M"
}
},
"virtualMachineProfile": {
"storageProfile": {
"imageReference": {
"id": "/subscriptions/xxx/resourceGroups/packer-images-uksouth/providers/Microsoft.Compute/galleries/hbtest/images/hbtest-elastic/versions/1.0.6"
}
},
"networkProfile": {
"healthProbe": {
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/probes/elasticsearch_http"
},
"networkInterfaceConfigurations": [
{
"name": "primary",
"properties": {
"primary": true,
"enableAcceleratedNetworking": false,
"networkSecurityGroup": {
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/networkSecurityGroups/master"
},
"dnsSettings": {
"dnsServers": []
},
"ipConfigurations": [
{
"name": "primary",
"properties": {
"subnet": {
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-testoce-network-uksouth/providers/Microsoft.Network/virtualNetworks/testoce/subnets/elastic"
},
"primary": true,
"privateIPAddressVersion": "IPv4",
"applicationGatewayBackendAddressPools": [],
"applicationSecurityGroups": [
{
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/applicationSecurityGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth"
}
],
"loadBalancerBackendAddressPools": [
{
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/backendAddressPools/master"
},
{
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/outbound/backendAddressPools/outbound"
}
],
"loadBalancerInboundNatPools": []
}
}
],
"enableIPForwarding": false
}
}
]
}
}
}
}
Which results in a scale set model of
{
"name": "elmas",
"id": "/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Compute/virtualMachineScaleSets/elmas",
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "uksouth",
"tags": {},
"identity": {
"type": "SystemAssigned, UserAssigned",
"principalId": "b5739d71-6857-4a06-b3cd-1f20536065d0",
"tenantId": "c66e509a-c812-41ee-9c00-ae1517b0bb7e",
"userAssignedIdentities": {
"/subscriptions/xxx/resourcegroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cbuk-core-hbtest-elasticsearch-elastic-uksouth": {}
}
},
"sku": {
"name": "Standard_D2s_v3",
"tier": "Standard",
"capacity": 3
},
"properties": {
"singlePlacementGroup": true,
"upgradePolicy": {
"mode": "Rolling",
"rollingUpgradePolicy": {
"maxBatchInstancePercent": 33,
"maxUnhealthyInstancePercent": 100,
"maxUnhealthyUpgradedInstancePercent": 0,
"pauseTimeBetweenBatches": "PT1M"
}
},
"scaleInPolicy": {
"rules": [
"Default"
]
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "elmas",
"adminUsername": "azureuser",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": false,
"timeZone": "GMT Standard Time",
"winRM": {
"listeners": []
}
},
"secrets": [],
"allowExtensionOperations": true,
"requireGuestProvisionSignal": true
},
"storageProfile": {
"osDisk": {
"createOption": "FromImage",
"caching": "ReadOnly",
"managedDisk": {
"storageAccountType": "Premium_LRS"
},
"diskSizeGB": 30
},
"imageReference": {
"id": "/subscriptions/xxx/resourceGroups/packer-images-uksouth/providers/Microsoft.Compute/galleries/hbtest/images/hbtest-elastic/versions/1.0.6"
}
},
"networkProfile": {"healthProbe":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/probes/elasticsearch_http"},"networkInterfaceConfigurations":[{"name":"primary","properties":{"primary":true,"enableAcceleratedNetworking":false,"networkSecurityGroup":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/networkSecurityGroups/master"},"dnsSettings":{"dnsServers":[]},"enableIPForwarding":false,"ipConfigurations":[{"name":"primary","properties":{"primary":true,"subnet":{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-testoce-network-uksouth/providers/Microsoft.Network/virtualNetworks/testoce/subnets/elastic"},"privateIPAddressVersion":"IPv4","applicationSecurityGroups":[{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/applicationSecurityGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth"}],"loadBalancerBackendAddressPools":[{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/cluster/backendAddressPools/master"},{"id":"/subscriptions/xxx/resourceGroups/cbuk-core-hbtest-elasticsearch-elastic-uksouth/providers/Microsoft.Network/loadBalancers/outbound/backendAddressPools/outbound"}]}}]}}]},
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": true,
"storageUri": "https://36few3807hp3syfyvymaguvr.blob.core.windows.net/"
}
},
"extensionProfile": {
"extensions": [
{
"name": "init-script",
"properties": {
"autoUpgradeMinorVersion": true,
"provisionAfterExtensions": [],
"forceUpdateTag": "368e7889cd30d2b98c6e5b6350c9b667c7145350",
"publisher": "Microsoft.Compute",
"type": "CustomScriptExtension",
"typeHandlerVersion": "1.10",
"settings": {"commandToExecute":"pwsh -noni -nop -file ./elasticsearch.ps1 -config_hash 368e7889cd30d2b98c6e5b6350c9b667c7145350","fileUris":["https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/Privilege.cs","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.ps1","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.psm1","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/elasticsearch.yml","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/jvm.options","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/roles.yml","https://7j67dcj6fweg8ejpshb4fylf.blob.core.windows.net/master/secrets.json"]}
}
}
]
},
"priority": "Regular"
},
"provisioningState": "Updating",
"overprovision": false,
"doNotRunExtensionsOnOverprovisionedVMs": false,
"uniqueId": "39ed1b28-1fdb-426e-8430-c8f5957c3916"
}
}
The data disks and diff disk settings are now missing
Expected Behavior
Updating the source_image_id works correctly
Actual Behavior
The scale set is broken because the data disk and ephemeral OS disk settings are removed.
It would appear that the VMSS api requires the complete storageProfile to be sent, including settings that have not changed, otherwise it removes the data disks and ephemeral os disk settings.
Steps to Reproduce
Changing the source_image_id and running terraform apply
References
Azure PowerShell appears to have the same issue
The Azure CLI works as expected as it sends the complete scale set model with the update request
hbuckle
👍11
All 12 comments
See also #6286
hbuckle
on 14 May 2020
👍2
I've had a slightly cryptic update from my Microsoft case on this issue saying Product Group has found the issue and they will apply a fix for it. - which seems to imply the API is at fault here (it certainly seems wrong that a patch operation is unsetting properties). I'll update when I hear more
hbuckle
on 21 May 2020
Thanks for the update @hbuckle. Any idea if this fix will be for both the windows and linux scaleset API's?
opsj
on 21 May 2020
It should do, as they both use the same API. Received another update today
Our product group has identified the issue “VMSS with SIG should not empty DataDisks with null DataDisks input” and ETA for the fix is 6/30/2020 .
hbuckle
on 26 May 2020
@hbuckle Any updates from Microsoft on this?
opsj
on 14 Jul 2020
They have now said it will be the end of July for the fix
hbuckle
on 14 Jul 2020
@hbuckle do you have a update from Microsoft on this? If possible are you able to share the ticket number opened?
rohrerb
on 27 Jul 2020
Hi @hbuckle thanks for this issue!
The VMSS service team has been aware of this issue and working on a fix for this. I also open an issue in the azure-rest-api-specs to track the progress.
ArcturusZhang
on 7 Aug 2020
I had a reply on my Microsoft support ticket today saying the fix is now end of August 😢
hbuckle
on 7 Aug 2020
Not sure why I'm surprised at this point, but the fix is now end of September
hbuckle
on 1 Sep 2020
hi @hbuckle would you mind sharing whether the fix was completed and that it has resolved the issue?
avirishuv
on 15 Oct 2020
Hi @hbuckle this issue should have been fixed by the service side and no changes needed in the terraform side. Could you please have a try to see if this is resolved or not? Thank you.
I just had a try of updating SKU and all the data disks are properly retained.
ArcturusZhang
on 23 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
alant94
·
3Comments
tomasaschan
·
3Comments
rolandjohann
·
3Comments
pdeva
·
3Comments
akamalov
·
3Comments
Most helpful comment
See also #6286