Terraform-provider-azurerm: Azure RM 2.0 extension approach incompatible with ServiceFabricNode extension requirements of being added at VMSS creation time.

[JSchenken]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.21

• provider.azurerm v2.0.0

Affected Resource(s)

azurerm_windows_virtual_machine_scale_set
azurerm_virtual_machine_scale_set_extension

Terraform Configuration Files

resource "azurerm_virtual_machine_scale_set_extension" "service_fabric_node" {
    virtual_machine_scale_set_id = azurerm_windows_virtual_machine_scale_set.main.id
    name                       = "${var.node_type}_ServiceFabricNode"
    publisher                  = "Microsoft.Azure.ServiceFabric"
    type                       = "ServiceFabricNode"
    type_handler_version       = "1.1"
    auto_upgrade_minor_version = false
    protected_settings         = <<PROTECTEDSETTINGS
        {
            "StorageAccountKey1": "${var.support_log_storage_account_primary_access_key}",
            "StorageAccountKey2": "${var.support_log_storage_account_secondary_access_key}"
        }
PROTECTEDSETTINGS
    settings                   = <<SETTINGS
        {
            "clusterEndpoint": "${var.cluster_endpoint}",
            "nodeTypeRef": "${var.node_type}",
            "dataPath": ${jsonencode(var.fabric_data_root)},
            "durabilityLevel": "${var.durability_level}",
            "certificate": {
                "commonNames": ["${var.sfserver_cert_common_name}"],
                "x509StoreName": "${var.certificate_store_value}"
            },
            "nicPrefixOverride": ${jsonencode(var.subnetprefix)}
        }
SETTINGS
  }

Debug Output

Panic Output

Expected Behavior

The extension should have been included in the VMSS definition at creation time, and not attempted to be added after the VMSS was already created.

Actual Behavior

The VMSS is created and then there is an attempt to add the extension to the VMSS, this is not workable for ServiceFabricNode extension, as seen in error message:

Error: Error creating Extension "management_ServiceFabricNode" (Virtual Machine Scale Set "management" / Resource Group "testgroup"): compute.VirtualMachineScaleSetExtensionsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status= Code="OperationNotAllowed" Message="VM Scale Set extensions of handler 'Microsoft.Azure.ServiceFabric.ServiceFabricNode' can be added only at the time of VM Scale Set creation."

on ....\modules\sf_nonhypernet_vmss\main.tf line 194, in resource "azurerm_virtual_machine_scale_set_extension" "service_fabric_node":
194: resource "azurerm_virtual_machine_scale_set_extension" "service_fabric_node" {

Steps to Reproduce

Attempt to define & create a VMSS with a ServiceFabricNode extension

References

• #0000

[anhowe]

This also affects custom script extension with azurerm_linux_virtual_machine_scale_set. By not in lining extension with the VMSS object, the extension installs and is stuck in "Creating" state until you run a rolling upgrade on the VMSS. Unfortunately, you can't easily set the upgrade state to "automatic" as that requires installation of a probe.

The only workaround is to use cloud-init, but cloud-init doesn't block on failures of the script. Also, cloud-init doesn't have the ability to take secrets like custom script's "protectedSettings" field.

Additionally this results in a performance penalty, since applying extensions in two steps causes the whole deployment to take much longer as it has to wait until all nodes are up before applying the extension, and the slowness of the rolling upgrade.

We want to use azurerm_linux_virtual_machine_scale_set to get access to new features like ephemeral OS disks.

[ShreyasNBS]

Hi,

Do we have an ETA on when this will be fixed? Currently getting around this by attaching the load balancer probe to scaleset, but I would prefer to de-couple them and set UpgradePolicy to Manual. Would be good to know if it's going to be picked up in any upcoming Milestone.

[neil-yechenwei]

It seems that vmss extension “Microsoft.Azure.ServiceFabric” cannot be added to existing vmss in vmss v2. In vmss v1, vmss and vmss extension would be created together. However, in vmss v2, vmss and vmss extension would be created in separate resource. So I think "Microsoft.Azure.ServiceFabric" can be created successfully with vmss v1.
@tombuildsstuff , could you also help to have a look? Thanks.

[tombuildsstuff]

@neil-yechenwei I'd suggest reaching out to the ServiceFabric team here, since this is an issue with the ServiceFabric Extension

[hbuckle]

@tombuildsstuff - I don't think it's an issue with the service fabric extension as such, Azure clearly allows extensions to require they are added at creation time so Terraform needs to account for that option. I doubt the service fabric team are going to change the behaviour, so the new scale set resource is going to remain incompatible.

[tedvanderveen]

@neil-yechenwei @tombuildsstuff any updates you can share on this?

[trav]

I'm also blocked by this. Seems like the extension should be considered as part of the creation process. Without a change, here, I'm not sure how to hook service fabric up to the vmss.

[AdamCoulterOz]

I am having a problem with this exact problem too (no option for provisioning extensions in-band with the vmss).

In my example I'm setting the Linux_VMSS; upgrade_mode = "rolling". This requires setting either:

• health_probe_id from a LB - reaches the node externally
• ApplicationHealthLinux extension - determines node health internally

I can't use an external LB as the app I am provisioning on the Linux VMSS is (ironically) HashiCorp Consul and doesn't use normal load balancing. I literally need to run health checking from the local vm using the loopback address (as the service I need to check isn't even exposed on anything else).

This creates a circular dependency, which I cant even get around by first provisioning the vmss as upgrade_mode = "manual", provisioning the vmss_extension, then changing the upgrade_mode attribute back as it is a ForceNew attribute!!! ...

upgrade_mode = "Manual" -> "Rolling" # forces replacement

... bringing me full circle....

please fix this...

[marcelkmfst]

Sam issue here - no chance to get this working... I use Application Gateway and no LB so the health extension is the only option...

[ArcturusZhang]

I am having a problem with this exact problem too (no option for provisioning extensions in-band with the vmss).

In my example I'm setting the Linux_VMSS; upgrade_mode = "rolling". This requires setting either:

• health_probe_id from a LB - reaches the node externally
• ApplicationHealthLinux extension - determines node health internally

I can't use an external LB as the app I am provisioning on the Linux VMSS is (ironically) HashiCorp Consul and doesn't use normal load balancing. I literally need to run health checking from the local vm using the loopback address (as the service I need to check isn't even exposed on anything else).

This creates a circular dependency, which I cant even get around by first provisioning the vmss as upgrade_mode = "manual", provisioning the vmss_extension, then changing the upgrade_mode attribute back as it is a ForceNew attribute!!! ...

upgrade_mode = "Manual" -> "Rolling" # forces replacement

... bringing me full circle....

please fix this...

LOL I have met this conundrum once while making the terminate_notification feature for VMSS. And lucky for me, in my case, health_probe_id also works...

[giovannifl]

This bug also affects custom script extension with azurerm_windows_virtual_machine_scale_set.

[ShreyasNBS]

So @tombuildsstuff are we to assume that this issue is not important enough for Terraform to fix? This has been there for months and it's frustrating that an "upstream" tag has been attached to it when it's got nothing to do with upstream. There is no "service fabric" limitation here. It's an issue happening with "every" scaleset we creating. Are you saying we should not use scalesets with extensions in the normal fashion (create scaleset and then apply extension)?

[hbuckle]

Just to clarify, I believe the reason the service fabric extension has to be added at creation time is because service fabric blocks some features of scale sets, so the extension needs to be present from the start to ensure the scale set is not created with incompatible options.

[ShreyasNBS]

@hbuckle is this new in AzureRM v2? I had the setup runing quite well in AzureRM v1 provider without using service fabric extension. Just curious. Any links to documentation would be appreciated please.

[hbuckle]

Service Fabric is the only extension I am aware of that is _required_ to be added at creation time. Other extensions should work fine being added post-creation, with the caveat that depending on the scale set settings you may need to trigger a rolling upgrade to get the extension to actually apply.

[ShreyasNBS]

That's interesting. When I went through all the tutorials for creating a VMSS, they all had instructions to create a VMSS and LB infront of it. No mention of Service Fabric mandatory extension. I currently have my VMSS running with LB probe setup without Service Fabric extension. MIght need to do some more reading.

[hbuckle]

When I say required, I mean it's required if you are creating a Service Fabric cluster - for other workloads obviously you don't need it.

[jackofallops]

I'm taking a look into this at the moment. It's a deceptively tricky thing to re-introduce, but we'll do what we can.

[rhys96]

I'm taking a look into this at the moment. It's a deceptively tricky thing to re-introduce, but we'll do what we can.

I was about to start a PR to create this feature because I'm dependent on this feature for #8120, however @tombuildsstuff mentioned you were about start something. If it helps, I started to add the following in virtual_machine_scale_set.go:

func VirtualMachineScaleSetExtensionSchema() *schema.Schema {
    return &schema.Schema{
        Type:     schema.TypeList,
        Optional: true,
        Elem: &schema.Resource{
            Schema: map[string]*schema.Schema{

                "name": {
                    Type:         schema.TypeString,
                    Required:     true,
                    ValidateFunc: validation.StringIsNotEmpty,
                },

                "publisher": {
                    Type:         schema.TypeString,
                    Required:     true,
                    ValidateFunc: validation.StringIsNotEmpty,
                },

                "type": {
                    Type:         schema.TypeString,
                    Required:     true,
                    ValidateFunc: validation.StringIsNotEmpty,
                },

                "type_handler_version": {
                    Type:         schema.TypeString,
                    Required:     true,
                    ValidateFunc: validation.StringIsNotEmpty,
                },

                "auto_upgrade_minor_version": {
                    Type:     schema.TypeBool,
                    Optional: true,
                    Default:  true,
                },

                "enable_automatic_upgrade": {
                    Type:     schema.TypeBool,
                    Optional: true,
                    Default:  true,
                },

                "force_update_tag": {
                    Type:     schema.TypeString,
                    Optional: true,
                },

                "protected_settings": {
                    Type:             schema.TypeString,
                    Optional:         true,
                    Sensitive:        true,
                    ValidateFunc:     validation.StringIsJSON,
                    DiffSuppressFunc: structure.SuppressJsonDiff,
                },

                "provision_after_extensions": {
                    Type:     schema.TypeList,
                    Optional: true,
                    Elem: &schema.Schema{
                        Type: schema.TypeString,
                    },
                },

                "settings": {
                    Type:             schema.TypeString,
                    Optional:         true,
                    ValidateFunc:     validation.StringIsJSON,
                    DiffSuppressFunc: structure.SuppressJsonDiff,
                },
            },
        },
    }
}

func ExpandVirtualMachineScaleSetExtension(input []interface{}) (*[]compute.VirtualMachineScaleSetExtension, error) {
    extensions := make([]compute.VirtualMachineScaleSetExtension, 0)

    for _, v := range input {
        raw := v.(map[string]interface{})

        settings := map[string]interface{}{}
        if settingsString := (raw["settings"].(string)); settingsString != "" {
            s, err := structure.ExpandJsonFromString(settingsString)
            if err != nil {
                return nil, fmt.Errorf("unable to parse settings: %s", err)
            }
            settings = s
        }

        provisionAfterExtensionsRaw := raw["provision_after_extensions"].([]interface{})
        provisionAfterExtensions := utils.ExpandStringSlice(provisionAfterExtensionsRaw)

        protectedSettings := map[string]interface{}{}
        if protectedSettingsString := raw["protected_settings"].(string); protectedSettingsString != "" {
            ps, err := structure.ExpandJsonFromString(protectedSettingsString)
            if err != nil {
                return nil, fmt.Errorf("unable to parse protected_settings: %s", err)
            }
            protectedSettings = ps

        }

        extension := compute.VirtualMachineScaleSetExtension{
            Name: utils.String(raw["name"].(string)),
            VirtualMachineScaleSetExtensionProperties: &compute.VirtualMachineScaleSetExtensionProperties{
                Publisher:                utils.String(raw["publisher"].(string)),
                Type:                     utils.String(raw["type"].(string)),
                TypeHandlerVersion:       utils.String(raw["type_handler_version"].(string)),
                AutoUpgradeMinorVersion:  utils.Bool(raw["auto_upgrade_minor_version"].(bool)),
                EnableAutomaticUpgrade:   utils.Bool(raw["enable_automatic_upgrade"].(bool)),
                ProtectedSettings:        protectedSettings,
                ProvisionAfterExtensions: provisionAfterExtensions,
                Settings:                 settings,
            },
        }

        extensions = append(extensions, extension)
    }

    return &extensions, nil
}

func FlattenVirtualMachineScaleSetExtension(input *[]compute.VirtualMachineScaleSetExtension) []interface{} {
    if input == nil {
        return []interface{}{}
    }

    output := make([]interface{}, 0)

    for _, v := range *input {
        // To do
    }

    return output
}

[jackofallops]

Thanks @rhys96 - I'm working through behaviours for create and update changes at the moment, if I can make use of your code I'll add you to the co-authors list 👍

[tfenster]

I am having the same issue and reverting to the old resource works, but would be very nice to use the new one

[jackofallops]

Just a quick note, beta functionality for this shipped in v2.26.0. To make use of it there's a new env var you can set to allow the use of the new extension property on both azurerm_linux_virtual_machine_scale_set and azurerm_windows_virtual_machine_scale_set. Set ARM_PROVIDER_VMSS_EXTENSIONS_BETA=true to use the new block.

I also included a basic example of the extension in use with Service Fabric extension in the examples folder also.
If there's feedback / issues with the beta functionality, please open a new issue (or add detail to any already open if someone else gets there first 😄 )

Thanks!

[josephzhaox]

Can I request this thread to be re-open ? I was trying to follow the sample but still got an error "unexpected end of JSON input"

2020-09-27T02:49:56.3333117Z azurerm_windows_virtual_machine_scale_set.sfab-vmss: Creating...
2020-09-27T02:49:56.6681025Z 
2020-09-27T02:49:56.6682264Z Error: failed to parse JSON from settings: unexpected end of JSON input
2020-09-27T02:49:56.6682838Z
2020-09-27T02:49:56.6683966Z  on service-fabric-vm.tf line 175, in resource "azurerm_windows_virtual_machine_scale_set" "sfab-vmss":
2020-09-27T02:49:56.6684680Z 175: resource "azurerm_windows_virtual_machine_scale_set" "sfab-vmss" {
2020-09-27T02:49:56.6685046Z 

[josephzhaox]

Let me know if you need me more information. thanks a million!

[josephzhaox]

Shall I open a new issue instead? Thanks

[jackofallops]

Shall I open a new issue instead? Thanks

Hi - There's an issue already open that I believe covers this, and a PR has also been opened to resolve.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!