Azure-sdk-for-go: CustomData is not working

[DavidGOrtega]

I try to setup customData in my machine

"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2020-06-30/compute"
"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-11-01/network"
"github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2020-06-01/resources"

"github.com/Azure/go-autorest/autorest/azure/auth"
"github.com/Azure/go-autorest/autorest/to"

go1.15 darwin/amd64

vmClient, _ := getVMClient(subscriptionID)
    futureVM, err := vmClient.CreateOrUpdate(
        ctx,
        gpName,
        vmName,
        compute.VirtualMachine{
            Location: to.StringPtr(region),
            /*
                Plan: &compute.Plan{
                    Name:      to.StringPtr(sku),
                    Publisher: to.StringPtr(publisher),
                    Product:   to.StringPtr(offer),
                },
            */
            VirtualMachineProperties: &compute.VirtualMachineProperties{
                HardwareProfile: &compute.HardwareProfile{
                    VMSize: compute.VirtualMachineSizeTypes(instanceType),
                },
                StorageProfile: &compute.StorageProfile{
                    ImageReference: &compute.ImageReference{
                        Publisher: to.StringPtr(publisher),
                        Offer:     to.StringPtr(offer),
                        Sku:       to.StringPtr(sku),
                        Version:   to.StringPtr(version),
                    },
                    OsDisk: &compute.OSDisk{
                        Name:         to.StringPtr(fmt.Sprintf(vmName + "-hdd")),
                        Caching:      compute.CachingTypesReadWrite,
                        CreateOption: compute.DiskCreateOptionTypesFromImage,
                        DiskSizeGB:   to.Int32Ptr(hddSize),
                        ManagedDisk: &compute.ManagedDiskParameters{
                            StorageAccountType: compute.StorageAccountTypesStandardLRS,
                        },
                    },
                },
                OsProfile: &compute.OSProfile{
                    CustomData:    to.StringPtr(customData),
                    AdminUsername: to.StringPtr(username),
                    LinuxConfiguration: &compute.LinuxConfiguration{
                        DisablePasswordAuthentication: to.BoolPtr(false),
                        SSH: &compute.SSHConfiguration{
                            PublicKeys: &[]compute.SSHPublicKey{
                                {
                                    Path:    to.StringPtr(fmt.Sprintf("/home/%s/.ssh/authorized_keys", username)),
                                    KeyData: to.StringPtr(keyPublic),
                                },
                            },
                        },
                    },
                },
                NetworkProfile: &compute.NetworkProfile{
                    NetworkInterfaces: &[]compute.NetworkInterfaceReference{
                        {
                            ID: nic.ID,
                            NetworkInterfaceReferenceProperties: &compute.NetworkInterfaceReferenceProperties{
                                Primary: to.BoolPtr(true),
                            },
                        },
                    },
                },
            },
        },
    )

where customData is generated by

buf, err := ioutil.ReadFile("./provision.sh")
if err != nil {
diags = append(diags, diag.Diagnostic{
    Severity: diag.Error,
    Summary:  fmt.Sprintf("Error creating cloudinit: %v", err),
})

return diags
}
customData := base64.StdEncoding.EncodeToString(buf)

The content of provision.sh is

#!/bin/bash

echo "APT::Get::Assume-Yes \"true\";" | sudo tee -a /etc/apt/apt.conf.d/90assumeyes

curl -sL https://deb.nodesource.com/setup_12.x | sudo bash
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update && sudo apt-get install -y terraform nodejs
sudo npm install -g git+https://github.com/iterative/cml.git#cml-runner

if I do a terraform --help inside the machine the command is not there

If I use the decoded base64 generated by my code to provision a machine within the portal it works perfectly.

[ArcturusZhang]

Hi @DavidGOrtega thanks for this issue!

Could you please enable the SDK request body logging to determine whether the customData field is in the REST API request body?
You could enable the logging by exporting an env var AZURE_GO_SDK_LOG_LEVEL=DEBUG

[DavidGOrtega]

@ArcturusZhang thanks for the response.
I have checked that the customData works. However, the ready state of the VM is way before the customData has been effectively put in place. As you may have been noticed I writing a TF provider. The issue is that the provider tests have TF provisioners (SSH) that are being executed after the VM is ready (i determine that using this lib).

Should not the futureVM.WaitForCompletionRef(ctx, vmClient.Client) wait for the customData to be setup?

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Drewm3, @avirishuv.

Issue Details

---

I try to setup customData in my machine

"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2020-06-30/compute"
"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-11-01/network"
"github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2020-06-01/resources"

"github.com/Azure/go-autorest/autorest/azure/auth"
"github.com/Azure/go-autorest/autorest/to"

go1.15 darwin/amd64

vmClient, _ := getVMClient(subscriptionID)
    futureVM, err := vmClient.CreateOrUpdate(
        ctx,
        gpName,
        vmName,
        compute.VirtualMachine{
            Location: to.StringPtr(region),
            /*
                Plan: &compute.Plan{
                    Name:      to.StringPtr(sku),
                    Publisher: to.StringPtr(publisher),
                    Product:   to.StringPtr(offer),
                },
            */
            VirtualMachineProperties: &compute.VirtualMachineProperties{
                HardwareProfile: &compute.HardwareProfile{
                    VMSize: compute.VirtualMachineSizeTypes(instanceType),
                },
                StorageProfile: &compute.StorageProfile{
                    ImageReference: &compute.ImageReference{
                        Publisher: to.StringPtr(publisher),
                        Offer:     to.StringPtr(offer),
                        Sku:       to.StringPtr(sku),
                        Version:   to.StringPtr(version),
                    },
                    OsDisk: &compute.OSDisk{
                        Name:         to.StringPtr(fmt.Sprintf(vmName + "-hdd")),
                        Caching:      compute.CachingTypesReadWrite,
                        CreateOption: compute.DiskCreateOptionTypesFromImage,
                        DiskSizeGB:   to.Int32Ptr(hddSize),
                        ManagedDisk: &compute.ManagedDiskParameters{
                            StorageAccountType: compute.StorageAccountTypesStandardLRS,
                        },
                    },
                },
                OsProfile: &compute.OSProfile{
                    CustomData:    to.StringPtr(customData),
                    AdminUsername: to.StringPtr(username),
                    LinuxConfiguration: &compute.LinuxConfiguration{
                        DisablePasswordAuthentication: to.BoolPtr(false),
                        SSH: &compute.SSHConfiguration{
                            PublicKeys: &[]compute.SSHPublicKey{
                                {
                                    Path:    to.StringPtr(fmt.Sprintf("/home/%s/.ssh/authorized_keys", username)),
                                    KeyData: to.StringPtr(keyPublic),
                                },
                            },
                        },
                    },
                },
                NetworkProfile: &compute.NetworkProfile{
                    NetworkInterfaces: &[]compute.NetworkInterfaceReference{
                        {
                            ID: nic.ID,
                            NetworkInterfaceReferenceProperties: &compute.NetworkInterfaceReferenceProperties{
                                Primary: to.BoolPtr(true),
                            },
                        },
                    },
                },
            },
        },
    )

where customData is generated by

buf, err := ioutil.ReadFile("./provision.sh")
if err != nil {
diags = append(diags, diag.Diagnostic{
    Severity: diag.Error,
    Summary:  fmt.Sprintf("Error creating cloudinit: %v", err),
})

return diags
}
customData := base64.StdEncoding.EncodeToString(buf)

The content of provision.sh is

#!/bin/bash

echo "APT::Get::Assume-Yes \"true\";" | sudo tee -a /etc/apt/apt.conf.d/90assumeyes

curl -sL https://deb.nodesource.com/setup_12.x | sudo bash
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update && sudo apt-get install -y terraform nodejs
sudo npm install -g git+https://github.com/iterative/cml.git#cml-runner

if I do a terraform --help inside the machine the command is not there

If I use the decoded base64 generated by my code to provision a machine within the portal it works perfectly.

Author:	DavidGOrtega
Assignees:	-
Labels:	`Compute - VM`, `Service Attention`, `customer-reported`, `question`
Milestone:	-

[ArcturusZhang]

Hi @DavidGOrtega unfortunately I do not know the answer. Therefore I tagged this issue so that the corresponding service team could have a look at this issue and might provide an answer and/or solution for this.

[amjads1]

@jaylabell Could you please look into this issue?

[jaylabell]

Aiming to reproduce this issue before introducing development.
Will follow up with updates week of 1/26

[Drewm3]

@DavidGOrtega, I believe you are saying that the issue here is that the VM is going to the ready state via the Azure Control plane before CloudInit executes your script that you provided via CustomData, and your expectation was that the VM would not be in the ready state until the script is executed. Is this the correct understanding of the issue?

Assuming the above is correct, then the behavior you are seeing is expected. The nature of CustomData/CloudInit is that it is executed as a "fire and forget" type of approach. The Azure platform doesn't wait for a response, and in fact the Azure platform doesn't even know CloudInit is running. Instead the Azure platform is just making the CustomData available to the VM.

Let me bring in @danielsollondon who is the CloudInit expert in the Azure Compute team to see if he has recommendations on how to effectively wait for a script to be executed by CloudInit. Daniel, please let David know if there are good options to wait on a script to be completed when using CloudInit.

The Azure platform mechanism to do what you want is through the use of Custom Script Extension. If you use an extension, then the VM does not return success on the create until all extensions return success, and the custom script extension doesn't return success until the script has executed. Using custom script is a bit more complex than CloudInit, but it will provide the behavior that you are looking to see out of the box.

[Drewm3]

Closing this issue because it appears to be by design. @DavidGOrtega, please let me know if my understanding from above is not correct.

[DavidGOrtega]

:wave: @Drewm3 thanks for the support.

please let me know if my understanding from above is not correct.

You are totally correct.

The Azure platform mechanism to do what you want is through the use of Custom Script Extension.

Can be implemented with this sdk?

[ArcturusZhang]

The Azure platform mechanism to do what you want is through the use of Custom Script Extension.

This is achievable in SDK if we could do it using REST API. Could you confirm? @Drewm3

[tombuildsstuff]

@Drewm3 for comparison other cloud providers do wait for the CloudInit to be completed here before marking the machine as "ready" - shouldn't this be a conditional behaviour that's exposed (in the same way as RequireGuestProvisionSignal is)?

[Drewm3]

@tombuildsstuff, could you provide a pointer to how this is done elsewhere? CustomData itself is just a data channel. The platform doesn't actually know that the data is going to be handled by something like CloudInit because there is no guarantee that CloudInit is installed on 100% of VMs. I believe to actually have the platform wait for CloudInit would require that the customer certify in some manner that CloudInit is installed on the image, and the details provided in CustomData is going to be processed by CloudInit, and then the platform would need to integrate with CloudInit in some manner to determine when CloudInit completed the execution of the script that was provided in CustomData.

Regardless, this is a feature request as compared to a question. As a feature request, please provide upvotes for the request over in Azure UserVoice. Here is the link for this request: https://feedback.azure.com/forums/216843-virtual-machines/suggestions/42835188-vm-ready-should-wait-for-scripts-in-customdata-to

[tombuildsstuff]

@Drewm3 I don't have a direct link to that unfortunately, but here's a link to the EC2 documentation: https://docs.aws.amazon.com/ec2/index.html

[DavidGOrtega]

@Drewm3 could you please leat me know if this can be achieved with this SDK? 🙏