Azure-sdk-for-go: Azure Compute API is not properly validating which VM types can deployed in an Availability Zone

[justaugustus]

Bug Report

I'm going to preface this with, "I have no clue where the best place to file this is, so apologies if it's in the wrong tracker".

• import path of package in question: github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute
• SDK version:
  github.com/Azure/azure-sdk-for-go v33.4.0+incompatible github.com/Azure/go-autorest/autorest v0.9.0 github.com/Azure/go-autorest/autorest/azure/auth v0.3.0 github.com/Azure/go-autorest/autorest/to v0.3.0 github.com/Azure/go-autorest/autorest/validation v0.2.0 // indirect
• output of go version: go version go1.12.9 linux/amd64

What happened?

This was originally filed in https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/294.
Cluster API Azure, as of filing this, does the following:

• Assumes that a user should want a VM with AZ and will attempt to create one by default
• Use a user-defined AZ. If one can't be found:
  
  
  
  • Attempts to find an available AZ for a VM, based on VM size and location
    
    
    
    
    
    • If an AZ can't be found, create the non-AZ VM
    
    
    
  
  

The code for that is here:

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/master/controllers/azuremachine_reconciler.go#L175-L223
• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/7f17ca164331dcfbcf9f1fa9e268635146e19c73/cloud/services/availabilityzones/availabilityzones.go

The reconciler finds an AZ to use and when the request is submitted, we get the following:

I1007 22:52:02.452034       1 azuremachine_reconciler.go:216] Selecting first available AZ as no availability zone was set or user-provided availability zone is not supported for VM size Standard_B2ms in location southcentralus
I1007 22:52:02.452089       1 azuremachine_reconciler.go:220] Selected availability zone 2 for cluster-1007-b-controlplane-0
E1007 22:52:03.042859       1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create AzureMachine VM: failed to create vm cluster-1007-b-controlplane-0 : failed to create or get machine: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code=\"ResourceTypeNotSupportAvailabilityZones\" Message=\"The resource type 'virtualMachines' does not support availability zones at location 'southcentralus' and api-version '2019-03-01'.\""  "controller"="azuremachine" "request"={"Namespace":"default","Name":"cluster-1007-b-controlplane-0"}

What did you expect or want to happen?

We should be able to successfully create virtual machines with or without AZs.
It seems whatever logic on the Azure side should correctly validate a virtual machine template and allow creation in the specified AZ.

How can we reproduce it?

To dig a little deeper, I wanted to see if the VM type wasn't actually supported in southcentralus:

az vm list-skus -l southcentralus --zone true --size Standard_B2ms -o table
ResourceType     Locations       Name           Zones    Capabilities                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Restrictions
---------------  --------------  -------------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  --------------
virtualMachines  southcentralus  Standard_B2ms  1,2      ['MaxResourceVolumeMB=16384', 'OSVhdSizeMB=1047552', 'vCPUs=2', 'HyperVGenerations=V1,V2', 'MemoryGB=8', 'MaxDataDiskCount=4', 'LowPriorityCapable=False', 'PremiumIO=True', 'vCPUsAvailable=2', 'vCPUsPerCore=1', 'CombinedTempDiskAndCachedIOPS=2400', 'CombinedTempDiskAndCachedReadBytesPerSecond=23592960', 'CombinedTempDiskAndCachedWriteBytesPerSecond=23592960', 'UncachedDiskIOPS=1920', 'UncachedDiskBytesPerSecond=23592960', 'EphemeralOSDiskSupported=False', 'AcceleratedNetworkingEnabled=False', 'RdmaEnabled=False', 'MaxNetworkInterfaces=3']  None

So it seems both AZ 1 and 2 are available in southcentralus.

Now with a location I normally test in, eastus:

$ az vm list-skus -l eastus --zone true --size Standard_B2ms -o table
ResourceType     Locations    Name           Zones    Capabilities                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Restrictions
---------------  -----------  -------------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  --------------
virtualMachines  eastus       Standard_B2ms  1,2,3    ['MaxResourceVolumeMB=16384', 'OSVhdSizeMB=1047552', 'vCPUs=2', 'HyperVGenerations=V1,V2', 'MemoryGB=8', 'MaxDataDiskCount=4', 'LowPriorityCapable=False', 'PremiumIO=True', 'vCPUsAvailable=2', 'vCPUsPerCore=1', 'CombinedTempDiskAndCachedIOPS=2400', 'CombinedTempDiskAndCachedReadBytesPerSecond=23592960', 'CombinedTempDiskAndCachedWriteBytesPerSecond=23592960', 'UncachedDiskIOPS=1920', 'UncachedDiskBytesPerSecond=23592960', 'EphemeralOSDiskSupported=False']  None

Again, multiple zones are available for selection.

Now if I try this from the Azure portal, I get the following:

{"code":"InvalidTemplateDeployment","message":"The template deployment failed with error: 'The resource with id: '/subscriptions/<subscription-id>/resourceGroups/capz-augustus/providers/Microsoft.Compute/virtualMachines/DFSADFASDF' failed validation with message: 'The resource type 'virtualMachines' does not support availability zones at location 'southcentralus' and api-version '2019-08-01'.'.'."}

If I try to deploy a machine from the portal in southcentralus without AZs, template validation is successful:

I've also validated within Cluster API Azure that if set the machine templates to disable AZs, I'm able to successfully create a VM.

Anything we should know about your environment.

This doesn't appear to be a environment-specific issue. It's presenting across multiple API versions and deployment methods.

The bug was initially discovered by @cecilerobertmichon and I believe we're also using different subscriptions.

cc: @juan-lee @devigned @ritazh @craiglpeters

[devigned]

I just tried to build a VM in southcentralus and UI popped a link to docs for regions and services with zones support. southcentralus wasn't listed amongst the regions. I'm thinking the API is sending back lies.

@jhendrixMSFT, I think this is a compute team issue.

@justaugustus, great detail in the issue. Thank you for opening this. I'm sure it's confused many folks.

[ArcturusZhang]

Actually similar issue here if I did not get your point wrong.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @mjconnection, @Drewm3

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @mjconnection, @Drewm3

[Drewm3]

I will check with the team to determine why the AZs are showing up in the SKUs API even though AZs are not currently enabled in South Central US.

[justaugustus]

@Drewm3 -- sounds good! Just as a note, it could very well be multiple locations that are returning incorrect results. I just stopped w/ southcentralus once I realized it was a Compute API thing and not a downstream Cluster API Azure thing.

[Drewm3]

After some additional research I have found out that for a VM to be available for a zonal deployment it must be available in the VM SKUs API and the resource must have a zone mapping in the Get Providers API (https://docs.microsoft.com/en-us/rest/api/resources/providers/get).

In the case of SouthCentralUS VMs are not available in any zones which is why you are seeing the failure. The behavior that you are seeing in SouthCentralUS is the expected behavior when AZs are being rolled out for a region, but they are not yet available publicly.

My recommendation is to leverage the Provider API to determine if AZs can be used instead of falling back to hard-coding regions which support AZs.

Please let me know if there are any other questions. Otherwise I will resolve this issue.

Note, here is example output from the Get Providers API:

{
  "id": "/subscriptions/<snip>/providers/Microsoft.Compute",
  "namespace": "Microsoft.Compute",
   . . .
    {
      "resourceType": "virtualMachines",
      "locations": [
        "East US",
         <snip>
        "South Central US",
        <snip>
      ],
      <snip>
      "zoneMappings": [
        {
          <snip>
          "location": "East US",
          "zones": [
            "1",
            "3",
            "2"
          ]
        },
        <snip>
        {
          "location": "South Central US",
          "zones": []
        }
      ],
      "capabilities": "CrossResourceGroupResourceMove, CrossSubscriptionResourceMove, SystemAssignedResourceIdentity, SupportsTags, SupportsLocation"
    },

[justaugustus]

@Drewm3 -- I've been staring at this for a bit and I'm not seeing zoneMappings exposed in the resources API (for Azure SDK for Go):

https://github.com/Azure/azure-sdk-for-go/blob/8c372e6e1243e0427dffb2d750099ee3ed161a44/services/resources/mgmt/2019-05-01/resources/models.go#L1775-L1789

Here's some sample code:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2019-05-01/resources"
    "github.com/Azure/go-autorest/autorest"
    "github.com/Azure/go-autorest/autorest/azure/auth"
    "github.com/pkg/errors"
)

//var _ azure.Service = (*Service)(nil)

// Service provides operations on Providers
type Service struct {
    Client resources.ProvidersClient
    //Scope  *scope.ClusterScope
}

func main() {
    authorizer, err := auth.NewAuthorizerFromEnvironment()
    if err != nil {
        errors.Wrapf(err, "failed to create azure session")
    }

    subscriptionID := os.Getenv("AZURE_SUBSCRIPTION_ID")
    if subscriptionID == "" {
        fmt.Errorf("error creating azure services. Environment variable AZURE_SUBSCRIPTION_ID is not set")
    }

    client := getProvidersClient(subscriptionID, authorizer)

    provider, err := client.Get(context.Background(), "Microsoft.Compute", "")
    if err != nil {
        errors.Wrap(err, "provider error")
    }

    if provider.ResourceTypes != nil {
        res := *provider.ResourceTypes

        var vm resources.ProviderResourceType
        for _, rtype := range res {
            if *rtype.ResourceType == "virtualMachines" {
                vm = rtype
                break
            }
        }

        fmt.Printf("%+v\n", *vm.ResourceType)
        fmt.Printf("%+v\n", *vm.Locations)
        fmt.Printf("%+v\n", *vm.APIVersions)
        fmt.Printf("%+v\n", *vm.Capabilities)
        fmt.Printf("%+v\n", vm.Properties)

        if vm.Aliases != nil {
            aliases := *vm.Aliases
            for _, alias := range aliases {
                fmt.Printf("%+v\n", *alias.Name)
            }
        }
    }
}

// getProvidersClient creates a new Providers client from subscriptionid.
func getProvidersClient(subscriptionID string, authorizer autorest.Authorizer) resources.ProvidersClient {
    providersClient := resources.NewProvidersClient(subscriptionID)
    providersClient.Authorizer = authorizer
    providersClient.AddToUserAgent("cluster-api-azure-services")

    return providersClient
}

...and the output:

[East US East US 2 West US Central US North Central US South Central US North Europe West Europe East Asia Southeast Asia Japan East Japan West Australia East Australia Southeast Australia Central Brazil South South India Central India West India Canada Central Canada East West US 2 West Central US UK South UK West Korea Central Korea South France Central South Africa North UAE North]
[2019-07-01 2019-03-01 2018-10-01 2018-06-01 2018-04-01 2017-12-01 2017-03-30 2016-08-30 2016-04-30-preview 2016-03-30 2015-06-15 2015-05-01-preview]
CrossResourceGroupResourceMove, CrossSubscriptionResourceMove, SystemAssignedResourceIdentity, SupportsTags, SupportsLocation
map[]

Any insights?

[devigned]

Looks like the zoneMappings keys is not available in the latest specification, thus is not generated in each of the languages: https://github.com/Azure/azure-rest-api-specs/blob/a5edf3b757d2c06091a5389b053a8cfa562525a0/specification/resources/resource-manager/Microsoft.Resources/stable/2019-08-01/resources.json#L4246-L4285

@Drewm3, I can see it clear enough via the REST response from Providers GET. It's just not defined in the spec...

[Drewm3]

Thanks for the update. I will work with the owners of this API to ensure zoneMappings are added to the spec so they show up in the SDKs.

[justaugustus]

Thanks for working on this, Drew!

[Drewm3]

This issue has been resolved at the resource manager layer by suppressing zonal details in regions which are not ready for zones yet. For example South Central US is no longer showing any sizes available for deployment to Availability Zones.

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.