Azure-rest-api-specs: Can't Update VMSS Shared Image Gallery Version

[dbilleci-lightstream]

Hello, here is an issue I posted in Terraform, I was told post here.

---

Using the current API version, you are allowed to launch a VMSS using a gallery image definition using either a specified version, or without:

imageReference.id = /galleries/StageGallery/images/Api
imageReference.id = /galleries/StageGallery/images/Api/versions/1.2.3

When not specifying the version, the image that gets used appears to be the "latest", which I assume is defined as the highest version number that is not "failed".

I've also verified that if you do not specify a version in imageReference.id (as example above), and you then push a newer version to the gallery that the next machine to launch on the VMSS will now launch using this newer image.

After, whether you specify a version as part of the ID or not, you are not able to ever change it:

````
#command
az vmss update --resource-group environment-stage-westus-rg --name engine-stage-wus-0 --set virtualMachineProfile.storageProfile.imageReference.id=/subscriptions/11111111-1111-1111-111-c9687acfa843/resourceGroups/SharedImageGallery-Stage-Eus2-Rg/providers/Microsoft.Compute/galleries/StageEus2Gallery/images/Engine/versions/18110921.43.58

#response
Updating compute gallery image reference for Virtual machine scale sets is not allowed.

````

What we would like to be able to do is to specify the version used for a VMSS, and then be able to update that version. After updating the version, the VMSS should show that the model is out of date for instances not running on this new version. This is similar to the workflow we currently have in place when using non-gallery managed images. We specify the managed image ID, and then we update the imageReference.id which marks the old VMSS instances as "out of date".

When using the Shared Image Gallery, you are not able to update imageReference.id or imageReference.version, both are denied by the API.

I sent in a ticket to Microsoft and was told to use the "platform image method" by updating the imageReference.version for changing the Shared Image Gallery version, but that does not work either:

````

The 'platform method' does not seem to work for Shared Image Gallery Images

#command    
root@742d13ead3ac:# az vmss update --resource-group environment-stage-westus-rg --name engine-stage-wus-0 --set virtualMachineProfile.storageProfile.imageReference.version=18110902.21.51

#response
The Image type for a Virtual Machine Scale Set may not be changed.

````

It seems to me that we should be treating the Shared Image Gallery as a de-facto private Marketplace, and using the parameters intended for the Marketplace such as:
Publisher, Offer, SKU, Version. I would imagine that we could use a new variable such as PrivateGalleryID which we can provide the ID of the Gallery. Since Gallery requires you to set the Publisher, Offer and SKU to create the image definition, that seems to make more sense to me than trying to stuff in the complete gallery scope into the imageReference.id.

In short the question is:

• Please explain the proper way to launch a VMSS off of a specific version, and allow the VMSS to be updated to a newer version at a later time?

If the answer above is "simply push a new version to the Gallery, and do not provide a Version on the imageReference.id" then a follow up question is:

• How does the VMSS "model" show as out-of-date for VMSS instances not running on the latest version?

[dbilleci-lightstream]

Any update here on what's going to happen here? Acknowledged? Will be Addressed? ETA?

[ivasilyev-servicetitan-com]

I faced the same problem. And would also like to have ability to update shared image version for VMSS created based on Shared Image.

On more issue I faced which is related as well:

• for VMSS created based on non-gallery managed image you can't update the image pointing Shared Image. You got

Id /subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/images-rg/providers/Microsoft.Compute/galleries/MyGallery/images/Dev2016/versions/1.0.0 is not a valid resource reference

. However you can successfully create VMSS using exactly the same imageReferenceId.

[slikk66]

As a workaround, I've written an AWS lambda function running on a cron event that queries the gallery "latest" version then updates a tag on the vmss. This causes the vmss instances that don't have that tag to show out of date model. It's not ideal because really you can only roll forward and not backwards, but until Azure fixes this up not sure what else can be done. I used lambda because I couldn't figure out AZ functions for the life of me.

[ivasilyev-servicetitan-com]

Thanks for suggestion. I saw similar workaround mentioned here. Will use the same until this issue is resolved

[slikk66]

FYI, I have a ticket in with Microsoft.. so far they've admitted it's likely a "product limitation" at this time but waiting on official word:

```Hello Sir,

We spoke as a team and we will send an internal ticket to have this documented and get an answer, but probably we will get the same answer.

This can take a while as well as it’s on preview, so I will keep you updated. Any question or comment just let me know.

Regards,

[slikk66]

Here's the official long and not very helpful response from MS:

````
Hi Dan,

It seems like an issue engineering is working already. They have a work item being tracked to fix. Please see if you can use the proposed workaround to allow upgrade of vmss model to another specific version.

You can create using “latest” version (say it is 1.0.0.0 at this time). All vm instances in vmss will get that version of image.

When a new version (say 2.0.0.0) is published to image, then you can re-image the instances in phases, (reimage instance 0, then 2,3,4, then all) to safely upgrade the vmss. This would be semi-manual rolling upgrade. When you reimage, the new instances will get new image version (2.0.0.0).

To add,

If you create a VMSS with ‘latest’ version, i.e. passing in just the image id instead of a specific version id in the vmss template, then you will get vms with new versions when:

1. You scale out
2. You reimage existing ones and new versions for the image are available.
3. Engineering are also making auto-upgrade feature, which will also allow upgrading vmss automatically.

We are also tracking a work-item to allow changing the image reference and allow upgrading one one particular version to another particular version. It will be shipped in next semester.

In future you can also opt in for auto os upgrade as well.

Automatic OS upgrades will be applied on a given VMSS in batches (honoring FD and AZ boundaries) and stop upgrade if VMs go unhealthy. Also, scale sets will be upgraded with SDP and so in multiple phases globally. So the immediate need to update gallery image reference won’t be solved with auto OS upgrade alternative.

This is the workaround they suggested for the moment in the meantime they work on this. Let me know if you have further questions on this,

Regards,
````

I prefer my workaround to this answer. They essentially say "push an update to gallery, then go through your machines and re-image/re-roll them" but which ones? Obviously in some cases that's straight forward enough, but I deal with multiple VMSS with >= 80 machines per VMSS... too large to use this method. We use the out-of-date model system to have machines auto terminate when they're old to redeploy to new model. Also, we can't just willy-nilly delete machines any time just because a new image was pushed.. they're busy and can only be safely deployed at certain times. Looks like I'll be running a Lambda until "next semester".

[ivasilyev-servicetitan-com]

@slikk66 I'd like to ask you about the workaround you're using.
To make VMSS instances "out-of-date" I tried to add new tag to VMSS.
I didn't manage to add tag using command like
az vmss update --resource-group dev-rg --name DEV_VMSS --set tags={2018120614.30.51}
I always got an error
az vmss update: 'utputformat' is not a valid value for '--output'. See 'az vmss update --help'.

I successfully added tag using command like
Set-AzureRmResource -Tag @{ ImageVersion="2018120614.30.51" } -ResourceId $r.ResourceId -Force
but this didn't make my VMSS instances to have LatestModel=No. I also tried to add tag using portal.azure.com UI - the same result - LatestModel is still Yes.

However, when I ran command to set "latest-style" SharedImage to VMSS using command like this

az vmss update --resource-group dev-rg --name DEV_VMSS --set virtualMachineProfile.storageProfil
e.imageReference.id=/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/images-rg/providers/Microsoft.Com
pute/galleries/MyGallery/images/Dev2016

instances of VMSS got "LatestModel=No" - that's what I needed.

Can you please let me know how you're updating tags on your VMSS and don't you have similar issue?

[ivasilyev-servicetitan-com]

And one more note: if you didn't produce new Image version for your SharedImage and just update imageReference of the VMSS - the instance is still LatestModel=Yes.
Only if you really have different "latest" version in you SharedImage then VMSS instance will show LatestModel=No (when you update imageReference).
So internally VMSS "almost" working fine, it controls the real "version" of image it's using even with "latest-style" imageReference definition.

[slikk66]

Hey ivasilyev, I'm working on my tag updater now too, and it looks like you're right, updating the tag only doesn't change the "model" - I'm not sure why I thought it did, I must have ran a bad test.
So you're saying if you try to change the image.id to a new version, even though it doesn't work (the image doesn't change), it still updates the model? I'm going to run some more tests myself.

[slikk66]

Hey again @ivasilyev-servicetitan-com so I tried to update the imageReference.id like you said, but I'm using the Python SDK.. If the VMSS has been set with the ID such as galleries/MyGallery/images/MyImage and then I run the command:

self.az_mgmt_client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdate( tags=tags, virtual_machine_profile=self.az_mgmt_client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdateVMProfile( storage_profile=self.az_mgmt_client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdateStorageProfile( image_reference=self.az_mgmt_client.virtual_machine_scale_sets.models.ImageReference( id="/subscriptions/{arm_subscription_id}/resourceGroups/{gallery_resource_group_name}/providers/Microsoft.Compute/galleries/{gallery_name}/images/{preset}".format( arm_subscription_id=self._get_ssm_param('arm_subscription_id'), gallery_resource_group_name=gallery_info['gallery_resource_group_name'], gallery_name=gallery_info['gallery_name'], preset=update['preset'] ) ) ) ) )

which should just set it again to the same thing, it's not updating my model :(

Any help you have would be appreciated.. I'm now searching for other things I could maybe change that would allow the set to update the model, such as maybe "admin_password" but that doesn't seem allowed by the objects: https://docs.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2018_06_01.models.virtualmachinescalesetupdateosprofile?view=azure-python

[slikk66]

Think I found a good workaround - I'm updating the custom_data with basically the same data, and adding a differing "touch" command at the end. When I update this, it sets the model as out of date (good) and keeps the bootstrap system in place (good) but changes it just enough for the desired effect.

I set up a Lambda at AWS that runs on a Cloudwatch event (I couldn't figure out AZ functions) that stores SPN credentials in KMS and runs every 20 min or so. It checks the tag "Image" on the vmss and also queries the Shared Image Gallery. If the tag is not the same as the latest version of the gallery, then it runs some python to do the update of both the custom_data and the tag. I had to hard code in the current custom_data because the CLI/SDK apparently has a bug where it won't send you the current custom_data, even though it says it should.. it just returns None/Null.. awesome work Azure 👍

Here's a link to that bug: https://github.com/Azure/azure-cli/issues/5866 So, because of that I had to write out our bootstrap commands into this system, but luckily it's just one command.. so here's basically the good stuff:

````

create new userdata

new_user_data = dedent("""
#cloud-config
#vim: tabstop=2 shiftwidth=2 expandtab syntax=yaml
---
users:
- default
package_upgrade: false
runcmd:
- "cd /opt/bootstrap && ./cloud-init.sh"
- "touch /tmp/{version}-{timestamp}"
""").format(
version=update['version'],
timestamp=datetime.datetime.utcnow().isoformat()
)

encode string to bytes, then base64 encode that

new_user_data = base64.b64encode(new_user_data.encode())

now byte decode the b64 string to get raw string b64 in 'dGVzdA==' format

new_user_data = new_user_data.decode()

client.virtual_machine_scale_sets.update(
resource_group_name,
resource_name,
client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdate(
tags=tags,
virtual_machine_profile=client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdateVMProfile(
os_profile=client.virtual_machine_scale_sets.models.VirtualMachineScaleSetUpdateOSProfile(
custom_data=new_user_data
)
)
)
)
````

hopefully this helps you out!

[Drewm3]

Cleaning up old issues from 2016 - 2018. Please reopen this issue if it is still a concern.

[Drewm3]

Let me reopen this one and fix the fact that this is a feature request. @dbilleci-lightstream , I is your primary question still:

Please explain the proper way to launch a VMSS off of a specific version, and allow the VMSS to be updated to a newer version at a later time?

And for this are you wanting to understand how to do this without updating existing instances to the new image or do you want all instances to be updated to the new image?

[ivasilyev-servicetitan-com]

The problem is:
Once you created VMSS with imageReference.Id = /galleries/<name>/images/<name>/versions/<version>, you can't update VMSS to use another version of this SharedImage.

On other hand, if you created VMSS using "latest" (imageReference.Id = /galleries/<name>/images/<name>), and then you pushed new "latest" to your SharedImage - you don't see "the model is not latest" for existing VMSS instances

[ivasilyev-servicetitan-com]

And actually I just tried to create VMSS with specific SharedImage version and it failed. When this ticket was created looks like it was working.

Error: compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="Id /subscriptions/.../resourceGroups/images-rg/providers/Microsoft.Compute/galleries/.../images/Dev2016/2019082907.49.11 is not a valid resource reference."

  on ..\..\..\modules\compute\create-vmss\create-vmss.tf line 97, in resource "azurerm_virtual_machine_scale_set" "vmss":
  97: resource "azurerm_virtual_machine_scale_set" "vmss" {

azurerm provider version is 1.29.0

UPDATE: the issue with this it my fault. I missed versions in the ImageId, I had to specify it as /subscriptions/.../resourceGroups/images-rg/providers/Microsoft.Compute/galleries/.../images/Dev2016/versions/2019082907.49.11

[Drewm3]

I just validated that the example using PowerShell from this location works as expected except for one change: https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/shared-images-powershell

In this example the property "-Id" in the cmdlet Set-AzVmssStorageProfile needs to be changed to "-ImageReferenceId". With this change I was able to create a VM from a shared image gallery image.

I was also able to update the ImageReferenceId in the VMSS configuration and it automatically updated the instances.

The one thing that is not available currently is the ability to specify "latest" in the image version, and then use automatic OS upgrade to update instances when a new version is available in the Shared Image Gallery. However, this is in our backlog to complete.

[ivasilyev-servicetitan-com]

I can confirm that I was successfully able to:

• create VMSS with "latest" SharedImage and update it to "specific"
• create VMSS with "specific" SharedImage and update it to "latest"
• create VMSS with "specific" SharedImage and update it to "specific"

And "Is Model Latest" is set to "No" on these changes. Really cool!

As for

The one thing that is not available currently is the ability to specify "latest" in the image version, and then use automatic OS upgrade to update instances when a new version is available

I'm using "upgradePolicy": "mode": "Manual" and it works fine for VMSS with "latest" approach, when I publish new Shared Image version and do manually Upgrade+Reimage - it uses the new version. For sure, when Azure will detect this modification automatically - that will be more convenient.

Thanks @Drewm3 for the update!

[Drewm3]

Thanks for the confirmation @ivasilyev-servicetitan-com. With this confirmation, I believe this issue can be closed. Please let me know if you disagree.