Azure-cli: Failing to break lease for PREMIUM_LRS storage or detach os disk

[dtissen]

I have several virtual machines which uses a bunch of blobs. these blobs are divided into two storages (PREMIUM_LRS and STANDARD_LRS). If i try to restore a blob, i shutdown the virutel machines which is using the blob and then break the lease (instead of deleting the VM). this works fine for STANDARD_LRS stored blobs. For PREMIUM_LRS Blobs i allways get the Message: "This blob is being used by the system." - but i did stop and deallocated the using virtuel machine?!

DEBUG LOG:

File logging enabled - Writing logs to '/root/.azure/logs'.
Command arguments ['storage', 'blob', 'lease', 'break', '-b', 'test02-db01-02.vhd', '-c', 'vhds', '--connection-string', 'DefaultEndpointsProtocol=https;EndpointSuffix=core.cloudapi.de;AccountName=[REPLACED];AccountKey=[REPLACED]']
Current cloud config:
{'endpoints': {'active_directory': 'https://login.microsoftonline.de',
               'active_directory_data_lake_resource_id': None,
               'active_directory_graph_resource_id': 'https://graph.cloudapi.de/',
               'active_directory_resource_id': 'https://management.core.cloudapi.de/',
               'batch_resource_id': 'https://batch.cloudapi.de/',
               'gallery': 'https://gallery.cloudapi.de/',
               'management': 'https://management.core.cloudapi.de/',
               'resource_manager': 'https://management.microsoftazure.de',
               'sql_management': 'https://management.core.cloudapi.de:8443/',
               'vm_image_alias_doc': 'https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/arm-compute/quickstart-templates/aliases.json'},
 'is_active': True,
 'name': 'AzureGermanCloud',
 'profile': 'latest',
 'suffixes': {'azure_datalake_analytics_catalog_and_job_endpoint': None,
              'azure_datalake_store_file_system_endpoint': None,
              'keyvault_dns': '.vault.microsoftazure.de',
              'sql_server_hostname': '.database.cloudapi.de',
              'storage_endpoint': 'core.cloudapi.de'}}
Registered application event handler 'CommandTableParams.Loaded' at <function add_id_parameters at 0x7f41b36039d8>
Registered application event handler 'CommandTable.Loaded' at <function add_id_parameters at 0x7f41b36039d8>
Successfully loaded command table from module 'storage'.
Extensions directory: '/root/.azure/cliextensions'
Application event 'CommandTable.Loaded' with event data {'command_table': {'storage blob lease break': <azure.cli.core.commands.CliCommand object at 0x7f41b35c5ef0>}}
Application event 'CommandParser.Loaded' with event data {'parser': AzCliCommandParser(prog='az', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)}
Application event 'CommandTableParams.Loaded' with event data {'command_table': {'storage blob lease break': <azure.cli.core.commands.CliCommand object at 0x7f41b35c5ef0>}}
Application event 'CommandParser.Parsing' with event data {'argv': ['storage', 'blob', 'lease', 'break', '-b', 'test02-db01-02.vhd', '-c', 'vhds', '--connection-string', 'DefaultEndpointsProtocol=https;EndpointSuffix=core.cloudapi.de;AccountName=[REPLACED];AccountKey=[REPLACED]']}
Application event 'CommandParser.Parsed' with event data {'command': 'storage blob lease break', 'args': Namespace(_command_package='storage', _jmespath_query=None, _log_verbosity_debug=False, _log_verbosity_verbose=False, _output_format='json', _parser=AzCliCommandParser(prog='az storage blob lease break', usage=None, description='Breaks the lease, if the blob has an active lease. Once a lease is broken, it cannot be renewed. Any authorized request can break the lease; the request is not required to specify a matching lease ID. When a lease is  [...]
Getting data service client service_type=BlockBlobService
azure.multiapi.storage.v2017_04_17.common._auth : String_to_sign=PUT


x-ms-client-request-id:fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd
x-ms-date:Wed, 06 Dec 2017 06:37:55 GMT
x-ms-lease-action:break
x-ms-version:2017-04-17
/test02fast/vhds/test02-db01-02.vhd
comp:lease
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Outgoing request: Method=PUT, Path=/vhds/test02-db01-02.vhd, Query={'comp': 'lease', 'timeout': None}, Headers={'x-ms-lease-id': None, 'x-ms-lease-action': 'break', 'x-ms-lease-duration': None, 'x-ms-lease-break-period': None, 'x-ms-proposed-lease-id': None, 'If-Modified-Since': None, 'If-Unmodified-Since': None, 'If-Match': None, 'If-None-Match': None, 'x-ms-version': '2017-04-17', 'User-Agent': 'Azure-Storage/0.37.0-0.36.0 (Python CPython 3.6.1; Linux 4.4.0-43-Microsoft) AZURECLI/2.0.20', 'x-ms-client-request-id': 'fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd', 'x-ms-date': 'Wed, 06 Dec 2017 06:37:55 GMT', 'Authorization': 'SharedKey [REPLACED]'}.
urllib3.connectionpool : Starting new HTTPS connection (1): test02fast.blob.core.cloudapi.de
urllib3.connectionpool : https://test02fast.blob.core.cloudapi.de:443 "PUT /vhds/test02-db01-02.vhd?comp=lease HTTP/1.1" 409 219
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Receiving Response: Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP Status Code=409, Message=This blob is being used by the system., Headers={'x-ms-lease-id': None, 'x-ms-lease-action': 'break', 'x-ms-lease-duration': None, 'x-ms-lease-break-period': None, 'x-ms-proposed-lease-id': None, 'If-Modified-Since': None, 'If-Unmodified-Since': None, 'If-Match': None, 'If-None-Match': None, 'x-ms-version': '2017-04-17', 'User-Agent': 'Azure-Storage/0.37.0-0.36.0 (Python CPython 3.6.1; Linux 4.4.0-43-Microsoft) AZURECLI/2.0.20', 'x-ms-client-request-id': 'fd4b8d8e-da4f-11e7-a7b7-a434d9460bfd', 'x-ms-date': 'Wed, 06 Dec 2017 06:37:55 GMT', 'Authorization': 'SharedKey test02fast:[REPLACED]'}.
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP status code=409, Exception=This blob is being used by the system.<?xml version="1.0" encoding="utf-8"?><Error><Code>SystemInUse</Code><Message>This blob is being used by the system.RequestId:1f00da01-001c-0014-045c-6eb6ae000000Time:2017-12-06T06:37:56.3333175Z</Message></Error>.
azure.multiapi.storage.v2017_04_17.common.storageclient : Client-Request-ID=fdb89488-da4f-11e7-a7b7-a434d9460bfd Retry policy did not allow for a retry: Server-Timestamp=Wed, 06 Dec 2017 06:37:55 GMT, Server-Request-ID=1f00da01-001c-0014-045c-6eb6ae000000, HTTP status code=409, Exception=This blob is being used by the system.<?xml version="1.0" encoding="utf-8"?><Error><Code>SystemInUse</Code><Message>This blob is being used by the system.RequestId:1f00da01-001c-0014-045c-6eb6ae000000Time:2017-12-06T06:37:56.3333175Z</Message></Error>.

This blob is being used by the system.

Environment summary

Install Method (e.g. pip, interactive script, apt-get, Docker, MSI, edge build) / CLI version (az --version) / OS version / Shell Type (e.g. bash, cmd.exe, Bash on Windows)

[troydai]

@williexu please talk to the Storage regarding this issue. The issue applies to one type of the storage account not the other.

[erosen03]

Did this issue get resolved? I've encountered the same issue.

[williexu]

@seguler

[seguler]

@mirobers - is there a difference in behavior for premium vs standard ?

[mirobers]

This is expected from the storage side. Premium disk leases can't be broken as long as the disk is in use. Detaching the disk should remove the lease.

[erosen03]

Looks like breaking a lease on a premium VM page blob is not supported as long as the blob is attached to a VM,even if the VM is stopped and de-allocated. This is quite problematic since it means you have to completely delete and recreate the VM if you need to access its OS disk. Definitely not an optimal solution.

Any chance this restriction will be lifted in the future?

Reference:
https://docs.microsoft.com/en-us/rest/api/storageservices/lease-blob

[mirobers]

I believe the VM keeps the lease on the disk even when deallocated, but will release it once detached. Could you try detaching?

[dtissen]

Why is this issue closed? As said by @erosen03 this is an lack of usage! Please explain. Is there a general difference between Premium Storage? For Standard storage braking lease is allowed without detaching the blobs....

[mirobers]

@dtissen, the correct workflow for removing the lease is to detach the disk. The issue as stated is by design because breaking the lease is the wrong way to go about this. If there is a problem in the detach workflow, we should reopen (and rename) this issue and involve the right folks from the compute side to check further. Could you state the issue you are hitting when attempting to detach that requires the VM to be deleted?

[erosen03]

@mirobers , The reason detaching the disk doesn't work is because the only way to 'detach' an OS disk is to delete and recreate the VM. That's a really heavy-handed approach and caries a significant risk of configuration data loss.

[williexu]

reopening since there seems to be further discussion

[mirobers]

@williexu Please also update the title to "OS disk can't be detached without deleting the VM" and remove the storage tag. Thanks!

[williexu]

@erosen03 have you tried this?: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/detach-disk#detach-a-data-disk-using-the-portal

You should be able to detach the disk after stopping your vm

[erosen03]

Thanks @williexu and @mirobers. I don't want to get off topic, but two other common scenarios that also currently require recreating a VM are:

1. Moving a VM to a different subnet or vnet
2. Adding a VM to or removing a VM from an availability or scale set

As were're looking at the 'recreate VM' requirement regarding an OS disk, it would be really nice if we come up with a solution that would also help the other two scenarios listed above.

Thanks.

[erosen03]

@williexu, yes I have used that before, and it works fine or data disks. You can also attach / detach data disks using the Azure portal. However, that does not work for OS disks, and you can not attach / detach OS disks using the Azure portal either.

[williexu]

@erosen03 what scenario are you trying to solve by breaking a lease on a os disk that is attached to a vm without deleting the vm? Since vms are tied to an os disk, this does not seem very useful to me.

@mirobers why is there a different policy in place for page blobs in premium vs standard storage? I am able to break a lease on an os disk blob in standard storage without detaching the disk.

[erosen03]

@williexu I'm applying Azure Storage Service Encryption (https://docs.microsoft.com/en-us/azure/storage/common/storage-service-encryption) for existing data. This requires that you copy the disk block out and back, which requires that the lease is broken.

Q: I would like to encrypt the current data in an existing Resource Manager storage account?

A: With SSE enabled by default for all storage accounts - Classic and Resource Manager storage accounts. However, data that were already present will not be encrypted. To encrypt existing data, you can copy them to another name or another container and then remove the unencrypted versions.

[dtissen]

@erosen03 for your problem: Use az group export (with default values). this gives you the vm also settings. Delete the VM. Do whatever you want with the blobs and deploy the configuration back using the exported Template by using az group deployment create .... simpliest way to restore a VM or even a whole resource Group.

Using the Template you can "detach" the OS disk and use it again as OS disk. Thats my current approach to snapshot and restore a VM with blobs including also the OS Disk. I made therefor a groovy scripts which create and restore vms with one command.... Before this, I asked this question, because breaking a lease and restore the blob using snapshots is less a sledgehammer approach than deleting the vm....

From my side this issue is more or less because of my curiosity, since i have a solution for restoring the whole VM. As said by @williexu there is a difference behaviour for premium vs standard storage. Looks like a bug.

@erosen03 : Encryption: As said in your Q/A: you can copy the blob with another name, therefor you dont need to break lease. Copy somerwhere should always be possible. Of course, if you want to use the same names you have to get rid of the lease before.....

@williexu not to be able to detaching an OS Disk: Is this ab bug or feature? I thought it is a feature, also i dont know why to forbid....

[erosen03]

@dtissen unfortunately it looks like exporting resource group templates is limited to resource groups with fewer than 200 resources, so that won't work for most of my environments.

[dtissen]

-.- sorry for that, but good to know. Another approach could be: create the template on your own... not very complex. Export an existing one, modify it so it fits your requirement. Since a template can also use a parameter file you can make a template and just change the parameters....

[erosen03]

@dtissen that's pretty much what we're doing now - using PowerShell to capture the VM configuration, delete the VM, perform whatever operations we need to do on the OS disk, and then recreate the VM using the captured configuration.

The issue with this approach is that you risk losing some VM configuration information if the config data is not captured completely, so it's really not a great solution, just the best one available at the moment.

The ideal solution is to not need to delete the VM in the first place, hence the need to break the lease.

[dtissen]

@erosen03 Thats exactly the same i do! ;-) ans also the same reason.... yes deleting the vm is very ugly approach

[williexu]

@erosen03 after stopping a vm, you should be able to make a copy of your os disk blob.
You can then perform operations on that blob, and then do a az vm update --set storageProfile.osDisk.vhd.uri=<copy url> to set the vm to the disk copy. Hope this helps.

[erosen03]

@williexu, unfortunately my experience is that you can't perform file level operations on a blob like copying it or overwriting it while the blob is leased. Also, you can not set the OS disk on an existing VM. You can only do that on a new VM configuration.

[williexu]

@erosen03 Do the following steps (assuming your container name is vhds):

1. stop your vm
2. run az storage blob copy start --source-blob <source blob name> --source-container vhds -b <your desired copy name> -c vhds --account-name <storage account name>
3. get your copied blob url using az storage blob url -c vhds --account-name <> -n <blob copy name> or portal
4. run az vm update --set storageProfile.osDisk.vhd.uri=<copy url> -g <group name> -n <vm name>
5. az vm show should show you an updated os disk

I was able to run these steps with no problems.

[williexu]

Closing this as this is not a problem with the cli.
@erosen03 hopefully the steps I outlined above help with your scenario

[erosen03]

@williexu I tried the steps using PowerShell and Azure storage explorer, and they worked. Thanks.

Only thing I'm wondering is how that procedure will affect things when you're using blob snapshots. I.e., since we're getting a completely new blob, none of the snapshots from the previous blob will be available, unless you keep the old blob around and track somewhere that it contains the older snapshots for the OS disk. Regardless, that's manageable and a lot better than deleting and recreating the VM.

But it would really make things a lot easier if breaking blob leases worked in premium storage the way it works in standard storage :)

Thanks again.