Azure-cli: VM Encryption does not support Ubuntu 16.04.0-LTS

[orezahc]

I am trying to encrypt a running Machine by azure-cli. The procedure failed with error message 'Encryption is not supported for current VM....', but the OS image of the running machine is Ubuntu 16.04.0-LTS provided by Azure. The VM size is Standard_D2. Looks like it validates the OS image with an incorrect SKU '16.04'.

    supported = [
        {
            'offer': 'RHEL',
            'publisher': 'RedHat',
            'sku': '7.2'
        },
        {
            'offer': 'RHEL',
            'publisher': 'RedHat',
            'sku': '7.3'
        },
        {
            'offer': 'CentOS',
            'publisher': 'OpenLogic',
            'sku': '7.2n'
        },
        {
            'offer': 'Ubuntu',
            'publisher': 'Canonical',
            'sku': '14.04'
        },
        {
            'offer': 'Ubuntu',
            'publisher': 'Canonical',
            'sku': '16.04'
        }]

[yugangw-msft]

@orezahc It is true only a few distros are supported. The error below is caused by a bug that sku matching at client side should not be that strict, particular for ubuntu. I will fix it today. Also I will change to a warning, rather an error that is to block users.

[orezahc]

@yugangw-msft Thanks for the immediate reply. One more question, because the VM encryption feature is important to us now, when would it be available? and can you provide a document on how to encrypt a VM in Azure CLI2.0.

[yugangw-msft]

The docker image containing the fix is available. The public release will be out in about 2 weeks
Sample instructions here:

1.  az ad sp create-for-rbac –skip-assignment (it will emit out the service principal details)
2.  az group create –n myRG –l eastus
3.  az keyvault create –g myRG –n myRG-keyvault -l eastus
4.  az keyvault key create -n myRG-key -p software --vault-name myRG-keyvault
5.  az keyvault set-policy -n myRG-keyvault  --spn 70a29bd0-d684-49d7-a8c4-999999999999 --key-permissions all --secret-permissions all
6.  az keyvault update -n myRG-keyvault --set properties.enabledForDiskEncryption=True
7.  az vm create –g myRG-junk –n myRG-junk-1 –image OpenLogic:CentOS:7.2n:7.2.20160629   (only version 7.2n supports encryption) 
8.  az vm disk attach –disk d1 –g myRG-junk –n myRG-junk-1 –new –size-gb 2
9.  (ssh into the VM and mount the data disk)
10. az vm encryption enable -g myRG-junk -n myRG-junk-1 --aad-client-id 70a29bd0-d684-49d7-a8c4-999999999999 --aad-client-secret secret-password --disk-encryption-keyvault /subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/myRG/providers/Microsoft.KeyVault/vaults/myRG-keyvault --key-encryption-key myRG-key --volume-type all
11. (#10 will emit out a warning to tell you to wait for the reboot message like “OS disk successfully encrypted, please reboot the VM”)
12. (Wait for a hour) and run ‘az vm encryption show -g myRG-junk -n myRG-junk-1’ to ensure the extension is waiting for VM reboot
13. az vm restart  -g myRG-junk -n myRG-junk-1
14. (Wait about 10 minutes and run ‘az vm encryption show -g myRG-junk -n myRG-junk-1’ again and it should show both disks are encrypted)
15. (Now you can ssh into it)

[orezahc]

Thanks. How long the encryption will take for Ubuntu 16.04.0-LTS? It has already taken around 5 hours.

[orezahc]

I got another error while encrypting VMs. Here is the information from
az vm encryption show -g group -n vm

{
  "dataDisk": "Unknown",
  "osDisk": "Unknown",
  "osDiskEncryptionSettings": {
    "diskEncryptionKey": {
      "secretUrl": "https://group-kv.vault.azure.net/secrets/xxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx",
      "sourceVault": {
        "id": "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/rg-xxxxxxxxxxxxxxxxxxx/providers/Microsoft.KeyVault/vaults/group-kv"
      }
    },
    "enabled": true,
    "keyEncryptionKey": {
      "keyUrl": "https://groupvm-kv.vault.azure.net/keys/DiskEncryption/xxxxxxxxxxxxxxxxxxx",
      "sourceVault": {
        "id": "/subscriptions/xxxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxxxxxxxx/providers/Microsoft.KeyVault/vaults/group-kv"
      }
    }
  },
  "osType": "Linux",
  "progressMessage": "Failed to encrypt OS volume with error: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n, stack trace: Traceback (most recent call last):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 166, in start_encryption\n    self.enter_unmount_oldroot()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 222, in trigger\n    return self.machine.process(f)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 526, in process\n    return trigger()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 247, in _trigger\n    if t.execute(event):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 148, in execute\n    self._change_state(event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 159, in _change_state\n    event_data.machine.get_state(self.dest).enter(event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 48, in enter\n    event_data.machine.callback(oe, event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 518, in callback\n    func(*event_data.args, **event_data.kwargs)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 114, in on_enter_state\n    super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py\", line 65, in on_enter_state\n    self.state_objs[self.state].enter()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py\", line 134, in enter\n    self.command_executor.Execute('umount /oldroot', True)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py\", line 70, in Execute\n    raise Exception(msg)\nException: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n\n, stack trace: Traceback (most recent call last):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/handle.py\", line 1522, in daemon_encrypt\n    os_encryption.start_encryption()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 184, in start_encryption\n    raise Exception(message)\nException: Attempt #1 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n, stack trace: Traceback (most recent call last):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 166, in start_encryption\n    self.enter_unmount_oldroot()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 222, in trigger\n    return self.machine.process(f)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 526, in process\n    return trigger()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 247, in _trigger\n    if t.execute(event):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 148, in execute\n    self._change_state(event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 159, in _change_state\n    event_data.machine.get_state(self.dest).enter(event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 48, in enter\n    event_data.machine.callback(oe, event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/transitions/transitions/core.py\", line 518, in callback\n    func(*event_data.args, **event_data.kwargs)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 114, in on_enter_state\n    super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/OSEncryptionStateMachine.py\", line 65, in on_enter_state\n    self.state_objs[self.state].enter()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py\", line 134, in enter\n    self.command_executor.Execute('umount /oldroot', True)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999283/main/CommandExecutor.py\", line 70, in Execute\n    raise Exception(msg)\nException: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n\n\n, machine state: unmount_oldroot"
}

[yugangw-msft]

//cc: @krkhan, please advise

[andschwa]

Oi, is this really the case? This is happening to me on Ubuntu 16.04 too, and I've tracked down several other issues that look similar.

[kHRISl33t]

@orezahc did you managed to resolve the issue? I'm experiencing the same when I'm trying to encrypt my VM's in AKS
AKS#629

[andyzhangx]

I got this exact same issue here:
Below are the VM kernel info:

Ubuntu 16.04.5 LTS
$ uname -a
Linux aks-nodepool1-20449952-1 4.15.0-1021-azure #21~16.04.1-Ubuntu SMP Fri Aug 10 12:36:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  "progressMessage": "Attempt #7 to unmount /oldroot failed with error: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n, stack trace: Traceback (most recent call last):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 170, in start_encryption\n    self.retry_unmount_oldroot()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/transitions/transitions/core.py\", line 222, in trigger\n    return self.machine.process(f)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/transitions/transitions/core.py\", line 526, in process\n    return trigger()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/transitions/transitions/core.py\", line 247, in _trigger\n    if t.execute(event):\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/transitions/transitions/core.py\", line 145, in execute\n    machine.callback(func, event_data)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/transitions/transitions/core.py\", line 518, in callback\n    func(*event_data.args, **event_data.kwargs)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/main/oscrypto/ubuntu_1604/Ubuntu1604EncryptionStateMachine.py\", line 114, in on_enter_state\n    super(Ubuntu1604EncryptionStateMachine, self).on_enter_state()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/main/oscrypto/OSEncryptionStateMachine.py\", line 65, in on_enter_state\n    self.state_objs[self.state].enter()\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/main/oscrypto/ubuntu_1604/encryptstates/UnmountOldrootState.py\", line 134, in enter\n    self.command_executor.Execute('umount /oldroot', True)\n  File \"/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-0.1.0.999322/main/CommandExecutor.py\", line 73, in Execute\n    raise Exception(msg)\nException: Command umount /oldroot failed with return code 32\nstdout:\n\nstderr:\numount: /oldroot: target is busy\n        (In some cases useful info about processes that\n         use the device is found by lsof(8) or fuser(1).)\n\n"

Can we reopen this issue?
/open

[kHRISl33t]

@andyzhangx

I got this message from Azure support: Regret to say that currently Encryption for AKS agent node is not possible. I did submit a form with my request for this, but I dont know when they are going to release this feature, they don't replied back anything.

I only managed to encrypt data disks attached to the agent nodes.

[andyzhangx]

@kHRISl33t I am asking internal disk encryption team to take a look at this issue, it's most likely due to the workload already running inside AKS node, it should be not specific to AKS, other Ubuntu VM may also hit this issue.

[yugangw-msft]

@vermashi can you please take a look? We are talking about a vm as an AKS node

[yugangw-msft]

Per internal mailing exchange, @andyzhangx, your issue should have been resolved.
Closing