Packer: MDI ID used for SIG publish missing subscriptionID when authenticating with managed identity

[pmozbert]

Overview of the Issue

I am using Azure ARM builder to create image and store in shared image gallery. I am using Azure VM managed identity for Packer to authenticate. However, this appears to be causing the statebag parameter for constants.ArmManagedImageSubscription to not be set.
In step_publish_to_shared_image_gallery.go
line 102...

var managedImageSubscription = stateBag.Get(constants.ArmManagedImageSubscription).(string)
var mdiID = fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/images/%s", managedImageSubscription, targetManagedImageResourceGroupName, targetManagedImageName)

s.say(fmt.Sprintf(" -> MDI ID used for SIG publish     : '%s'", mdiID))

produces this output

==> azure-arm:  -> MDI ID used for SIG publish     : '/subscriptions//resourceGroups/tempimagebuildrg/providers/Microsoft.Compute/images/packertempwin2019dc'

resulting in this error

==> azure-arm: ERROR: -> LinkedAuthorizationFailed : The client has permission to perform action 'Microsoft.Resources/subscriptions/tempimagebuildrg/Microsoft.Compute/packertempwin2019dc/read' on scope '/subscriptions/12ef276f-c342-456f-b802-909283fed03a/resourceGroups/prod.pce.sharedservices.imagegallery.eastus2/providers/Microsoft.Compute/galleries/prod.pce.sharedservices.imagegallery.eastus2.01/images/GF-Base-WindowsServer-2019-Datacenter/versions/2019.352.22', however the linked subscription 'resourceGroups' was not found. 

Reproduction Steps

This is the packer config file

{
    "variables": {
        "virtual_network_name": "{{env `VIRTUAL_NETWORK_NAME`}}",
        "virtual_network_subnet_name": "{{env `VIRTUAL_NETWORK_SUBNET_NAME`}}",
        "virtual_network_resource_group_name": "{{env `VIRTUAL_NETWORK_RESOURCE_GROUP_NAME`}}",
        "gallery_resource_group": "{{env `GALLERY_RESOURCE_GROUP`}}",
        "gallery_name": "{{env `GALLERY_NAME`}}",
        "gallery_location": "{{env `GALLERY_LOCATION`}}",
        "gallery_image_version": "{{env `GALLERY_IMAGE_VERSION`}}",
        "ImageExpireDate": "{{env `GALLERY_IMAGE_EXPIRATION_DATE`}}",
        "managed_image_resource_group_name": "{{env `MANAGED_IMAGE_RESOURCE_GROUP_NAME`}}"
    },
    "builders": [
        {
            "type": "azure-arm",
            "os_type": "Windows",
            "image_publisher": "MicrosoftWindowsServer",
            "image_offer": "WindowsServer",
            "image_sku": "2019-Datacenter",
            "communicator": "winrm",
            "winrm_use_ssl": "true",
            "winrm_insecure": "true",
            "winrm_timeout": "10m",
            "winrm_username": "packer",
            "virtual_network_name": "{{user `virtual_network_name`}}",
            "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",
            "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}",
            "managed_image_name": "packertempwin2019dc",
            "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
            "location":"{{user `gallery_location`}}",
            "vm_size": "Standard_DS2_v2",
            "disk_additional_size": 500,
            "polling_duration_timeout": "3h",
            "shared_image_gallery_destination": {
                "resource_group": "{{user `gallery_resource_group`}}",
                "gallery_name": "{{user `gallery_name`}}",
                "image_name": "GF-Base-WindowsServer-2019-Datacenter",
                "image_version": "{{user `gallery_image_version`}}",
                "replication_regions": ["eastus", "eastus2", "northcentralus","uksouth","ukwest"]
            },
            "azure_tags": {
                "ImageExpireDate": "{{user `ImageExpireDate`}}",
                "ImageBase": "Windows-2019-Datacenter-latest"
            }
        }
    ],
    "provisioners": [
        {
            "type": "powershell",
            "inline": [
                " # NOTE: the following *3* lines are only needed if the you have installed the Guest Agent.",
                "  while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
                "  while ((Get-Service WindowsAzureTelemetryService).Status -ne 'Running') { Start-Sleep -s 5 }",
                "  while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }",

                "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",
                "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit /mode:vm",
                "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
            ]
        }
    ]
}

Packer version

Packer v1.4.5

Operating system and Environment details

Packer is running on a Azure Windows Server 2019 Datacenter VM.

[pmozbert]

I have verified that if I switch to using client id, client secret, and subscription id, them the problem is fixed. The gallery subscription id parameter is now populated correctly and I get this output.

==> azure-arm:  -> MDI ID used for SIG publish     : '/subscriptions/12ef276f-c342-456f-b802-909283fed03a/resourceGroups/tempimagebuildrg/providers/Microsoft.Compute/images/packertempwin2019dc'

This is the working configuration file. The only change is the addition of the parameters for tenant id, subscription id, client id, and client secret

{
    "variables": {
        "tenant_id": "{{env `BUILD_TENANT_ID`}}",
        "subscription_id": "{{env `BUILD_SUBSCRIPTION_ID`}}",
        "client_id": "{{env `BUILD_CLIENT_ID`}}",
        "client_secret": "{{env `BUILD_CLIENT_SECRET`}}",
        "virtual_network_name": "{{env `VIRTUAL_NETWORK_NAME`}}",
        "virtual_network_subnet_name": "{{env `VIRTUAL_NETWORK_SUBNET_NAME`}}",
        "virtual_network_resource_group_name": "{{env `VIRTUAL_NETWORK_RESOURCE_GROUP_NAME`}}",
        "gallery_resource_group": "{{env `GALLERY_RESOURCE_GROUP`}}",
        "gallery_name": "{{env `GALLERY_NAME`}}",
        "gallery_location": "{{env `GALLERY_LOCATION`}}",
        "gallery_image_version": "{{env `GALLERY_IMAGE_VERSION`}}",
        "ImageExpireDate": "{{env `GALLERY_IMAGE_EXPIRATION_DATE`}}",
        "managed_image_resource_group_name": "{{env `MANAGED_IMAGE_RESOURCE_GROUP_NAME`}}"
    },
    "sensitive-variables": ["client_secret"],
    "builders": [
        {
            "type": "azure-arm",
            "tenant_id": "{{user `tenant_id`}}",
            "subscription_id": "{{user `subscription_id`}}",
            "client_id": "{{user `client_id`}}",
            "client_secret": "{{user `client_secret`}}",
            "os_type": "Windows",
            "image_publisher": "MicrosoftWindowsServer",
            "image_offer": "WindowsServer",
            "image_sku": "2019-Datacenter",
            "communicator": "winrm",
            "winrm_use_ssl": "true",
            "winrm_insecure": "true",
            "winrm_timeout": "10m",
            "winrm_username": "packer",
            "virtual_network_name": "{{user `virtual_network_name`}}",
            "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",
            "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}",
            "managed_image_name": "packertempwin2019dc",
            "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
            "location":"{{user `gallery_location`}}",
            "vm_size": "Standard_DS2_v2",
            "disk_additional_size": 500,
            "polling_duration_timeout": "3h",
            "shared_image_gallery_destination": {
                "resource_group": "{{user `gallery_resource_group`}}",
                "gallery_name": "{{user `gallery_name`}}",
                "image_name": "GF-Base-WindowsServer-2019-Datacenter",
                "image_version": "{{user `gallery_image_version`}}",
                "replication_regions": ["eastus", "eastus2", "northcentralus","uksouth","ukwest"]
            },
            "azure_tags": {
                "ImageExpireDate": "{{user `ImageExpireDate`}}",
                "ImageBase": "Windows-2019-Datacenter-latest"
            }
        }
    ],
    "provisioners": [
        {
            "type": "powershell",
            "inline": [
                " # NOTE: the following *3* lines are only needed if the you have installed the Guest Agent.",
                "  while ((Get-Service RdAgent).Status -ne 'Running') { Start-Sleep -s 5 }",
                "  while ((Get-Service WindowsAzureTelemetryService).Status -ne 'Running') { Start-Sleep -s 5 }",
                "  while ((Get-Service WindowsAzureGuestAgent).Status -ne 'Running') { Start-Sleep -s 5 }",

                "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",
                "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit /mode:vm",
                "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
            ]
        }
    ]
}

[nywilken]

hi @pmozbert thanks for opening up this issue and for the detailed repro/working configurations. I've been testing SIG related things using client id, client secret, and subscription id so have not seen this issue. But we will take a look as soon as we get a chance. If you'd like to open a PR with a possible fix we would be happy to review.

[pmozbert]

hi @nywilken i don't know the code well enough to offer a fix, but it looks to me like there should be another parameter 'gallery_subscription' under shared_image_gallery_destination. This would also be useful to support cases where the gallery is in a different subscription from where packer is running and creating the image.

[rahulsen]

I am experiencing this exact problem with packer 1.5.1 on Ubuntu 18. Is there a fix on the horizon ?

[SwampDragons]

I'll try to have someone take a look for the next release.

[nywilken]

Hi @pmozbert @rahulsen thanks for reaching out here. The Azure builder will use the instance metadata for retrieving the subscriptionId when authenticating via a managed identity. ~It is a simple HTTP Get request, which I can see coming back empty if the metadata service is not ready; thus the empty value for subscriptionId.~

By any chance are you seeing any indication that the VM is failing to get the subscriptionId in the logs? An error message like error fetching subscriptionID from VM metadata service for Managed Identity authentication.

[nywilken]

Hi Folks! 
I think I have found a fix for this. Looking at the issue further it appears that at the beginning of a build in Prepare the Azure builder will persist the value of subscription_id, as defined in the build template, to the statebag. Which is empty when using a managed identity for authentication.

Once the build starts to execute in Run a call is made to obtain the actual subscription_id from the metadata server, which is updated on the config object, but not in the statebag; thus causing the following code to return an empty string stateBag.Get(constants.ArmManagedImageSubscription).(string).

8685 fixes the issue by ensuring the subscription id from the metadata server is persisted in the statebag so that it can be used by the publish to shared image gallery step. Here are some binaries https://app.circleci.com/jobs/github/hashicorp/packer/31760/artifacts. 
Could you please tell me if the fix works for you? :smile:

Thank you!

[sgerace]

@nywilken I happened to run into this same problem this afternoon and found my way to this issue. Imagine my surprise when I saw that you had posted a proposed fix mere hours before! I wanted to let you know that I tried the updated binaries you posted and now the issue appears to be resolved. Thanks!

[nywilken]

@nywilken I happened to run into this same problem this afternoon and found my way to this issue. Imagine my surprise when I saw that you had posted a proposed fix mere hours before! I wanted to let you know that I tried the updated binaries you posted and now the issue appears to be resolved. Thanks!

@sgerace this is awesome to hear! Thanks for testing the proposed fix, and I'm glad that it was able to help you get past your current issue. This fix will be part of the next release, but if you would like to get access to it sooner it will be in our nightly release once merged. Cheers.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.