Terraform-provider-azurerm: Cannot create azurerm_storage_container in azurerm_storage_account that uses network_rules

[phil-bevan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.11.11

• provider.azurerm v1.21.0
  

Affected Resource(s)

• azurerm_storage_account
• azurerm_storage_container

Terraform Configuration Files

resource "azurerm_storage_account" "test-storage-acct" {
  name                     = "${var.prefix}storacct"
  resource_group_name      = "${var.resgroup}"
  location                 = "${var.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  network_rules {
    ip_rules                   = ["aaa.bbb.ccc.ddd/ee"]
    virtual_network_subnet_ids = ["${var.subnetid}"]
  }
}
resource "azurerm_storage_container" "provisioning" {
  name                  = "${var.prefix}-provisioning"
  resource_group_name   = "${var.resgroup}"
  storage_account_name  = "${azurerm_storage_account.test-storage-acct.name}"
  container_access_type = "private"
}

Debug Output

• azurerm_storage_container.provisioning: Error creating container "philtesting1-provisioning" in storage account "philtesting1storacct": storage: service returned error: StatusCode=403, ErrorCode=AuthorizationFailure, ErrorMessage=This request is not authorized to perform this operation.
  RequestId:a7f9d2e1-701e-00b3-4e74-cf3b34000000
  Time:2019-02-28T14:45:53.7885750Z, RequestInitiated=Thu, 28 Feb 2019 14:45:53 GMT, RequestId=a7f9d2e1-701e-00b3-4e74-cf3b34000000, API Version=, QueryParameterName=, QueryParameterValue=
  

  Expected Behavior

  Container can be created in a storage account that uses network rules

  Actual Behavior

  After applying a network_rule to a storage account I cannot provision a container into it. My public IP is included in the address range specified in the network rule. I can successfully create the container via the Azure portal

  Steps to Reproduce

  1. terraform apply

[bs-matil]

Does reproduce also for v1.22.1

[sschu]

The reason is most likely that listing existing storage containers in a storage account directly accesses the storage accounts REST API. It fails if there are firewall rules in place not containing the ip of the host terraform runs on. It works if this IP is added. However, finding that IP is a challenge when terraform is run from Azure Devops as we do. This might not be easy to fix. Maybe storage account firewall rules should be their own resources that need to be added last in a deployment? Or creating a storage container resource first disables the firewall on the storage account and enables it afterwards?

[farseeker]

We just ran into this ourselves. Nice to see someone else has already raised the issue with excellent documentation.

The workaround we are testing is to call out to an ARM template for creating the containers. This is not ideal for several reasons:

1. It's not Terraform-native
2. It's more moving parts and more complicated to manage
3. ARM templates only apply once, so if the configuration drifts over time Terraform will not set it back

But it's what we've got. This could be a workaround for you if you need this.

I'm using two parts - a JSON file with the ARM, and a Terraform azurerm_template_deployment

storage-containers.json

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "storageAccountName": {
            "type": "string"
        },
        "location": {
            "type": "string"
        }
    },
    "resources": [
        {
            "name": "[parameters('storageAccountName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2018-07-01",
            "location": "[parameters('location')]",
            "resources": [
                {
                    "name": "default/images",
                    "type": "blobServices/containers",
                    "apiVersion": "2018-07-01",
                    "dependsOn": [
                        "[parameters('storageAccountName')]"
                    ]
                },
                {
                    "name": "default/backups",
                    "type": "blobServices/containers",
                    "apiVersion": "2018-07-01",
                    "dependsOn": [
                        "[parameters('storageAccountName')]"
                    ]
                }
            ]
        }
    ]
}

main.tf

resource "azurerm_storage_account" "standard-storage" {
  name                = "stdstorage"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"

  account_tier              = "Standard"
  account_replication_type  = "${var.standard_replication_type}"
  enable_blob_encryption    = "${var.standard_enable_blob_encryption}"
  enable_https_traffic_only = true

  network_rules {
    ip_rules                   = "${var.firewall_allow_ips}"
    virtual_network_subnet_ids = ["${var.vm_subnet_id}"]
  }
}

resource "azurerm_template_deployment" "stdstorage-containers" {
  name                = "stdstorage-containers"
  resource_group_name = "${var.resource_group_name}"
  deployment_mode     = "Incremental"

  depends_on = [
    "azurerm_storage_account.standard-storage",
  ]

  parameters {
    location           = "${var.location}"
    storageAccountName = "${azurerm_storage_account.standard-storage.name}"
  }

  template_body = "${file("${path.module}/storage-containers.json")}"
}

[ranokarno]

hi @tombuildsstuff ,
has this issue resolved ? is it related with PR#416

[tombuildsstuff]

@ranokarno not yet, this issue's open/tracking this bug

[amdiodati]

I'm hitting a very similar issue except that I'm trying to create a Storage Queue. Otherwise it's very similar from a technical standpoint (using ADO for deployment too).

[jeffadavidson]

I hit this bug using terraform 0.12.17 with AzureRM provider 1.37.0 and 1.38.0.

@sschu I am also deploying from Azure DevOps hosted machines. The workaround I created was:

• Prior to the terraform tasks, run an Azure PowerShell task that gets the hosts public IP and adds it to the network rules using 'Add-AzStorageAccountNetworkRule'
• Run Terraform steps
• Remove network rule using Remove-AzStorageAccountNetworkRule

It's workable, but still a pain

[shea-parkes]

I believe I just encountered this with v2.3.0 of the Azure provider. Given that this is still open, I'm assuming it hasn't been fixed?

[boillodmanuel]

Hello,
It's seems related to this azure-cli issue: https://github.com/Azure/azure-cli/issues/10190

Currently, the creation of a storage container resource (blob, share) seems to use the storage container API which is behind the firewall.
Instead, it should use the Resource Manager provider. In the issue mentionned above, I just discover that az cli has a az storage share-rm create in addition to existing az storage share create. I don't know if there is an equivalent for blob, and if this exists in the azure rest API or in terraform :)

[Duchynko]

Experiencing the same error with Terraform v0.12.25 and azurerm v2.9.0

resource "azurerm_storage_account" "sa_balancesheet_upload" {
  name                      = var.name
  resource_group_name       = var.resource_group_name
  location                  = var.location
  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = var.account_replication_type
  enable_https_traffic_only = var.enable_https_traffic_only

  network_rules {
    default_action = "Deny"
    ip_rules       = var.ip_rules
  }

  tags = {
    environment = var.environment
  }
}

resource "azurerm_storage_container" "sc_balancesheets" {
  name                  = "balancesheets"
  storage_account_name  = azurerm_storage_account.sa_balancesheet_upload.name
  container_access_type = "private"
}

[mcalnd70]

When using Azure DevOps hosted agents to deploy, I ended up writing this piece of Powershell that invokes Azure CLI to allow that specific agent's public IP address to be allowed into the Storage Account that had IP restrictions enabled. Like @jeffadavidson

It's a script you can call as part of your deployments that will toggle the public IP of that agent either on or off (-mode switch).

As mentioned I use it for Azure DevOps pipeline deployments, but it could be used anywhere else by other deployment tools...

<#
.SYNOPSIS
Set (by mode: ON OFF) the Storage Account Firewall Rules by Public IP address. Used by Azure DevOps Build/Release agents
See here : https://github.com/terraform-providers/terraform-provider-azurerm/issues/2977
.DESCRIPTION
Using Azure CLI
.EXAMPLE
.\SetMode_PublicIPAddress_SA.ps1 -storageaccount sa12345random -resourcegroup RG-NDM-TEST -mode on
.NOTES
Written by Neil McAlister - March 2020
#>
param (
    [Parameter(Mandatory=$true)]
    [string]$storageaccount,
    [Parameter(Mandatory=$true)]
        [string]$resourcegroup,
        [Parameter(Mandatory=$true)]
    [string]$mode
)
#
$ip = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
write-host $ip
#
if ($mode -eq 'on') { 
az storage account network-rule add --resource-group $resourcegroup --account-name $storageaccount --ip-address $ip
} 
#
if ($mode -eq 'off') {
az storage account network-rule remove --resource-group $resourcegroup --account-name $storageaccount --ip-address $ip
}

I have this with as a step in my deployments with a -mode on that allows access to the SA

I also have another step at the end with -mode off Note that you should run the -mode off step even if your deployment fails/crashes out, otherwise your SA firewall rules are going to get messy with lots of orphaned IP addresses in it.

If you are using YAML based pipelines, that setting is...

condition: always()

...if using GUI based releases it is a setting under ADVANCED options

[jjindrich]

I have same problem. I cannot create container from my workstation. I enabled virtual_network_subnet_ids (for application backend) and ip_rules for my workstation (running TF scripts). I'm getting Status=403 Code="AuthorizationFailure"

[TheKangaroo]

Same error in terraform v0.2.12

[tenderitaf]

hi, Is there any plan to fix this ?

[DustinChaloupka]

Reiterating @boillodmanuel 's comment about using the resource manager API instead of the storage account API which is behind the firewall. This isn't just for the create request either. Terraform refreshing the storage container properties also fails if the network rules prevent it. There is a resource manager API available that can be used instead, eg using the Azure CLI tool:

➜  ~ az storage container-rm list --storage-account myStorageAccount
Command group 'storage container-rm' is in preview. It may be changed/removed in a future release.
[
  {
    "defaultEncryptionScope": "$account-encryption-key",
    "deleted": false,
    "deletedTime": null,
    "denyEncryptionScopeOverride": false,
    "etag": "\"0x8D8720723A5BBDF\"",
    "hasImmutabilityPolicy": false,
    "hasLegalHold": false,
    "id": "/subscriptions/<subscription id>/resourceGroups/my-resource-group/providers/Microsoft.Storage/storageAccounts/myStorageAccount/blobServices/default/containers/myContainer",
    "immutabilityPolicy": null,
    "lastModifiedTime": "2020-10-16T19:17:37+00:00",
    "leaseDuration": null,
    "leaseState": "Available",
    "leaseStatus": "Unlocked",
    "legalHold": null,
    "metadata": null,
    "name": "myStorageContainer",
    "publicAccess": "None",
    "remainingRetentionDays": 0,
    "resourceGroup": "my-resource-group",
    "type": "Microsoft.Storage/storageAccounts/blobServices/containers",
    "version": null
  }
]

This was run on a storage account/container that had network rules preventing me from accessing the storage container through the storage account API.