Terraform-provider-azurerm: Error while creating storage account

[shivamsriva31093]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.3

• provider.azurerm v1.32.0
• provider.null v2.1.2
• provider.random v2.1.2
• provider.template v2.1.2

Affected Resource(s)

• azurerm_storage_account

Terraform Configuration Files

`
variable "storage_name" {}
variable "region" {}
variable "resource_group_name" {}
variable "environment" {}
variable "usage" {}
variable "dependencies" {
type = list(string)
}

resource "null_resource" "dependency_getter" {
triggers = {
my_dependencies = join(",", var.dependencies)
}
}

Generate random text for a unique name

resource "random_id" "randomId" {
keepers = {
# Generate a new ID only when a new resource group is defined
resource_group = var.resource_group_name
}
byte_length = 8
}

resource "azurerm_storage_account" "stgacc" {
name = "diag${random_id.randomId.hex}"
resource_group_name = var.resource_group_name
location = var.region
account_tier = "Standard"
account_replication_type = "LRS"

tags = {
environment = var.environment
}

depends_on = [null_resource.dependency_getter]
}

resource "azurerm_storage_account" "stgaccount" {
name = var.storage_name
resource_group_name = var.resource_group_name
location = var.region
account_tier = "Standard"
account_replication_type = "LRS"

tags = {
environment = var.environment
}

depends_on = [null_resource.dependency_getter]
}
`

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

Panic Output

Expected Behavior

Terraform successful creation statement.

Actual Behavior

Error: Error reading queue properties for AzureRM Storage Account "diag5d01f638b17d3d58": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeffAuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission.\nRequestId:4554aa06-9003-00dc-7bb2-43c9be000000\nTime:2019-07-26T13:01:08.2516744Z" error: invalid character 'ï' looking for beginning of value

Error: Error reading queue properties for AzureRM Storage Account "tboxfkartstorage": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeffAuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission.\nRequestId:df5abaf5-5003-0010-03b2-43d411000000\nTime:2019-07-26T13:00:58.3772146Z" error: invalid character 'ï' looking for beginning of value

Steps to Reproduce

1. terraform apply

Important Factoids

The resources are created and visible in the azure portal with proper configuration. But the script ends without success message.

References

• #0000

[katbyte]

Hi @shivamsriva31093, this sounds a lot like #3925. Would you by any chance have a firewall enabled for the storage account?

[shivamsriva31093]

@katbyte I checked with the team. There is no firewall enabled for the storage account.

[dky]

@katbyte I'm experiencing this too, quick google lead me to this. We do not have fw enabled on our storage accounts either. If it helps

Getting the following error. Something about the container in the storage account.

Error: Error retrieving Container "vhds" (Account "sa" / Resource Group "sa"): containers.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:7e9fcabd-201c-0025-3839-49f975000000\nTime:2019-08-02T13:55:06.0606086Z</Message></Error>" error: invalid character 'ï' looking for beginning of value

[sesispla]

I'm experiencing something similar. Deployed 5 storage accounts for 5 different environments some time ago. Exact same deployment but the only difference is that the one that fails is GRS.

Terraform v0.11.11
provider.azurerm v1.32.0

Trying to apply some environment changes and had this error during terraform plan:

module.my-storage-account.azurerm_storage_account.my-storage-account: azurerm_storage_account.my-storage-account: 
Error reading queue properties for AzureRM Storage Account "mystorage": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=500 
-- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" 
encoding=\"utf-8\"?><Error><Code>
InternalError</Code><Message>
Server encountered an internal error. Please try again after some time.
\nRequestId:b0e6852b-7003-0015-2308-47d453000000\nTime:2019-07-30T18:57:11.2853333Z</Message></
Error>" error: invalid character 'ï' looking for beginning of value‌

This storage account was deployed with Terraform some time ago, so I suspect something has changed in the AzureAPI that broke this.

[backerman]

I thought this was related to my account coming off the free trial, but I ran az login again and the az account information corresponds to the current portal state. This configuration reproduces the bug:

variable "location" {
  default = "eastus"
}

provider "azurerm" {
  version = "~> 1.32"
}

resource "azurerm_resource_group" "rg" {
  name = "azurerm-bug3939-test"
  location = "${var.location}"
}

resource "azurerm_storage_account" "sa" {
  name                      = "bug3939testme"
  resource_group_name       = "${azurerm_resource_group.rg.name}"
  location                  = "${var.location}"
  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = "LRS"
  enable_https_traffic_only = "true"
}

[chaseWillden]

I'm actually having the same issue. Any eta on a fix?

[loderunner]

I am deploying to the France Central region, and facing the same problem.

I originally updated from 1.32.0 to 1.32.1 because I was blocked by #3968. This issue seems to be indeed fixed. But I'm facing #3939 now.

Terraform configuration:

resource "azurerm_resource_group" "main" {
  name     = "bug3939-resources"
  location = "francecentral"
}

resource "azurerm_storage_account" "blob_storage" {
  name                     = "bug3939storage"
  resource_group_name      = "${azurerm_resource_group.main.name}"
  location                 = "${azurerm_resource_group.main.location}"
  account_kind             = "BlobStorage"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "blob_container" {
  name                  = "bug3939-container"
  storage_account_name  = "${azurerm_storage_account.blob_storage.name}"
  container_access_type = "blob"
}

Error message:

Error: Error reading queue properties for AzureRM Storage Account "************
******": queues.Client#SetServiceProperties: Failure responding to request: Sta
tusCode=403 -- Original Error: autorest/azure: error response cannot be parsed:
 "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationPer
missionMismatch</Code><Message>This request is not authorized to perform this o
peration using this permission.\nRequestId:37d11365-1003-0045-5d12-49453f000000
\nTime:2019-08-02T09:10:09.0124693Z</Message></Error>" error: invalid character
 'ï' looking for beginning of value

My current workaround is to move the location to North Europe.

[shivamsriva31093]

@loderunner I didn't try changing the location. I will try it.

[danvy]

I have the same issue with North Europe and West Europe using version 1.32.0 and 1.32.1.

[dky]

Hey guys, I ended up rolling back the azure rm provider to 1.28 and looks like this is fixed on my end so it's definitely something in the 1.3x release.

[loderunner]

I've been trying to reproduce this issue over the week-en, and found that the same error manifests at different points in different scenarios.

I built a simple Terraform Azure configuration that should deploy a storage account and container, and made variables for location and storage kind.

provider "azurerm" {
  version = "~> 1.32.1"
}

variable "region" {}

variable "storage_kind" {}

resource "azurerm_resource_group" "main" {
  name     = "loderunner-bugtest-resources"
  location = "${var.region}"
}

resource "azurerm_storage_account" "blob_storage" {
  name                     = "loderunnerbugteststorage"
  resource_group_name      = "${azurerm_resource_group.main.name}"
  location                 = "${azurerm_resource_group.main.location}"
  account_kind             = "${var.storage_kind}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "blob_container" {
  name                  = "loderunnerbugtest"
  storage_account_name  = "${azurerm_storage_account.blob_storage.name}"
  container_access_type = "blob"
}

I ran the same scenario for 4 different combinations:

1. terraform apply
2. terraform apply (whatever the result of 1.)
3. terraform destroy

Ypu can find the logs from these operations in this gist.

• Location: France Central, Storage Kind: BlobStorage: no error
• Location: France Central, Storage Kind: StorageV2: fails at 2. while refreshing state
• Location: East US, Storage Kind: BlobStorage: fails at 1. and 2. while creating resources
• Location: East US, Storage Kind: StorageV2: fails at 1. and 2. while creating resources, fails at 3. while refreshing state

The ideal configuration for my current project if France Central/StorageV2.

Hope this helps, keep me posted if you want me to investigate more.

[sesispla]

I raised a Support Ticket to Microsoft Azure. This is the answer from the support team to this 500 error:

After further checking from our backend log, we found the request was failed by authentication error since our backend side was doing the certificate rollover.

Failed to get access token. StatusCode: 401, Response: {"error":"invalid_client","error_description":"AADSTS700030: Invalid certificate}

Please try to apply the request again, it should be working now.

Tried again and the error was gone. So looks like a Microsoft problem, not a Terraform one. Nevertheless, I don't know if there is anything Terraform can do to make this error more user-friendly... at least in my scenario where I received a 500 error.

[backerman]

Still doing it in eastus with 1.3x, working with 1.28.

[conork123]

Having same issues in West US 2 with 1.32.1 - can't go to 1.28 due to other resources utilising features of 1.32.0. ie https://github.com/terraform-providers/terraform-provider-azurerm/pull/3785

[shivamsriva31093]

@loderunner I changed the region from southeast asia to east asia and the script ran successfully. I will check with version downgrading like @backerman .

[vermaaakansha]

Im experiencing the same issue.
tf code:
resource "azurerm_storage_account" "StorageAccount" {
name = "${var.storage-account-name}"
location = "${var.location}"
resource_group_name = "${var.resource-group-name}"
account_kind = "${var.storage-account-kind}"
account_tier = "${var.storage-account-tier}"
account_replication_type = "${var.storage-replication-type}"
enable_https_traffic_only = "${var.enable-https-traffic-only}"
is_hns_enabled = "${var.is-hns-enabled}"

network_rules {
ip_rules = ["205.145.64.0/18"]
virtual_network_subnet_ids = ["${data.azurerm_subnet.subnet.id}"]
}
}

go code:

package test

import (
"github.com/gruntwork-io/terratest/modules/terraform"
"testing"
//terraformCore "github.com/hashicorp/terraform/terraform"
)

func TestTerraformAzureStorage(t *testing.T) {
t.Parallel()
terraformOptions := &terraform.Options{
TerraformDir: "/Users/aakansha.a.verma/go/src/storage-account/ref.cloud.azure.golden.storageaccount",

    Vars: map[string]interface{}{
        "storage-account-name": "storaccname1310112",
        "location":     "eastus",
    },
}
terraform.InitAndApply(t, terraformOptions)

}
error:
Error reading queue properties for AzureRM Storage Account "storaccname1310112": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeffAuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission.\nRequestId:7518a50c-6003-0015-7c10-5219ec000000\nTime:2019-08-13T19:53:18.1412641Z" error: invalid character 'ï' looking for beginning of value?

CAN ANYONE SUGGEST ON THIS ERROR? any go developer here?

[sciurus]

I can reproduce this. With no changes to my terraform code or infrastructure state, but upgrading my provider from 1.28.0 to 1.32.1, terraform plan started to fail with the error from this report. This is a Storage (general purpose v1) account East US, West US. Reverting to 1.28.0 makes the error disappear.

[tombuildsstuff]

:wave:

So I'm struggling to reproduce this error, with different account kinds in different regions - but I believe the root cause is a permissions issue.

Beginning in v1.32 of the Azure Provider we now can manage the Storage Properties for a Queue, which are configured on the top-level Storage Account; as such even if none are defined we make a call to the SetServiceProperties API call for Queues to ensure these are nil-ed out if any exist.

As such I believe this may need an additional permission to configure these properties (outlined here) - but without being able to reproduce this error makes it hard for me to confirm for sure; that said - looking at the documentation I believe this should be the Storage Queue Data Contributor permission.

So that we can debug this further, would it be possible for someone to confirm the permissions for the account/service principal they're using for the Resource Group/Storage Account/Container, and which means of authentication you're using (e.g. Azure CLI, MSI, Service Principal etc)?

Thanks!

[mikhailshilkov]

I got these errors with Azure CLI authenticating as myself - the subscription owner, so I definitely have all the required permissions

[sladeedmonds]

I am using Azure cli with an AAD account that has Contributor on the Subscription.

[tombuildsstuff]

@sladeedmonds @mikhailshilkov thanks for confirming - out of interest when were these storage accounts created, and in which regions?

[sladeedmonds]

The storage account in question is roughly 1 year old and was not created with Terraform. I've created several other storage accounts configured exactly the same (aside from the name, of course) in the same region, with Terraform and they all created without issue and plan/apply without issue. These storage accounts are in us gov virginia. The only real noticeable difference between the ones that work and this one that causes the error is in the naming. The storage account that triggers the failure has the digits 001 at the end of the account name.

I will try and import another storage account that was created at around the same time and see if it behaves the same.

[sladeedmonds]

So I just imported another storage account that also had a numeric suffix... this one 01. This storage account was created at roughly the same time as the one causing issues. It imported without issue and plan/apply ran as expected. The storage account causing the issues is not an important account which I determined could easily be re-created without worry of data loss, etc. So I removed it and let Terraform re-create it. Same issue. I then removed it, re-created it with Terraform and instead of having 001 at the end, I changed it to 01. Same issue... it creates the account but also throws the error on a plan/apply Just for giggles, I blew it away again, this time creating it without any numerical suffix, and that works without issue.

There seems to be something with the name of the account itself that is causing this?

[mikhailshilkov]

@tombuildsstuff I can repro it with a new storage account with this gist:

1. Change the version to 1.31 in the above gist.
2. terraform init
3. terraform apply
4. Change the version back to 1.32.1
5. terraform init
6. terraform apply fails at refreshing state with

Error: Error reading queue properties for AzureRM Storage Account "atftest": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.\nRequestId:f0b76b70-1003-0077-1dce-56b422000000\nTime:2019-08-19T20:43:30.5170838Z</Message></Error>" error: invalid character 'ï' looking for beginning of value

[seanknox]

Hitting this repeatedly too.

• Changing regions doesn't seem to help: tried South Central US, West US 2, East US.
• Can't downgrade the Azure provider to 1.28 unfortunately, since my Terraform manifest requires features in 1.3.x.
• Tried different storage names, v1 and v2 types, hot and cold settings, etc.

Additionally, if I create an account in the portal and then try to import, I get a different error, which may or may not be related to this issue:

azurerm_storage_account.main: Importing from ID "mystorageaccount"...
azurerm_storage_account.main: Import prepared!
  Prepared azurerm_storage_account for import
azurerm_storage_account.main: Refreshing state... [id=mystorageaccount]

Error: Cannot parse Azure ID: parse mystorageaccount: invalid URI for request

[tombuildsstuff]

@seanknox

Additionally, if I create an account in the portal and then try to import, I get a different error, which may or may not be related to this issue:

The Azure Provider uses the Resource ID as the Identifier rather than the Name, as such you'll need to use this to import it (an example can be found in the docs).

[tombuildsstuff]

@mikhailshilkov thanks for the repro - but unfortunately I'm still unable to repro this using the steps you've outlined when authenticating using the Azure CLI or a Service Principal (and when going from 1.31.0 -> 1.32.1 or 1.32.1 -> master):

$ tf init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (terraform-providers/azurerm) 1.31.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ tf apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.rg will be created
  + resource "azurerm_resource_group" "rg" {
      + id       = (known after apply)
      + location = "westus"
      + name     = "testResourceGroup"
      + tags     = (known after apply)
    }

  # azurerm_storage_account.atftest will be created
  + resource "azurerm_storage_account" "atftest" {
      + access_tier                      = (known after apply)
      + account_encryption_source        = "Microsoft.Storage"
      + account_kind                     = "Storage"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + account_type                     = (known after apply)
      + enable_blob_encryption           = true
      + enable_file_encryption           = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + location                         = "westus"
      + name                             = "atftes999t"
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + resource_group_name              = "testResourceGroup"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)
      + tags                             = (known after apply)

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }
    }

  # azurerm_storage_table.test will be created
  + resource "azurerm_storage_table" "test" {
      + id                   = (known after apply)
      + name                 = "mysampletable"
      + resource_group_name  = "testResourceGroup"
      + storage_account_name = "atftes999t"
    }

  1 provider "azurerm" {
  # azurerm_storage_table.test2 will be created
  + resource "azurerm_storage_table" "test2" {
      + id                   = (known after apply)
      + name                 = "mysampletable2"
      + resource_group_name  = "testResourceGroup"
      + storage_account_name = "atftes999t"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.rg: Creating...
azurerm_resource_group.rg: Creation complete after 3s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup]
azurerm_storage_account.atftest: Creating...
azurerm_storage_account.atftest: Still creating... [10s elapsed]
azurerm_storage_account.atftest: Still creating... [20s elapsed]
azurerm_storage_account.atftest: Creation complete after 25s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup/providers/Microsoft.Storage/storageAccounts/atftes999t]
azurerm_storage_table.test: Creating...
azurerm_storage_table.test2: Creating...
azurerm_storage_table.test: Creation complete after 3s [id=https://atftes999t.table.core.windows.net/mysampletable]
azurerm_storage_table.test2: Creation complete after 3s [id=https://atftes999t.table.core.windows.net/mysampletable2]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

$ tf apply
azurerm_resource_group.rg: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup]
azurerm_storage_account.atftest: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup/providers/Microsoft.Storage/storageAccounts/atftes999t]
azurerm_storage_table.test2: Refreshing state... [id=https://atftes999t.table.core.windows.net/mysampletable2]
azurerm_storage_table.test: Refreshing state... [id=https://atftes999t.table.core.windows.net/mysampletable]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

$ vim main.tf

$ tf init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (terraform-providers/azurerm) 1.32.1...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ tf apply
azurerm_resource_group.rg: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup]
azurerm_storage_account.atftest: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/testResourceGroup/providers/Microsoft.Storage/storageAccounts/atftes999t]
azurerm_storage_table.test2: Refreshing state... [id=https://atftes999t.table.core.windows.net/Tables('mysampletable2')]
azurerm_storage_table.test: Refreshing state... [id=https://atftes999t.table.core.windows.net/Tables('mysampletable')]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

---

That said looking at the Azure Docs here (and this Stack Overflow page) it appears the Storage Blob Data Contributor role is necessary when using AzureAD to authenticate to the Storage API's, which we're now doing (asa of 1.32):

Next, explicitly assign the `Storage Blob Data Contributor` role to the user account under which you will run the sample code. For instructions on how to assign this role in the Azure portal, see Grant access to Azure blob and queue data with RBAC in the Azure portal.

Note: When you create an Azure Storage account, you are not automatically assigned permissions to access data via Azure AD. You must explicitly assign yourself an RBAC role for Azure Storage. You can assign it at the level of your subscription, resource group, storage account, or container or queue.

Whilst I can't reproduce this, I believe this extra permission is the root cause of this - and whilst we could ask folks to assign this extra permission (which isn't ideal), it appears this may not be possible if a DataBricks resource is provisioned.

Version 1.32 of the Azure Provider switched to using AzureAD Authentication for accessing the Storage Queues API, which is why this permission is now required; as such I'm going to open a PR to switch this over to using SharedKey for authentication instead - which should resolve this.

Thanks!

[mikhailshilkov]

@tombuildsstuff Yeah, my repro has nothing special in it, so it must be related to the environment... Do you know where

invalid character 'ï' looking for beginning of value

comes from? I'm on Windows on a non-English locale, so one of my guesses were some encoding issues somewhere.

[tombuildsstuff]

@mikhailshilkov

invalid character 'ï' looking for beginning of value

That's coming from Azure/go-autorest which doesn't correctly parse the errors out of XML responses properly (since it's assumes the response is JSON by default) - I'd been meaning to send a PR for it but haven't got to that yet unfortunately.

[ljmsc]

hi guys, I have the same issue. Not a fix but a workaround for me was to create and delete a queue. Can someone confirm this as workaround or was it just a coincident?

[sladeedmonds]

@ljmsc must be a coincidence, didn't resolve the issue for me. Also, adding myself as a Storage Blob Contributor had no effect.

[ljmsc]

@ljmsc must be a coincidence, didn't resolve the issue for me. Also, adding myself as a Storage Blob Contributor had no effect.

Thanks @sladeedmonds for testing. I did a few more things before it worked. Maybe you can try them as well?

• reregister storage for the subscription
  az provider register -n Microsoft.Storage --subscription <subscription_id> --debug
• setting the subscription explicitly via the az cli (I have multiple subscriptions in use)
  `az account set -s
• adding myself explicitly as Owner / Contributor to the subscription.
• adding myself as Storage Queue Data Contributor in the storage account

After each step I did a az login

[hashibot[bot]]

This has been released in version 1.33.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.33.0"
}
# ... other configuration ...

[sladeedmonds]

I have confirmed that 1.33.0 resolves the issue I was having. Thanks much, @tombuildsstuff !

[invidian]

This is still affecting as after updating to 1.33. We consistently get 403 error, which cannot be decoded. Could it be that the API used for creating them changed and we lack permissions now?

[invidian]

Indeed it seems to be a firewall issue. Once I enabled firewall rule for storage account, which disallows my IP address, I am able to reproduce the issue:

However, our other account is allowed to be accessed from any network and the problem is still happening. Is there some other, global policy which could block it?

[invidian]

It seems this example is able to reproduce it:

variable "location" {
  default = "westeurope"
}

provider "azurerm" {
  version = "~> 1.33"
}

resource "azurerm_resource_group" "rg" {
  name = "mateusz-azurerm-bug3939-test-newiwest"
  location = "${var.location}"
}

resource "azurerm_storage_account" "sa" {
  name                      = "bug3939testmemateusz23"
  resource_group_name       = "${azurerm_resource_group.rg.name}"
  location                  = "${var.location}"
  account_kind              = "StorageV2"
  account_tier              = "Premium"
  account_replication_type  = "LRS"
  enable_https_traffic_only = "true"
}

resource "azurerm_storage_container" "blob_container" {
  name                  = "bug3939-container-mateusz24"
  storage_account_name  = "${azurerm_storage_account.sa.name}"
  container_access_type = "blob"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
}

I am not even able to create the storage container. I always get:

Error: Error applying plan:

1 error occurred:
    * azurerm_storage_container.blob_container: 1 error occurred:
    * azurerm_storage_container.blob_container: Error creating Container "bug3939-container-mateusz24" (Account "bug3939testmemateusz23" / Resource Group "mateusz-azurerm-bug3939-test-newiwest"): containers.Client#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:3541a8ac-901c-000a-1ff0-586415000000\nTime:2019-08-22T13:47:45.9619686Z</Message></Error>" error: invalid character 'ï' looking for beginning of value

[loderunner]

All my cases from this comment have been fixed by 1.33.0

:+1: Thanks!

[sciurus]

I can also confirm that after upgrading to 1.33, plan is working again for me.

[tombuildsstuff]

@invidian that's being tracked in https://github.com/terraform-providers/terraform-provider-azurerm/issues/2977

[invidian]

@tombuildsstuff I'm aware of #2977, but it looks like issue with premium storage accounts, as in example above happens regardless of network policies. Should I create another issue to track it?

[damianskobel]

unfortunately in my case problem persists

provider "azurerm" {
  version = "=1.33.0"
}

locals {
  stackName = "issue-3939-${terraform.workspace}"
  storageAccountName = "${replace(local.stackName, "/[^a-z]/", "")}"
}

terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-rg"
    storage_account_name = "issue3939tfstate"
    container_name       = "states"
    key                  = "state.terraform.tfstate"
    access_key           = "<access_key>"
  }
}

resource "azurerm_resource_group" "issue3939" {
  name     = "${local.stackName}-rg"
  location = "westus"
}

resource "azurerm_storage_account" "issue3939" {
  name = "issue3939"
  resource_group_name = "${azurerm_resource_group.issue3939.name}"
  location = "East US"
  account_tier = "Standard"
  account_replication_type = "RAGRS"
  account_kind = "StorageV2"

  network_rules {
    bypass = ["AzureServices"]
    default_action = "Allow"
  }

  enable_https_traffic_only = true
  enable_blob_encryption = true
  enable_file_encryption = true
  account_encryption_source = "Microsoft.Storage"
  access_tier = "Hot"

  queue_properties {
    cors_rule {
      allowed_origins = ["*"]
      allowed_methods = ["DELETE", "GET", "HEAD", "MERGE", "POST", "OPTIONS", "PUT", "PATCH"]
      max_age_in_seconds = 1
      exposed_headers = ["*"]
      allowed_headers = ["*"]
    }
  }

}

resource "azurerm_storage_container" "issue3939" {
  name                  = "web"
  storage_account_name  = "${azurerm_storage_account.issue3939.name}"
  container_access_type = "private"

}

issue occurs when queue_properties cors_rule is added

[tombuildsstuff]

@damianskobel this issue is being tracked in https://github.com/terraform-providers/terraform-provider-azurerm/issues/2977

[damianskobel]

@tombuildsstuff I'm not sure it's related, my error is more like above

Error: Error updating Azure Storage Account `queue_properties` "issue3939": queues.Client#SetServiceProperties: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>InvalidXmlDocument</Code><Message>XML specified is not syntactically valid.\nRequestId:0e9737f3-c003-0069-32a0-59d99b000000\nTime:2019-08-23T10:52:50.8725110Z</Message><LineNumber>2</LineNumber><LinePosition>56</LinePosition><Reason>Unexpected value for Version.</Reason></Error>" error: invalid character 'ï' looking for beginning of value

[seanknox]

Confirming that upgrading to v1.33, plan and apply are working for storage accounts again.

[invidian]

It seems I found the solution, at least to my problem. Switching Authentication signature from SharedKeyLite to SharedKey allows to create containers on premium storage accounts again. The temporary patch is here: https://github.com/kinvolk/terraform-provider-azurerm/commit/5d3dde77709a6ad77d1fbd0fdc5ae60d5a8f99e2. @tombuildsstuff how should I proceed with this patch?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!