Terraform-provider-azurerm: Intermittent Error: Unable to locate Storage Account "<<storage_account_name>>"!

[smushtaq]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

1.35.0
1.36.0
1.37.0
1.38.0
1.39.0

Description

I am getting this error intermittently when using azurerm provider 1.35.0, 1.36.0, 1.37.0, 1.38.0, 1.39.0. This issue doesnot occur on azurerm provider 1.34.0 and on earlier versions.

Anyone else had seen this issue. Below is the log for terraform plan command

2020-01-03T14:29:08.9014786Z ##[section]Starting: Terraform plan
2020-01-03T14:29:08.9018892Z ==============================================================================
2020-01-03T14:29:08.9018984Z Task         : Terraform
2020-01-03T14:29:08.9019106Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2020-01-03T14:29:08.9019365Z Version      : 0.0.142
2020-01-03T14:29:08.9019475Z Author       : Microsoft Corporation
2020-01-03T14:29:08.9019548Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2020-01-03T14:29:08.9019668Z ==============================================================================
2020-01-03T14:29:09.0425642Z [command]/opt/hostedtoolcache/terraform/0.12.17/x64/terraform providers
2020-01-03T14:29:09.5124944Z .
2020-01-03T14:29:09.5126416Z └── provider.azurerm =1.39.0
2020-01-03T14:29:09.5126663Z 
2020-01-03T14:29:09.5157691Z [command]/opt/hostedtoolcache/terraform/0.12.17/x64/terraform plan -var-file=envs/dev/vars.auto.tfvars -out=tfplan -no-color -input=false
2020-01-03T14:29:11.1176401Z Refreshing Terraform state in-memory prior to plan...
2020-01-03T14:29:11.1179110Z The refreshed state will be used to calculate this plan, but will not be
2020-01-03T14:29:11.1179729Z persisted to local or remote state storage.
2020-01-03T14:29:11.1180065Z 
2020-01-03T14:29:13.9250714Z data.azurerm_resource_group.power: Refreshing state...
2020-01-03T14:29:13.9614330Z azurerm_key_vault.xxxxxxxxxx: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/mykv]
2020-01-03T14:29:13.9639469Z azurerm_storage_account.power_storage_account: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.Storage/storageAccounts/mysa]
2020-01-03T14:29:13.9644189Z azurerm_application_insights.power_app_insights: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/microsoft.insights/components/myai-we]
2020-01-03T14:29:13.9656039Z azurerm_app_service_plan.func_apps_consumption_plan: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.Web/serverfarms/mycsp]
2020-01-03T14:29:14.1513932Z azurerm_key_vault_access_policy.spn_access_policy: Refreshing state... [id=/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/myvault/objectId/xxxxx-xxxx-xxxx-xxxxx]
2020-01-03T14:29:16.1417214Z 
2020-01-03T14:29:16.1418746Z Error: Unable to locate Storage Account "mysa"!
2020-01-03T14:29:16.1418814Z 
2020-01-03T14:29:16.1418851Z 
2020-01-03T14:29:16.2364297Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.17/x64/terraform' failed with exit code 1
2020-01-03T14:29:16.2381103Z ##[section]Finishing: Terraform plan

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

data "azurerm_resource_group" "power" {
  name = var.resource_group_name
}

resource "azurerm_storage_account" "power_storage_account" {
  name                      = "${var.product_code}sa${var.env_code}${var.location_code_we}"
  resource_group_name       = data.azurerm_resource_group.power.name
  location                  = var.location
  account_tier              = var.storage_account_tier
  account_replication_type  = var.storage_account_replication_type
  account_kind              = "StorageV2"
  enable_https_traffic_only = true
  tags                      = var.default_tags  
}

resource "azurerm_application_insights" "power_app_insights" {
  name                = "${var.product_code}-df-${var.env_code}-ai-${var.location_code_we}"
  location            = var.location
  resource_group_name = data.azurerm_resource_group.power.name
  application_type    = "web"
  tags                = var.default_tags
}

Debug Output

Panic Output

Expected Behavior

It should locate the existing storage account

Actual Behavior

Error: Unable to locate Storage Account "<>"!

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[ttjackott]

We get this issue either on a 'terraform apply' or a 'terraform plan' and it's completely intermittent, happening for us 20-30% of the time (without having changed anything). It seems terraform is unable to find the storage account intermittently.

Oddly enough, for us this can even happen when the storage account in question even exists as a resource within the main.tf terraform config e.g.:
`
data.azurerm_resource_group.main: Refreshing state...
module.storage_account.data.azurerm_client_config.current: Refreshing state...
module.storage_account.data.azurerm_resource_group.main: Refreshing state...
module.storage_account.azurerm_storage_account.storage_account: Refreshing state... [id=/subscriptions/xxxx/resourceGroups/xxxx/providers/Microsoft.Storage/storageAccounts/mystorageaccount]
module.azure_function.data.azurerm_resource_group.main: Refreshing state...
module.azure_function.data.azurerm_client_config.current: Refreshing state...
module.azure_function.azurerm_application_insights.insights: Refreshing state... [id=/subscriptions/xxxx/resourceGroups/xxxx/providers/microsoft.insights/components/xxxx]
module.azure_function.azurerm_app_service_plan.plan: Refreshing state... [id=/subscriptions/xxxx/resourceGroups/xxxx/providers/Microsoft.Web/serverfarms/xxxx]
azurerm_storage_container.images_container: Refreshing state... [id=https://mystorageaccount.blob.core.windows.net/images]
module.azure_function.module.storage_account_for_function.data.azurerm_resource_group.main: Refreshing state...
module.azure_function.module.storage_account_for_function.data.azurerm_client_config.current: Refreshing state...
module.azure_function.module.storage_account_for_function.azurerm_storage_account.storage_account: Refreshing state... [id=/subscriptions/xxxx/resourceGroups/xxxx/providers/Microsoft.Storage/storageAccounts/mystorageaccount]

Error: Unable to locate Storage Account "mystorageaccount"!
`

This is frustrating though as it leads to our pipelines to fail intermittently through no fault of our own

[lbrulet]

same problem here... We are using the version 2.9 and got the same problem
We have to run the script twice to mount the storage.

[tcz001]

Recently we see a lot of issues introduced by this error,

I'm assuming it's a issue caused by Azure Storage account List API handling in terraform-azurerm-provider because of following reasons, will try to collect debug logs in places that reproducing this error to prove the assumption.

Behaviors:

1. It's a known issue reported by community github issue
2. it's causing a relative issue on child container state refresh, because of the error handling to "empty" account list response: github source code
   
   
   
   1. account list API is used in azurerm_storage_container
   
   
   2. account getProperties API is used in azurerm_storage_data_lake_gen2_filesystem
   
   
3. it's happening Intermittently, probably be a bug of Azure Storage API state inconsistency and misuse of HTTP response statusCode, maybe a network issue of agent (unlikely to be agent issue because there is no error message responded, but simply account == nil)

Short term workarround:

1. if you are using ADLS Gen2 Storage account, use azurerm_storage_data_lake_gen2_filesystem instead of azurerm_storage_container (because of the API issue)
2. if you see a Unable to locate try rerun the job, and check if it still happen
3. if you see a container creation failure due to 409 already exist, try terraform import to get back the state
4. if you see a follow-up issue different from what reported here, please respond to this thread

Potential long term solution:

1. report this issue to MSFT to fix the response status code and errors in response, instead of responding with an empty list
2. change the error handling in this Terraform provider, remove the "assuming removed, removing state" feature and use GetProperties for Storage Account existence checking

[tcz001]

@tombuildsstuff @katbyte can you take a look please?

[manali14]

We are facing a similar issue with the latest release i.e, 2.22. Do we have any update on this?

[Farhie]

As @tcz001 suggested, I am currently pushing this with MSFT account manager.

I'll put a PR in to move to using the more reliable getProperties API.

[tcz001]

We got some updates after troubleshooting with MSFT,
the List API will response 200 OK with empty list that causes this issue, when Azure Management Storage Provider Limits hit: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#storage-resource-provider-limits
it usually should response with 429, but the List accounts API is misbehaving on this scenario

[tcz001]

As MSFT support team mentions the List API we use is responding a pagination response, which is ignored by the terraform azurerm provider, a similar issue is addressed on other resource: https://github.com/terraform-providers/terraform-provider-azurerm/pull/7036/files

[nerddtvg]

I'm seeing similar but slightly different results. Using versions 2.20.0 and 2.31.1 we're receiving:

Error: Error retrieving Account "{{{ account name }}}" for Container "logs": Error retrieving next page of storage accounts: storage.AccountsClient#listNextResults: Failure sending next results request: StatusCode=429 -- Original Error: context deadline exceeded

We get a 429 ResourceCollectionRequestsThrottled instead of the improper 200, however we also do not receive the limit headers anymore either.

When performing the List operation manually via command line, it there is a nextLink only 4 times. I don't see how we're hitting a 100 request / 5 minutes limit so I truly think this is a problem on MS' side but can't prove it.