terraform remote config Azure backend doesn't work

[mbravorus]

TL;DR
using remote state with azure backend doesn't work as expected

Terraform Version

0.7.0

Affected Resource(s)

terraform_remote_state
(with azure backend)

Terraform Configuration Files

the relevant part of the config is almost verbatim from the example in the docs and looks like this:

# setup remote state data source
data "terraform_remote_state" "azureremote" {
  backend = "azure"
  config {
    resource_group_name = "playground"
    storage_account_name = "remotetf"
    container_name       = "terraform-state"
    key                  = "play.terraform.tfstate"
  }
}

relevant credentials, namely:

• subscription_id
• tenant_id
• client_id
• client_secret

are also defined in the .tf file and have been thoroughly tested as working (except for the remote state thing)

same credentials have been exposed through ARM_* env variables (and re-tested to exclude copypasting errors)

Debug Output

the remote state configuration command-line and resulting error look like this (very short, no gist required):

 terraform remote config   -backend=azure -backend-config="resource_group_name=playground" -backend-config="storage_account_name=remotetf" -backend-config="container_name=terraform-state"   -backend-config="key=play.terraform.tfstate"
2016/08/15 18:09:28 [INFO] Terraform version: 0.7.0  e822a79165dbc06bbf8271ee349fe256867d53dc
2016/08/15 18:09:28 [DEBUG] Detected home directory from env var: /Users/mbravo
2016/08/15 18:09:28 [DEBUG] Detected home directory from env var: /Users/mbravo
2016/08/15 18:09:28 [DEBUG] Attempting to open CLI config file: /Users/mbravo/.terraformrc
2016/08/15 18:09:28 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2016/08/15 18:09:28 [DEBUG] Detected home directory from env var: /Users/mbravo
Remote configuration updated
2016/08/15 18:09:31 [DEBUG] plugin: waiting for all plugin processes to complete...
Error while performing the initial pull. The error message is shown
below. Note that remote state was properly configured, so you don't
need to reconfigure. You can now use `push` and `pull` directly.

Error reloading remote state: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:990fe1d5-0001-00b5-2c07-f7a1d9000000
Time:2016-08-15T15:09:30.8879450Z, RequestId=990fe1d5-0001-00b5-2c07-f7a1d9000000, QueryParameterName=, QueryParameterValue=

I have checked that I do not have any noticeable time drift which could be the cause for auth failure:

$ sudo ntpdate pool.ntp.org
15 Aug 17:56:56 ntpdate[73978]: adjust time server 109.226.40.40 offset 0.002268 sec

I'm running on a Mac, so there is no immediate way for me to intercept actual requests to infer more detailed error message, short of using tcpdump; if there will be no other way, I'm willing to do that, later perhaps

tfstate for the storage account mentioned looks correct and contains both primary and secondary access keys

Steps to Reproduce

after this remote config command, nothing works (including push), any action receives the same error (the only way to come back is to remove .terraform subdir and copy terraform.tfstate back from backup)

[radeksimko]

Hi @mbravorus
out of interest - how did you come up to a conclusion that terraform_remote_state affects terraform remote config or generally should be used within the same context/config? It is not the first time I saw people making the same mistake and it would be super helpful if we could prevent that - maybe by making it more clear in docs that Example Usage means push/pull of current working directory and Example Referencing means pull-only from anything that's not current working directory?

https://www.terraform.io/docs/state/remote/azure.html

Admittedly this may not be the cause of AuthenticationFailed error message. This error

Error reloading remote state

either way comes from the terraform remote config command and terraform_remote_state has nothing to do with it.

This however caught my eye:

relevant credentials, namely ... are also defined in the .tf file

The terraform remote config does not read any configuration from any *.tf file as mentioned above. Instead you will need to specify these credentials either via -backend-config flags or (probably better) ENV variables.

Can you try specifying the credentials as mentioned and let us know whether you were able to setup the remote state correctly?

[mbravorus]

Hi @radeksimko ,

thanks for getting back to me so quickly. Let me explain - I understand (at least I'd like to think so :) ) the relationship between remote config and remote_state - the latter is a data struct populated from a state file available through the former; I have just tried to follow the suggested bugreporting format and provide maximum information

Crucially, I did pass the required credentials through ARM_* env variables (mentioned in the issue), here's a censored output:

$ export | grep ARM
declare -x ARM_CLIENT_ID="censored"
declare -x ARM_CLIENT_SECRET="censored"
declare -x ARM_SUBSCRIPTION_ID="censored"
declare -x ARM_TENANT_ID="censored"

in fact, without those, the command wouldn't even start to execute

let me know of this answers your question and/or if I can be of any further help

[radeksimko]

From your initial example it looked like you're using terraform_remote_state in one of the *.tf files in a directory in which you tried executing terraform remote config which makes no sense as you would be referencing the same state you would upload.

If that's not the case and terraform_remote_state is in a different directory, separate from the directory in which you execute terrafom remote config then that's 👌 .

---

I personally have no experience with Azure, so as long as you understood the relationship between the data source & remote config command and you have all ENV variables in place, I'll leave this for someone who understands Azure and/or uses this remote backend already.

What I would normally do if I experienced similar issue with AWS is that I'd try using the exact same credentials for CRUDing some resources (i.e. configuring a provider) or try official AWS CLI to see if the credentials are _actually_ valid. I assume there's some kind of official Azure CLI too?

[mbravorus]

@radeksimko You are entirely correct, there's a CLI, I use it with the same credentials and it works, I also successfully CRUDed stuff with terraform and these credentials, before moving on to implementing remote config. That's what I meant when I wrote that I tested them :)

[scottzilla]

@mbravorus, since you're getting a 403, I suspect you're missing the Access Key to the Azure storage account itself; so in addition to the env vars you set above, just add the ARM_ACCESS_KEY and you should be in like gin:

export | grep ARM
declare -x ARM_CLIENT_ID="censored"
declare -x ARM_CLIENT_SECRET="censored"
declare -x ARM_SUBSCRIPTION_ID="censored"
declare -x ARM_TENANT_ID="censored
declare -x ARM_ACCESS_KEY="censored"

[avishnyakov]

Having similar issue while configuring remote state via CLI with 0.7.0 / 0.7.1
Wrapped into a PS script as the following:

& $terraformBinPath remote config -backend=azure `
            -backend-config="resource_group_name=$azure_resource_group_name" `
            -backend-config="storage_account_name=$azure_storage_account_name" `
            -backend-config="container_name=terraform-state" `
            -backend-config="key=$terraformKeyName" `
            -backend-config="arm_subscription_id=$azure_subscription_id" `
            -backend-config="arm_client_id=$azure_client_id" `
            -backend-config="arm_client_secret=$azure_client_secret" `
            -backend-config="arm_tenant_id=$azure_tenant_id"

Output looks close to @mbravorus original post.

Remote configuration updated
terraform.exe : Error while performing the initial pull. The error message is shown
At 
_ci_build_terraform_config.ps1:313 
char:9
+         & $terraformBinPath remote config -backend=azure `
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error whil...essage is shown:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

below. Note that remote state was properly configured, so you don't
need to reconfigure. You can now use `push` and `pull` directly.
Error reloading remote state: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server 
failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:3ef7cdb5-0001-0063-1ad7-fda872000000
Time:2016-08-24T07:18:47.4841314Z, RequestId=3ef7cdb5-0001-0063-1ad7-fda872000000, QueryParameterName=, QueryParameterValue=

push / pull give same error

[rjinski]

Similar problem. From the documentation I'd assumed I could either authenticate with the Access Key or the Resource Manager credentials however it seems as the only way to get this working is to give the Access Key (which does not require the Resource Manager credentials...)?

[rjinski]

To add to the above, the error messages are showing the wrong key name for resource_group. The error message:

missing 'resource_group' configuration

If the error message above mentions requiring or modifying configuration
options, these are set using the `-backend-config` flag. Example:
-backend-config="name=foo" to set the `name` configuration

However it is actually resource_group_name as per the documentation here.

Also when using ARM to authenticate I am getting this:

$ terraform remote config \
> -backend=azure \
> -backend-config="storage_account_name=tfstate" \
> -backend-config="container_name=tfstate" \
> -backend-config="key=terraform.tfstate" \
> -backend-config="resource_group_name=tfstate-rg" \
> -backend-config="arm_subscription_id=******" \
> -backend-config="arm_client_id=*******" \
> -backend-config='arm_client_secret=********' \
> -backend-config="arm_tenant_id=***********"

Remote configuration updated
Error while performing the initial pull. The error message is shown
below. Note that remote state was properly configured, so you don't
need to reconfigure. You can now use `push` and `pull` directly.

Error reloading remote state: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:9e482698-0001-0037-6120-13d81f000000
Time:2016-09-20T09:23:34.4048489Z, RequestId=9e482698-0001-0037-6120-13d81f000000, QueryParameterName=, QueryParameterValue=

Using the access key with no ARM credentials works as expected.

[jen20]

Hi all. The Azure backend uses only storage_account_name, container_name, key and access_key. This can be seen here. It's not possible to use ARM credentials for remote state authentication at the moment, so it is necessary to have first obtained access keys.

[rjinski]

Documentation needs to be updated to reflect this

[oblakeerickson]

The above azure remote documentation url is broken. Here is the new one:

https://www.terraform.io/docs/backends/types/azurerm.html

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.