Terraform-provider-google: backend "gcs" and GCE in different projects.

[movergan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

terraform -v
Terraform v0.11.10

• provider.google v1.19.1
• provider.google-beta v1.19.0

Affected Resource(s)

• terraform backend "gcs"

Terraform Configuration Files

terraform {
  backend "gcs" {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials = "craedentials.json"

  }
}

data "terraform_remote_state" "d001" {
  backend = "gcs"
  config {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
  }
}

Debug Output

terraform init
2018/10/26 11:30:56 [INFO] Terraform version: 0.11.10 17850e9a55d33c43d7c31fd6ac122ba97a51d899
2018/10/26 11:30:56 [INFO] Go runtime version: go1.11.1
2018/10/26 11:30:56 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}
2018/10/26 11:30:56 [DEBUG] Attempting to open CLI config file: /home/alex/.terraformrc
2018/10/26 11:30:56 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/10/26 11:30:56 [INFO] CLI command args: []string{"init"}
2018/10/26 11:30:56 [DEBUG] command: loading backend config file: /home/alex/development/terraform/terraform/dev/d001
Initializing modules...

• module.x001wadc00

Initializing the backend...
2018/10/26 11:30:56 [WARN] command: backend config change! saved: 7348461056038369515, new: 3247140105523421037
Backend configuration changed!

2018/10/26 11:30:56 [DEBUG] plugin: waiting for all plugin processes to complete...
Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Error initializing new backend:
Error configuring the backend "gcs": Error parsing credentials 'craedentials.json': invalid character 'c' looking for beginning of value

Please update the configuration in your Terraform files to fix this error
then run this command again.

Expected Behavior

According to the documentation

credentials / GOOGLE_CREDENTIALS - (Optional) Local path to Google Cloud Platform account credentials in JSON format. If unset, Google Application Default Credentials are used. The provided credentials need to have the devstorage.read_write scope and WRITER permissions on the bucket.

Terraform should use file with credentials. If I specify path it fails with the error I provided.

If I change configuration to:

terraform {
  backend "gcs" {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials="${file("credentials.json")}"

  }
}

It fails because I can't use interpolations in backend:

terraform init
2018/10/26 11:37:16 [INFO] Terraform version: 0.11.10  17850e9a55d33c43d7c31fd6ac122ba97a51d899
2018/10/26 11:37:16 [INFO] Go runtime version: go1.11.1
2018/10/26 11:37:16 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}
2018/10/26 11:37:16 [DEBUG] Attempting to open CLI config file: /home/alex/.terraformrc
2018/10/26 11:37:16 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/10/26 11:37:16 [INFO] CLI command args: []string{"init"}
2018/10/26 11:37:16 [DEBUG] command: loading backend config file: /home/alex/development/terraform/terraform/dev/d001
Initializing modules...
- module.x001wadc00

2018/10/26 11:37:16 [DEBUG] plugin: waiting for all plugin processes to complete...
Initializing the backend...
Error loading backend config: 1 error(s) occurred:

* terraform.backend: configuration cannot contain interpolations

The backend configuration is loaded by Terraform extremely early, before
the core of Terraform can be initialized. This is necessary because the backend
dictates the behavior of that core. The core is what handles interpolation
processing. Because of this, interpolations cannot be used in backend
configuration.

If you'd like to parameterize backend configuration, we recommend using
partial configuration with the "-backend-config" flag to "terraform init".

Using environment variable is not an option because it has higher priority and GCE for instance then uses this account too, instead the one specified in its' config. It is not acceptable when Google Storage and CGE are in different projects with different Service accounts.

Actual Behavior

Error configuring the backend "gcs": Error parsing credentials 'craedentials.json': invalid character 'c' looking for beginning of value

Steps to Reproduce

1. terraform init

Important Factoids

Workaround here is using env var:
export GOOGLE_APPLICATION_CREDENTIALS=credentials.json
but in only works if GCE and Storage for Backend are in the same project and can be accessed via the same service account.

References

• #0000

[chrisst]

You don't need to use interpolation or file(*) for the credentials file. Just pass the file path in as a string and terraform will load it correctly.

[danawillow]

Closing this since it sounds like @chrisst gave a solution, but please let us know if that doesn't work!

[movergan]

I am sorry for late response.
As you can see from my issue description I am passing credentials just like a string:

terraform {
  backend "gcs" {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials = "craedentials.json"

  }
}

And the error I am getting is:

Error initializing new backend:
Error configuring the backend "gcs": Error parsing credentials 'craedentials.json': invalid character 'c' looking for beginning of value

That is the reason I opened this issue.

[movergan]

@danawillow can we reopen it please?

[chrisst]

@movergan I was able to create a remote backend using my local credentials. I'm going to take a guess here and is it possible that you have misspelled craedentials.json ? Especially if GOOGLE_APPLICATION_CREDENTIALS=credentials.json seems to work for you and that is spelled differently.

[movergan]

I am sorry for typo. I've checked again.

1. file with credentials:

ll credentials.json
-rwxrwxrwx 1 alex alex 2316 Oct 31 00:30 credentials.json*

1. Backend config:

terraform {
  backend "gcs" {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials= "credentials.json"

  }
}

data "terraform_remote_state" "d001" {
  backend = "gcs"
  config {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
  }
}

Terraform init works fine:

terraform init
Initializing modules...
- module.x001wadc00

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 1.19"
* provider.google-beta: version = "~> 1.19"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform plan fails:

# terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.d001: Refreshing state...

Error: Error refreshing state: 2 error(s) occurred:

* provider.google: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
* data.terraform_remote_state.d001: 1 error(s) occurred:

* data.terraform_remote_state.d001: data.terraform_remote_state.d001: error initializing backend: storage.NewClient() failed: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

[chrisst]

It looks like you have gotten passed the first error and are now trying to refresh an existing state from the local google provider, which I'm guessing means you don't have your credentials specified in the provider block. Something like:

provider "google" {
  credentials = "${file(var.credentials_path)}"
  project     = "chrisst-test"
  region      = "us-west1"
  zone        = "us-west1-a"
}

Whether that is the issue or not, I'm not sure but I was able to successfully create remote state in a GCS bucket with a bare bones config. I recommend that you start with an empty directory, and a brand new config file with only the backend specified and try to get that working first.

[movergan]

Tried to do everything form scratch:

1. Removed folder in Google Storage
2. Removed .terraform folder
3. Ran terraform init, folder in Google got created with state file in it:

terraform init
Initializing modules...
- module.x001wadc00
  Getting source "../../terraform-sb1-modules/gce-vm-windows"

Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "google-beta" (1.19.0)...
- Downloading plugin for provider "google" (1.19.1)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 1.19"
* provider.google-beta: version = "~> 1.19"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

1. Ran terraform plan, it failed with the same error:

terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.d001: Refreshing state...

Error: Error refreshing state: 1 error(s) occurred:

* data.terraform_remote_state.d001: 1 error(s) occurred:

* data.terraform_remote_state.d001: data.terraform_remote_state.d001: error initializing backend: storage.NewClient() failed: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Provider config is below if it's important:

provider "google-beta" {
  credentials = "${file("credentials-provider.json")}"
  project     = "${var.project}"
  region      = "europe-west2"
}

Cred files:

ll credentials*
-rwxrwxrwx 1 alex alex 2316 Oct 31 00:30 credentials.json*
-rwxrwxrwx 1 alex alex   61 Oct 25 16:08 credentials-provider.json*

[chrisst]

Based on the version output it looks like you are using both the google and google-beta providers. Because Terraform uses string prefixing to determine what the provider is it will default to google for any google resource even though you have specified a config block for the google-beta provider. So if you are hoping to use google-beta you will need to specify `provider = "google-beta" for all of the resources or data-sources you are using.
eg:

data "terraform_remote_state" {
    provider = "google-beta"
    ...
}

Or you could just duplicate the provider block so you can mix and match:

provider "google" {
  credentials = "${file("credentials-provider.json")}"
  ...
}
provider "google-beta" {
  credentials = "${file("credentials-provider.json")}"
  ...
}

[movergan]

Hello,

Thank you for your update. In order to remove additional complications i switched to "google" provider.

$ terraform init
Initializing modules...
- module.x001wadc00

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 1.19"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
alex@ALEX:~/development/terraform/terraform/support/x001-deploy$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.dc: Refreshing state...
google_compute_address.default[0]: Refreshing state... (ID: altenar-ps01/europe-west1/x001-wadc-0)
google_compute_address.default[1]: Refreshing state... (ID: altenar-ps01/europe-west1/x001-wadc-1)
google_compute_instance.windows[0]: Refreshing state... (ID: x001wadc000)
google_compute_instance.windows[1]: Refreshing state... (ID: x001wadc001)

Error: Error refreshing state: 1 error(s) occurred:

* data.terraform_remote_state.dc: 1 error(s) occurred:

* data.terraform_remote_state.dc: data.terraform_remote_state.dc: error initializing backend: storage.NewClient() failed: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

It didn't change anything. Is there anything else i need to check.

[chrisst]

I took a look back at the original post and it looks like you are missing credentials in your data "terraform_remote_state" block as well. All GCP resources and datasources will inherit from the google provider block but remote state is a native terraform feature so you'll need to inject the credentials into that as well.

[movergan]

Aha! Looks like that made the trick! Thank you very much!
For people who don't want to read the entire post, conclusion - add credentials option into both blocks:

terraform {
  backend "gcs" {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials = "credentials.json"

  }
}

data "terraform_remote_state" "d001" {
  backend = "gcs"
  config {
    project = "myproject"
    bucket  = "terraform-state"
    prefix  = "terraform/state/dev/d001"
    credentials = "credentials.json"
  }
}

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!