Terraform: New s3 backend config not working with non AWS s3 endpoint

[trodemaster]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Terraform v0.9.0-dev (dd8a93ec8de96d81c7588fe9838b4fd8e09a12ad)

Affected Resource(s)

s3 backend

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "projectname-terraform"
    endpoint = "private-s3.intranet.company.com"
    key = "/terraform.tfstate"
    region = "region1"
    access_key = "${var.tfstate_access_key}"
    secret_key = "${var.tfstate_secret_key}"
  }
}

Debug Output

https://git.corp.adobe.com/gist/bgarner/7afe59ffaec18b1d7f084955c3225fd4

Expected Behavior

Prompt to init new backend and after successfully doing that connect to s3 bucket and show no state configured.

Actual Behavior

Terraform is unable to connect to the s3 bucket. Seems like the endpoint specified is not being used and default aws endpoint does not recognize my keys.

```$ terraform init
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your environment. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform show
Failed to load state: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.
status code: 403, request id: 87f1ba6a-f1ae-1a99-b7b3-a0369f327a26

### Steps to Reproduce
Setup a custom s3 backend and configure it per https://github.com/hashicorp/terraform/blob/master/website/source/docs/backends/types/s3.html.md

init the backend and do a terraform show

### Important Factoids
We are using an on premises s3 server that is 100% api compatible. The previous syntax worked without issue. 

```hcl
data "terraform_remote_state" "phantom" {
    backend = "s3"
    config {
        bucket = "itcloud-terraform"
        endpoint = "s3-sj1.corp.adobe.com"
        key = "/terraform.tfstate"
        region = "region1"
        access_key = "${var.tfstate_access_key}"
        secret_key = "${var.tfstate_secret_key}"
    }
}

[mitchellh]

Hm, this code actually hasn't changed at all (zero diff between 0.9 and 0.8.8). "s3" hasn't been converted to the new backend style so its actually just pass-through to the old codebase.

I believe the issue is that you have an interpolation in your backend config. Interpolations do not work. I added validation in beta 2 but I think I actually forgot to call it. 😢 Anyways, that's the issue here!

[roidelapluie]

I can reproduce this. endpoint not working.

terraform {
  backend "s3" {
    bucket = "terraform"
    key    = "/foo/terraform.tfstate"
    region = "us-east-1"
    access_key = "55KB0E9NJ1O0NWRW52Q4"
    secret_key = "5i8QySbzJESEr3/ALMPXwgqjEno6lwh2lQjv7kQU"
    endpoint = "http://127.0.0.1:9000/"
  }
}

strace: Process 17302 attached
strace: Process 17303 attached
strace: Process 17304 attached
strace: Process 17305 attached
[pid 17301] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 17301] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 17301] setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 17301] bind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
[pid 17301] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 17301] setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 17301] bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
strace: Process 17306 attached
strace: Process 17307 attached
strace: Process 17308 attached
strace: Process 17309 attached
strace: Process 17310 attached
strace: Process 17311 attached
strace: Process 17312 attached
strace: Process 17313 attached
strace: Process 17314 attached
strace: Process 17315 attached
[pid 17311] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
[pid 17311] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 6
[pid 17311] setsockopt(6, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 17311] bind(6, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
[pid 17311] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 7
[pid 17311] setsockopt(7, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 17311] bind(7, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
strace: Process 17316 attached
strace: Process 17317 attached
strace: Process 17318 attached
strace: Process 17319 attached
strace: Process 17320 attached
strace: Process 17321 attached
strace: Process 17322 attached
[pid 17319] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6
[pid 17319] setsockopt(6, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17319] connect(6, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 17311] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 8
[pid 17311] setsockopt(8, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17311] connect(8, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 17314] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 9
[pid 17314] setsockopt(9, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17314] connect(9, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 17319] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 10
[pid 17319] setsockopt(10, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17319] connect(10, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 17317] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
[pid 17317] setsockopt(11, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17311] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
[pid 17317] connect(11, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...>
[pid 17311] <... socket resumed> )      = 12
[pid 17317] <... connect resumed> )     = 0
[pid 17311] setsockopt(12, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 17317] getsockname(11, {sa_family=AF_INET, sin_port=htons(38325), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 17317] getpeername(11, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 17311] <... setsockopt resumed> )  = 0
[pid 17311] connect(12, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
[pid 17311] getsockname(12, {sa_family=AF_INET, sin_port=htons(54432), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 17311] getpeername(12, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 17316] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
[pid 17316] setsockopt(11, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 17316] connect(11, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("205.251.243.54")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 17311] getsockopt(11, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
[pid 17311] getsockname(11, {sa_family=AF_INET, sin_port=htons(35174), sin_addr=inet_addr("14.10.69.68")}, [16]) = 0
[pid 17311] getpeername(11, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("205.251.243.54")}, [16]) = 0
[pid 17311] setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 17311] setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 17311] setsockopt(11, SOL_TCP, TCP_KEEPINTVL, [30], 4) = 0
[pid 17311] setsockopt(11, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0

[pid 17314] +++ exited with 1 +++
[pid 17318] +++ exited with 1 +++
[pid 17317] +++ exited with 1 +++
[pid 17320] +++ exited with 1 +++
[pid 17316] +++ exited with 1 +++
[pid 17315] +++ exited with 1 +++
[pid 17313] +++ exited with 1 +++
[pid 17312] +++ exited with 1 +++
Error configuring the backend "s3": InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 9453526b-239f-11e7-984f-f3941fb2424e

Please update the configuration in your Terraform files to fix this error
then run this command again.

[pid 17319] +++ exited with 1 +++
[pid 17311] +++ exited with 1 +++
[pid 17301] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=17311, si_uid=1000, si_status=1, si_utime=15, si_stime=2} ---
[pid 17321] +++ exited with 1 +++
[pid 17310] +++ exited with 1 +++
[pid 17309] +++ exited with 1 +++
[pid 17308] +++ exited with 1 +++
[pid 17307] +++ exited with 1 +++
[pid 17306] +++ exited with 1 +++
[pid 17305] +++ exited with 1 +++
[pid 17304] +++ exited with 1 +++
[pid 17303] +++ exited with 1 +++
[pid 17302] +++ exited with 1 +++
[pid 17322] +++ exited with 1 +++
+++ exited with 1 +++

[jolbax]

I do have the same problem. Working with Terraform v0.9.5
It worked as well with the v8.* and the old syntax.

Is there any solution?
Please help!

terraform {
  backend "s3" {
    endpoint = "https://ch.s3objectstore.com"
    bucket = "terraform"
    key = "mypaht/terraform.tfstate"
    region = "ch"
    access_key = "###########"
    secret_key = "###############"
  }
}

terraform show                                                     
Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

Failed to load backend: Initialization required. Please see the error message above.

[vovandodev]

Did you fix it? I have the same one.

[jolbax]

Unfortunately not. Still looking for a fix, workaround or advice

[bonifaido]

I have pushed a fix for this last week! ^^

[MrKoopaKiller]

Same here.
In my case i'm using environment variables to run terraform init

provider "aws" {
  region  = "${var.region}"
}

terraform {
  backend "s3" {
    bucket = "myRemoteBucket"
    encrypt = "true"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

I got this error when i ran terraform init:

Error configuring the backend "s3": InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: fc4f9fce-6dee-11e7-1234-317ca78aass7

Please update the configuration in your Terraform files to fix this error
then run this command again.

[jolbax]

@mitchellh
Any news if it is working in the new version 0.10.2?

[bonifaido]

There is fix already in a PR: https://github.com/hashicorp/terraform/pull/15553, it needs review.

[bonifaido]

How can I request a review here on the aforementioned PR?

[trodemaster]

FYI a fix for this has been merged into master recently. I have successfully tested our cloudian s3 service as a backend. See this issue for details. https://github.com/hashicorp/terraform/pull/15553#issuecomment-334724247

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.