Vault: AWS IAM Authentication without region set in config defaults to us-east-1

[AndreasFolz]

Describe the bug
Usage of Vault container version 1.2.4 breaks communication, because the identified region is us-east-1.

To Reproduce
Steps to reproduce the behavior:

1. Setup a vault docker container that should use a config file without region identified. See below for the config file
2. Try to boot the container, assuming you have already access set up for this IAM role to vault.
3. See error:

  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>
  </Error>

Expected behavior
The vault container should behave in a way where it identifies the region according to GetOrDefaultRegion
https://github.com/hashicorp/vault/blob/master/helper/awsutil/region.go#L40

Environment:

• Vault Server Version: irrelevant for the bug behavior, since no communication to the server is possible.
• Vault Agent Version: 1.2.4
• Server Operating System/Architecture: Vault agent docker container

Vault server configuration file(s):

disable_mlock = true
vault {
  address = "your-vault-url.com"
}

auto_auth {
  method "aws" {
    type = "aws"
    config = {
      type = "iam"
      role = "your-vault-role"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_disable = true
}

Additional context
The issue was introduced in this commit:
https://github.com/hashicorp/vault/pull/7632/files#diff-b9f729c0f975915bf910ecd47e67f8f7R75
Here the value is initialized and only replaced by something else if a region is actually specified in the config file.
Else it transfers the awsUtil.DefaultRegion which is us-east-1 to the GenerateLoginData with a configured region. This is then used within it: https://github.com/hashicorp/vault/blob/master/builtin/credential/aws/cli.go#L42
This contains the logic to use the default region, therefore a previous initialization with the region is not necessary.

Documentation:

1. User-provided configuration is the most explicit.
2. Environment variables are potentially shared across many invocations and so they have less precedence.
3. Configuration in `~/.aws/config` is shared across all invocations of a given user and so this has even less precedence.
4. Configuration retrieved from the EC2 instance metadata service is shared by all invocations on a given machine, and so it has the lowest precedence.

An easy fix would be an initialization with empty string instead of the default region.

A workaround would be to modify the vault file to

disable_mlock = true
vault {
  address = "your-vault-url.com"
}

auto_auth {
  method "aws" {
    type = "aws"
    config = {
      type = "iam"
      role = "your-vault-role"
      region = ""
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_disable = true
}

[kalafut]

@AndreasFolz The auto detection of region based on environment has been a source of confusion and a number of Github issues, especially in multi-region environments, since it was introduced. User who didn't configure sts_endpoint were running into CLI and Agent errors when they spanned regions, and while the CLI issues could be worked around with the region parameter, Agent was more challenging.

The decision was made to remove this behavior in 1.2.4 (see the "CHANGES" section in https://github.com/hashicorp/vault/blob/master/CHANGELOG.md#124-november-7th-2019.) If you were relying on auto-detection, it does require a config file addition. But the overall operation is simpler and more deterministic since (1) if you don't configure sts_endpoint, you don't need to specify the region in Agent or the CLI, (2) if you do configure it, you always provide that same region as configuration to Agent or the CLI.

[AndreasFolz]

@kalafut I understand why this was done, but this now removes all possibility to set the AWS_REGION as an environment variable and forces all users to add the region parameter to their file.
This is especially annoying, since it does not allow us to use the same docker image of vaultagent across multiple regions and we always have to add this in our bootstrap as opposed to just use an environment variable inside the vaultagent docker container.
Would creating a feature request for "Allow setting the AWS config from environment variables" then be a better option?

[kalafut]

@AndreasFolz I think a feature request is a good idea to capture the discussion. Having such flexibility to configure the region determination scheme (whether via a parameter, special value for region, etc.) is definitely worth consideration IMHO.

[AndreasFolz]

Okay, then I'm gonna close this issue and reopen a feature request. After that I'll link it!