Terraform-provider-vault: Auth Login on Performance Standby Nodes

[in0rdr]

Can the auth_login method with userpass or AppRole authentication backend be used in combination with a loadbalancer health check perfstandbyok=true?

I set up a Haproxy loadbalancer according to "Using External Load Balancer", but with perfstandbyok=true, to make use of the (ent) perf. standby nodes. The Vault is configured with api_addr=<node-addr> (not loadbalancer addr).The problem is that requests with the TF provider Vault to the loadbalancer fail with error msg cannot write to readonly storage. This message appears, when the VAULT_ADDR/address points to a passive node. The TF plan is successful, when issued against the active Vault backend node directly. Finding the Vault leader first (before running the TF provider) increases the complexity of the pipeline. Also, I'd rather not maintain a dedicated loadbalancer frontend/backend (without perfstandbyok=true) just for TF runs.

The same loadbalancer config works fine when authenticating with the env variable VAULT_TOKEN directly (empty TF provider config section, see "Terraform Configuration Files" section below).

(Terraform) Version

tf: v0.12.28
provider: vault_v2.12.2_x4
vault: 1.5.0+ent

Affected Resource(s)

Provider login auth_login with userpass or AppRole auth backend

Terraform Configuration Files

Configuration example to test the failed login at perf standby nodes with a userpass backend:

provider "vault" {
  auth_login {
    path = "auth/userpass/login/user1"

    parameters = { 
      password = "<PASS_FOR_USER1>"
    }   
  }
}

Configuration example to test the failed login at perf standby nodes with an AppRole backend:

provider "vault" {
  auth_login {
    path = "auth/approle/login"

    parameters = { 
      role_id   = "0cea74ba-71a0-6165-5197-636e0b750c32"
      secret_id = "<SECRET_ID>"
    }   
  }
}

Configuration example with externall VAULT_TOKEN env var to test the successful login at perf standby nodes:

provider "vault" {}

Debug Output

The plan fails with VAULT_ADDR=https://<LOADBALANCER_ADDR> due to unsuccessful login (ip redacted):

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


Error: Error making API request.

URL: GET https://<LOADBALANCER_ADDR>/v1/auth/token/lookup-self
Code: 400. Errors:

* error performing token check: failed to persist lease entry: cannot write to readonly storage

  on terraform.tf line 1, in provider "vault":
   1: provider "vault" {

Expected Behavior

The plan should succeed when VAULT_ADDR=https://<LOADBALANCER_ADDR> is set, where LOADBALANCER_ADDR is the address of the loadbalancer.

Actual Behavior

The plan fails with error message cannot write to readonly storage, see debug output above.

Steps to Reproduce

1. Setup a Vault (ent) cluster with 3 nodes using Raft integrated storage
2. Create a Haproxy loadbalancer in front of the cluster, use perfstandbyok=true for the health check, in order to leverage the ent performance standby. Alternatively, if setting up a loadbalancer is too cumbersome, run step (4) against one of the standby nodes directly. The behavior should be the same, as explain in the beginning of this report.
3. Create a simple Terraform configuration as outlined in section "Terraform Configuration Files". Use the first snippet that includes login via "userpass" method and auth_login.
4. Run terraform plan

To reproduce the local env (sensitive values redacted):

$ printenv | grep -i vault
VAULT_TOKEN=<VAULT_TOKEN_HERE>
VAULT_ADDR=https://<LOADBALANCER_ADDR>
VAULT_SKIP_VERIFY=true

Additional information to reproduce the positive/negative test cases as follows.

To test the failing behavior, make sure the VAULT_ADDR points to either:

• the loadbalancer with health check perfstandbyok=true
• or the address of a perf standby Vault node

To test the case, where the error cannot be reproduced, the VAULT_ADDR points either to:

• a loadbalancer without perfstandbyok=true in the health check URI
• or an active Vault node
• or the auth_login is disabled altogether, as shown in the snippet above in section "Terraform Configuration Files"

Important Factoids

• The Vault storage backend is Raft(integrated).
• The loadbalancer is configured with perfstandbyok=true health check.
• The login succeeds when the loadbalancer is configured with perfstandbyok=false (default), because requests are consistently routed to the Vault leader node.
• The Vault is configured with api_addr=<node-addr> (not the loadbalancer addr).
• It seems like this applies to the auth_login configuration in general, because "userpass" and "approle" auth login methods were tested positively for this behavior.

References

• https://registry.terraform.io/providers/hashicorp/vault/latest/docs#example-auth_login-usage
• https://www.vaultproject.io/api-docs/system/health
• https://learn.hashicorp.com/tutorials/vault/reference-architecture#load-balancing-using-external-load-balancer

[dedamico]

FYI, there was a fix (not on the provider) included in Vault 1.5.4 Enterprise which helps prevent this error from occurring. @in0rdr can you confirm if you still see this issue running against a Vault server on 1.5.4?

[in0rdr]

@dedamico I cannot reproduce the issue anymore running Vault 1.5.4+ent. Thank you for fixing this issue.

[hugespoon]

@dedamico We are running into this issue with Vault 1.5.5+prem currently. We see the same behavior with regard to perfstandbyok=true in our healthchecks. Our setup is essentially identical to what has been described by @in0rdr. We seem to hit it fairly regularly even under 0 load on the cluster.

[dedamico]

Thanks @hugespoon, we have heard other reports of users still seeing this error even after moving to 1.5.4/1.5.5. Can you send details regarding your disk characteristics on your Vault integrated storage nodes (SSD yes/no, configured IOPS, etc)?

[hugespoon]

We are using 50GB pd-ssd disks on GCP. So that puts us at max sustained read/write IOPS of ~1500 (if I'm reading https://cloud.google.com/compute/docs/disks#pdspecs correctly). Is it recommended to use local SSD disks instead?

[dedamico]

Thanks @hugespoon this is really helpful. Overall with Integrated Storage, the better the iOPS the better Vault will perform. ~1500 is pretty low from my understanding with respect to Vault with Integrated Storage.

In any case, it would be very useful as a data point to see if you can increase the IOPS to 5k to 10k (or higher) if you still see the issue, OR to find out at what point the increased IOPS leads to this error going away.

Our reference architecture doc speaks to the SSD/disk performance at a high level:

However, when using integrated storage the infrastructure should have a relatively high-performance hard disk subsystem, hence the non-burstable SSD requirement. If many secrets are being generated or rotated frequently, this information will need to flush to disk often and can impact performance if slower hard drives are used.