Vault: http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request

[innovia]

• Vault Version: 0.9.3
• Operating System/Architecture: Kubernetes 1.8.6 ->Debian Jessi Docker

Vault Config File:

{
  "ha_storage": {
    "consul": {
      "address": "consul.default.svc.cluster.local:8443",
      "path": "vault",
      "scheme": "https",
      "tls_ca_file": "/etc/tls/ca.pem",
      "tls_cert_file": "/etc/tls/consul.pem",
      "tls_key_file": "/etc/tls/consul-key.pem"
    }
  },
  "listener": {
    "tcp": {
      "address": "[::]:8200",
      "tls_cert_file": "/etc/tls/consul.pem",
      "tls_client_ca_file": "/etc/tls/ca.pem",
      "tls_disable": 0,
      "tls_key_file": "/etc/tls/consul-key.pem"
    }
  },
  "storage": {
    "consul": {
      "address": "consul.default.svc.cluster.local:8443",
      "path": "vault",
      "scheme": "https",
      "tls_ca_file": "/etc/tls/ca.pem",
      "tls_cert_file": "/etc/tls/consul.pem",
      "tls_key_file": "/etc/tls/consul-key.pem"
    }
  }
}

Startup Log Output:

+ vault-5db7776dc6-6fptt › vault
+ vault-5db7776dc6-z8bxj › vault
+ vault-5db7776dc6-lltnn › vault
vault-5db7776dc6-z8bxj vault ==> Vault server configuration:
vault-5db7776dc6-z8bxj vault
vault-5db7776dc6-z8bxj vault               HA Storage: consul
vault-5db7776dc6-z8bxj vault                      Cgo: disabled
vault-5db7776dc6-z8bxj vault          Cluster Address: https://vault.default.svc.cluster.local:8201
vault-5db7776dc6-z8bxj vault               Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "enabled")
vault-5db7776dc6-z8bxj vault                Log Level: info
vault-5db7776dc6-z8bxj vault                    Mlock: supported: true, enabled: true
vault-5db7776dc6-z8bxj vault         Redirect Address: https://vault.default.svc.cluster.local:8200
vault-5db7776dc6-z8bxj vault                  Storage: consul
vault-5db7776dc6-z8bxj vault                  Version: Vault v0.9.3
vault-5db7776dc6-z8bxj vault              Version Sha: 5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d
vault-5db7776dc6-z8bxj vault
vault-5db7776dc6-z8bxj vault 2018/01/30 15:04:14.795739 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-z8bxj vault 2018/01/30 15:04:14.796389 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-z8bxj vault ==> Vault server started! Log data will stream in below:
vault-5db7776dc6-z8bxj vault
vault-5db7776dc6-z8bxj vault 2018/01/30 16:31:20.737792 [INFO ] core: vault is unsealed
vault-5db7776dc6-z8bxj vault 2018/01/30 16:31:20.737902 [INFO ] core: entering standby mode
vault-5db7776dc6-z8bxj vault 2018/02/01 13:20:13.875950 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-z8bxj vault 2018/02/01 13:20:13.875984 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault ==> Vault server configuration:
vault-5db7776dc6-6fptt vault
vault-5db7776dc6-6fptt vault               HA Storage: consul
vault-5db7776dc6-6fptt vault                      Cgo: disabled
vault-5db7776dc6-6fptt vault          Cluster Address: https://vault.default.svc.cluster.local:8201
vault-5db7776dc6-6fptt vault               Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "enabled")
vault-5db7776dc6-6fptt vault                Log Level: info
vault-5db7776dc6-6fptt vault                    Mlock: supported: true, enabled: true
vault-5db7776dc6-6fptt vault         Redirect Address: https://vault.default.svc.cluster.local:8200
vault-5db7776dc6-6fptt vault                  Storage: consul
vault-5db7776dc6-6fptt vault                  Version: Vault v0.9.3
vault-5db7776dc6-6fptt vault              Version Sha: 5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d
vault-5db7776dc6-6fptt vault
vault-5db7776dc6-6fptt vault ==> Vault server started! Log data will stream in below:
vault-5db7776dc6-6fptt vault
vault-5db7776dc6-6fptt vault 2018/01/30 15:04:14.775580 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-6fptt vault 2018/01/30 15:04:14.776211 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-6fptt vault 2018/01/30 15:07:39.225352 [INFO ] core: vault is unsealed
vault-5db7776dc6-6fptt vault 2018/01/30 15:07:39.225521 [INFO ] core: entering standby mode
vault-5db7776dc6-6fptt vault 2018/02/01 13:23:09.398926 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:23:09.398966 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:25:04.639965 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:25:04.640000 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:26:57.608010 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:26:57.608047 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.886295 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.886329 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.891254 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.891276 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.895107 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.895130 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.901323 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.901346 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.905099 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.905120 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.908630 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.908664 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.912525 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.912548 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.916298 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.916320 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.919932 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.919955 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.923957 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.923979 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.927451 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.927472 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.931151 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.931173 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.939683 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.939814 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.945415 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.945438 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.948938 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.948961 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.952483 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.952505 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.956106 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.956128 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.959601 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.959623 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.963310 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.963332 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.966858 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.966880 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.970343 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.970368 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.973939 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.973961 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.977750 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.977772 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.981248 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.981273 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.984706 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.984728 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.988321 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.988343 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.992059 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.992082 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.995642 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
vault-5db7776dc6-lltnn vault ==> Vault server configuration:
vault-5db7776dc6-lltnn vault
vault-5db7776dc6-lltnn vault               HA Storage: consul
vault-5db7776dc6-lltnn vault                      Cgo: disabled
vault-5db7776dc6-lltnn vault          Cluster Address: https://vault.default.svc.cluster.local:8201
vault-5db7776dc6-lltnn vault               Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "enabled")
vault-5db7776dc6-6fptt vault 2018/02/01 13:29:04.995663 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault-5db7776dc6-lltnn vault                Log Level: info
vault-5db7776dc6-lltnn vault                    Mlock: supported: true, enabled: true
vault-5db7776dc6-lltnn vault         Redirect Address: https://vault.default.svc.cluster.local:8200
vault-5db7776dc6-lltnn vault                  Storage: consul
vault-5db7776dc6-lltnn vault                  Version: Vault v0.9.3
vault-5db7776dc6-lltnn vault              Version Sha: 5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d
vault-5db7776dc6-lltnn vault
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:15.020569 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:15.021161 [WARN ] physical/consul: appending trailing forward slash to path
vault-5db7776dc6-lltnn vault ==> Vault server started! Log data will stream in below:
vault-5db7776dc6-lltnn vault
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.388878 [INFO ] core: vault is unsealed
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.388959 [INFO ] core: entering standby mode
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.404984 [INFO ] core: acquired lock, enabling active operation
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.440617 [INFO ] core: post-unseal setup starting
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.442197 [INFO ] core: loaded wrapping token key
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.442313 [INFO ] core: successfully setup plugin catalog: plugin-directory=
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.445662 [INFO ] core: successfully mounted backend: type=kv path=secret/
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.445824 [INFO ] core: successfully mounted backend: type=system path=sys/
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.446145 [INFO ] core: successfully mounted backend: type=identity path=identity/
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.446177 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.452359 [INFO ] expiration: restoring leases
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.452538 [INFO ] rollback: starting rollback manager
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.457896 [INFO ] identity: entities restored
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.458944 [INFO ] identity: groups restored
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.460482 [INFO ] core: post-unseal setup complete
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.460628 [INFO ] core/startClusterListener: starting listener: listener_address=[::]:8201
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.460733 [INFO ] core/startClusterListener: serving cluster requests: cluster_listen_address=[::]:8201
vault-5db7776dc6-lltnn vault 2018/01/30 15:04:38.464990 [INFO ] expiration: lease restore complete

Expected Behavior:
vault should forward on standby to active node.

Actual Behavior:
transit failure on rpc rquests.

Steps to Reproduce:

Important Factoids:

References:

[chrishoffman]

This error most likely means that the standby node cannot connect to the active node. Is communication allowed over port 8201 between your standby and active. Can you also provide the configuration for your active node?

[innovia]

Thanks @chrishoffman for that pointer!

i've opened up the port in the service and the pod and it seems like its all working now.