Vault: Vault unable to read Consul storage after implementing TLS Unable to read storage migration status.

Created on 13 Jan 2020 · 11Comments · Source: hashicorp/vault

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Run vault write ...
Run vault login....
See error

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

Vault Server Version (retrieve with vault status):
Vault CLI Version (retrieve with vault version):
Server Operating System/Architecture:

Vault server configuration file(s):

# Paste your Vault config here.
# Be sure to scrub any sensitive values

Additional context
Add any other context about the problem here.

bug corconfig

Source

rckjacobs1

All 11 comments

Describe the bug
Once TLS is enabled between Vault and the Consul backend, Vault generates the following
message on startup
WARNING! Unable to read storage migration status.

To Reproduce
Configure TLS communuication between Vault and Consul
Vault Configuration
{
"storage": {
"consul": {
"address": "192.168.47.180:8500",
"scheme": "https",
"path": "vault/",
"tls_skip_verify": 1,
"cluster_addr": "http://192.168.47.180:8201",
"tls_ca_file": "/etc/vault.d/consul-agent-ca.pem",
"tls_cert_file": "/etc/vault.d/dc1-server-consul-0.pem",
"tls_key_file": "/etc/vault.d/dc1-server-consul-0-key.pem"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": 0,
"tls_cert_file":"/etc/vault.d/vault-issued-cert-bundle.pem",
"tls_key_file":"/etc/vault.d/vault-issued-private_key.pem"
}
},
"ui": true
"log_level'; "Debug"
}

Consul configuration
{
"datacenter":"dc1",
"data_dir":"/etc/consul.d/data",
"encrypt":"zfJ0PyLW8h5hpmuxACIXxw==",
"log_level":"INFO",
"enable_debug":true,
"node_name":"node1",
"advertise_addr":"192.168.47.180",
"server":true,
"ui":true,
"bootstrap": true,
"leave_on_terminate":false,
"skip_leave_on_interrupt":true,
"rejoin_after_leave":true,
"retry_join":[
"localhost"
],
"enable_agent_tls_for_checks": false,
"verify_incoming": false,
"verify_outgoing": false,
"verify_server_hostname": false,
"ca_file": "/etc/consul.d/consul-agent-ca.pem",
"cert_file": "/etc/consul.d/dc1-server-consul-0.pem",
"key_file": "/etc/consul.d/dc1-server-consul-0-key.pem",
"ports": {
"dns": 8600,
"http": -1,
"https": 8500
},

 "addresses": {
     "http": "0.0.0.0",
     "dns": "0.0.0.0"
 }

}

Expected behavior

The expected behavior is vault will communicate with consul backend on startup
Problem: Vault receives the following error when attempting to communication with consul backend
WARNING! Unable to read storage migration status.
The expectation is that Vault will communicate with consul using TLS
NOTE: The vault to consul communication works fine WITHOUT TLS

Environment:

Vault Server Version Vault v1.2.3
Vault CLI Version Vault v1.2.3
Server Operating System/Architecture:
ubuntu 4.15.0-1051-aws
Vault server configuration
{
"storage": {
"consul": {
"address": "192.168.47.180:8500",
"scheme": "https",
"path": "vault/",
"tls_skip_verify": 1,
"cluster_addr": "http://192.168.47.180:8201",
"tls_ca_file": "/etc/vault.d/consul-agent-ca.pem",
"tls_cert_file": "/etc/vault.d/dc1-server-consul-0.pem",
"tls_key_file": "/etc/vault.d/dc1-server-consul-0-key.pem"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": 0,
"tls_cert_file":"/etc/vault.d/vault-issued-cert-bundle.pem",
"tls_key_file":"/etc/vault.d/vault-issued-private_key.pem"
}
},
"ui": true
"log_level'; "Debug"
}

rckjacobs1 on 13 Jan 2020

Vault server configuration
{
"storage": {
"consul": {
"address": "192.168.47.180:8500",
"scheme": "https",
"path": "vault/",
"tls_skip_verify": 1,
"cluster_addr": "http://192.168.47.180:8201",
"tls_ca_file": "/etc/vault.d/consul-agent-ca.pem",
"tls_cert_file": "/etc/vault.d/dc1-server-consul-0.pem",
"tls_key_file": "/etc/vault.d/dc1-server-consul-0-key.pem"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": 0,
"tls_cert_file":"/etc/vault.d/vault-issued-cert-bundle.pem",
"tls_key_file":"/etc/vault.d/vault-issued-private_key.pem"
}
},
"ui": true
"log_level'; "Debug"
}

rckjacobs1 on 13 Jan 2020

vault.hcl.txt

rckjacobs1 on 13 Jan 2020

Is there a working example of vault utilizing consul backend with TLS enabled?

rckjacobs1 on 13 Jan 2020

Hello - would it be possible share more information regarding the error? Are there any other parts in the log or around the error that you could share without disclosing any secret information? It would be helpful to see if there are any network related errors in the logs.

The only thing that I see at this time in the configuration you shared is cluster_addr is included in the consul block, where cluster_addr should be a top-level configuration setting. I'm skeptical if that's part of the issue though.

For examples, you can check out the Vault deployment guide and the Vault High Availability with Consul guide.

catsby on 13 Jan 2020

WARNING! Unable to read storage migration status.
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config path set: path=vault/
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config service set: service=vault
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config service_tags set: service_tags=
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config service_address set: service_address=
2020-01-14T21:59:58.064Z [DEBUG] storage.consul: config address set: address=192.168.0.41:8500
2020-01-14T21:59:58.068Z [WARN] storage migration check error: error="Unexpected response code: 400"
2020-01-14T22:00:00.073Z [WARN] storage migration check error: error="Unexpected response cod

rckjacobs1 on 16 Jan 2020

It also looks like you have TLS enabled for Vault's listener, but the cluster_addr is using an http://... address instead of https://... one.

calvn on 16 Jan 2020

Forcing the https session to use 8500 in consul config
consul-server.hcl
{
"datacenter":"vaultdc",
"data_dir":"/data",
"encrypt": "",
"log_level":"DEBUG",
"enable_debug":true,
"node_name":"RichVault3",
"advertise_addr":"192.168.0.232",
"server":true,
"ui":true,
"bootstrap":false,
"start_join": ["192.168.0.92", "192.168.0.41"],
"leave_on_terminate":false,
"skip_leave_on_interrupt":true,
"rejoin_after_leave":true,
"enable_agent_tls_for_checks": false,
"verify_incoming": false,
"verify_outgoing": false,
"verify_server_hostname": false,
"ca_file": "/config/consul-agent-ca.pem",
"cert_file": "/config/dc1-server-consul-0.pem",
"key_file": "/config/dc1-server-consul-0-key.pem",
"ports": {
"dns": 8600,
"http": -1,
"https": 8500
},

 "addresses": {
     "http": "0.0.0.0",
     "https": "0.0.0.0",
     "dns": "0.0.0.0"
 }

}

vault.hcl
{
"storage": {
"consul": {
"address": "192.168.0.41:8500",
"path": "vault/",
"tls_skip_verify": 1,
"cluster_addr": "http://192.168.47.180:8201",
"tls_ca_file": "/config/consul-agent-ca.pem",
"tls_cert_file": "/config/dc1-server-consul-0.pem",
"tls_key_file": "/config/dc1-server-consul-0-key.pem"

                  }
    },
    "listener": {
            "tcp": {
                    "address": "0.0.0.0:8200",
                    "tls_disable": 1,
                    "tls_cert_file":"/key-material/vault-bundle-2020.pem",
                    "tls_key_file":"/key-material/vault-private_key-2020.pem"
            }
    },
    "ui": true

}

rckjacobs1 on 16 Jan 2020

👍1

Hi @rckjacobs1!

Were the answers above helpful? What errors do you see in your logs?

Cheers,
Michel