vault CLI tool fails to parse root CA certificate

[shore]

Below is a session log in which I create a new dev instance, mount a pki backend, intialize its root CA, then request the certificate. The tool fails to parse the response.

I'm using port 8201 to reproduce because I already had an instance on :8200 that I didn't want to destroy.

If I use curl to fetch the certificate instead, via $ curl -H X-Vault-Token:\ $(< .vault-token) -vsS http://127.0.0.1:8201/v1/pki/ca/pem, the response appears as plain text in the body, and openssl x509 happily parses the certificate.

$ vault version
Vault v0.6.0
$ vault server -dev -dev-listen-address=127.0.0.1:8201 &
[1] 7975
$ ==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.

The only step you need to take is to set the following
environment variables:

    export VAULT_ADDR='http://127.0.0.1:8201'

The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.

Unseal Key: 59b2f048fde4cd419448ed19df8f2c1ab439abc18f677014d6358161d69ab848
Root Token: 8ff5bcfe-6716-a011-2b86-2e041f819da3

==> Vault server configuration:

                 Backend: inmem
              Listener 1: tcp (addr: "127.0.0.1:8201", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Version: Vault v0.6.0

==> Vault server started! Log data will stream in below:

2016/07/28 01:13:22 [INFO] core: security barrier not initialized
2016/07/28 01:13:22 [INFO] core: security barrier initialized (shares: 1, threshold 1)
2016/07/28 01:13:22 [INFO] core: post-unseal setup starting
2016/07/28 01:13:22 [INFO] core: mounted backend of type generic at secret/
2016/07/28 01:13:22 [INFO] core: mounted backend of type cubbyhole at cubbyhole/
2016/07/28 01:13:22 [INFO] core: mounted backend of type system at sys/
2016/07/28 01:13:22 [INFO] rollback: starting rollback manager
2016/07/28 01:13:22 [INFO] core: post-unseal setup complete
2016/07/28 01:13:22 [INFO] core: root token generated
2016/07/28 01:13:22 [INFO] core: pre-seal teardown starting
2016/07/28 01:13:22 [INFO] rollback: stopping rollback manager
2016/07/28 01:13:22 [INFO] core: pre-seal teardown complete
2016/07/28 01:13:22 [INFO] core: vault is unsealed
2016/07/28 01:13:22 [INFO] core: post-unseal setup starting
2016/07/28 01:13:22 [INFO] core: mounted backend of type generic at secret/
2016/07/28 01:13:22 [INFO] core: mounted backend of type cubbyhole at cubbyhole/
2016/07/28 01:13:22 [INFO] core: mounted backend of type system at sys/
2016/07/28 01:13:22 [INFO] rollback: starting rollback manager
2016/07/28 01:13:22 [INFO] core: post-unseal setup complete

$ export VAULT_ADDR='http://127.0.0.1:8201'
$ vault mounts
Path        Type       Default TTL  Max TTL  Description
cubbyhole/  cubbyhole  n/a          n/a      per-token private secret storage
secret/     generic    system       system   generic secret storage
sys/        system     n/a          n/a      system endpoints used for control, policy and debugging
$ vault mount pki
2016/07/28 01:17:20 [INFO] core: mounted 'pki/' type: pki
Successfully mounted 'pki' at 'pki'!
$ vault write pki/root/generate/internal common_name="Root CA"
Key             Value
---             -----
certificate     -----BEGIN CERTIFICATE-----
MIIDKTCCAhGgAwIBAgIUQLBFW8lVw4nZ6AcGz6aajKQkosEwDQYJKoZIhvcNAQEL
BQAwEjEQMA4GA1UEAxMHUm9vdCBDQTAeFw0xNjA3MjgwMTE3MzRaFw0xNjA4Mjcw
MTE4MDRaMBIxEDAOBgNVBAMTB1Jvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC0hRgY1ILA4YovCeFFJF//l/9mUCecDdEXLviglpWfZJI/Hjp9
qMPAfWAjFCErNneSRMuL7xv4UfFOC7cyKf95wC3OO7beOH9dtMaFA6FgmSc4BrFJ
Jjgo08/Soh1JW2ssZ+ECJfuNZ0744v6RBEmC+MJ+4GbkjN9KJWXtqHvWbaJPAC+h
02Mfzaj/N8qJOWRyH7MVhon3LAJcscxSFrHSpfDckCUnn8Mmanu5eHvcEmUIAOnC
4RI3PKUDDkw2dWMJS3ziEuy/XEprCgYJfaaX+v4+Nwuj5/EtqSJ52gGVSvuoJ+1l
mWyDcyUWxKJWhAnrMaEXB5L4PU0NICZqXW+xAgMBAAGjdzB1MA4GA1UdDwEB/wQE
AwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSH6SLqAQXqFGSiGcY8JtKy
Jc0r7TAfBgNVHSMEGDAWgBSH6SLqAQXqFGSiGcY8JtKyJc0r7TASBgNVHREECzAJ
ggdSb290IENBMA0GCSqGSIb3DQEBCwUAA4IBAQBsm9NEP3UWyE6IKSUHS12pXD9j
p/tQ7aUvMQuJ/mLQqBHNRAUud4Wr7oPBXmxj+VP61+rL5e7q0FDJ0/x1qXj9dCo5
XVPFW0VrJzkURqSeV4XvomGmPQwa9qUzDsN3c5Y/vt/XJyWIHGhxYliNcTFI/GO0
tKtFGGkIn0vObJhge/VvxDn7QCYuG8vL02RI5Gkk+Vmn3tgMle5hY8i5icSCleiy
cZnTvdFr5yEiKlun42Z/VHoeuGFN8YMMAvG5bGYAn8iirSmqDXY+y1gCKNbkg7/G
V9J5s8r8ToIbUZ/mMursuzhj8DACSN53O8l/oAOIFgutPrSXgxb88xL64iBL
-----END CERTIFICATE-----
expiration      1472260684
issuing_ca      -----BEGIN CERTIFICATE-----
MIIDKTCCAhGgAwIBAgIUQLBFW8lVw4nZ6AcGz6aajKQkosEwDQYJKoZIhvcNAQEL
BQAwEjEQMA4GA1UEAxMHUm9vdCBDQTAeFw0xNjA3MjgwMTE3MzRaFw0xNjA4Mjcw
MTE4MDRaMBIxEDAOBgNVBAMTB1Jvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC0hRgY1ILA4YovCeFFJF//l/9mUCecDdEXLviglpWfZJI/Hjp9
qMPAfWAjFCErNneSRMuL7xv4UfFOC7cyKf95wC3OO7beOH9dtMaFA6FgmSc4BrFJ
Jjgo08/Soh1JW2ssZ+ECJfuNZ0744v6RBEmC+MJ+4GbkjN9KJWXtqHvWbaJPAC+h
02Mfzaj/N8qJOWRyH7MVhon3LAJcscxSFrHSpfDckCUnn8Mmanu5eHvcEmUIAOnC
4RI3PKUDDkw2dWMJS3ziEuy/XEprCgYJfaaX+v4+Nwuj5/EtqSJ52gGVSvuoJ+1l
mWyDcyUWxKJWhAnrMaEXB5L4PU0NICZqXW+xAgMBAAGjdzB1MA4GA1UdDwEB/wQE
AwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSH6SLqAQXqFGSiGcY8JtKy
Jc0r7TAfBgNVHSMEGDAWgBSH6SLqAQXqFGSiGcY8JtKyJc0r7TASBgNVHREECzAJ
ggdSb290IENBMA0GCSqGSIb3DQEBCwUAA4IBAQBsm9NEP3UWyE6IKSUHS12pXD9j
p/tQ7aUvMQuJ/mLQqBHNRAUud4Wr7oPBXmxj+VP61+rL5e7q0FDJ0/x1qXj9dCo5
XVPFW0VrJzkURqSeV4XvomGmPQwa9qUzDsN3c5Y/vt/XJyWIHGhxYliNcTFI/GO0
tKtFGGkIn0vObJhge/VvxDn7QCYuG8vL02RI5Gkk+Vmn3tgMle5hY8i5icSCleiy
cZnTvdFr5yEiKlun42Z/VHoeuGFN8YMMAvG5bGYAn8iirSmqDXY+y1gCKNbkg7/G
V9J5s8r8ToIbUZ/mMursuzhj8DACSN53O8l/oAOIFgutPrSXgxb88xL64iBL
-----END CERTIFICATE-----
serial_number   40:b0:45:5b:c9:55:c3:89:d9:e8:07:06:cf:a6:9a:8c:a4:24:a2:c1
$ vault read pki/ca/pem
Error reading pki/ca/pem: invalid character '-' in numeric literal
$ vault read pki/ca
Error reading pki/ca: json: cannot unmarshal number into Go value of type api.Secret
$

[jefferai]

Hi @shore,

You want to be using pki/cert/ca. As the docs for the pki/ca endpoint indicate, it's a raw endpoint that doesn't return a standard data structure, so the CLI doesn't know how to parse it.

[mateuszkwiatkowski]

Hi @jefferai,
Could you reopen this issue and treat it as a user experience bug? The first thing I thought after seeing this error is "it must be a bug!" and I came directly to Github issues. Maybe vault CLI could detect raw endpoints and display link to documentation instead of displaying error? I saw a couple more issues related to this so it would prevent users from opening unnecessary issues and save users' time devoted to troubleshooting.

[jefferai]

@mateuszkwiatkowski The docs (https://www.vaultproject.io/api/secret/pki/index.html#read-ca-certificate) say:

This endpoint retrieves the CA certificate in raw DER-encoded form. This is a bare endpoint that does not return a standard Vault data structure and cannot be read by the Vault CLI.

If you have suggestions to make that more explicit that would be great, but it is rather explicitly documented as something the CLI can't parse.

[mateuszkwiatkowski]

@jefferai, yes, I'm aware that it's well-documented behavior. I learned about it after googling the error message though. :-)
For me, this issue is about intuitiveness of command line interface. Ability to use a utility without memorizing the whole documentation and without googling "standard" errors is an attribute of well designed interface.

Today Vault communicates this way:

$ vault read pki/ca/pem
Error reading pki/ca/pem: invalid character '-' in numeric literal

But it could do better, example:

$ vault read pki/ca/pem
Error reading pki/ca/pem: this is a raw endpoint. To learn more: https://www.vaultproject.io/api/secret/pki/index.html#read-ca-certificate

[jefferai]

There isn't really a way for Vault to communicate this to the client (if there was, it wouldn't be a raw endpoint).

Therefore the CLI has no way of knowing whether it's just bogus data, or whether it's meant to be that way.

[mateuszkwiatkowski]

@jefferai How about Content-Type header?

Content-Type: application/pkix-cert

[jefferai]

@mateuszkwiatkowski Sure, we could add in hardcoded special cases, but that only works in some situations. There is no purely programmatic way for the client to know.

[nkissebe]

I have to agree with the OP. This is really unexpected behavior (documented or not). You see the same error when using nomad templates. As such I don't see how to read the root CA from Vault into a template controlled file in nomad. I can fetch it by fingerprint, which is ok'ish.

If the endpoint's results aren't detected as json it should just return the output re-wrapped as a single "data" key. Better than puking a technobabble error message. There may well be better ways to do what I want.

[awwithro]

I just ran into this myself and have similar thoughts to the OP. Its a rough experience to not be able to work with the whole api using the official tool. Integrating with vault right now consists of a mix of vault and curl commands which isn't fun to maintain.

It would be nice if pki was a top level command in the cli that could with issuing and retrieving certs/CAs. Maybe then the pki subcommand could have the logic for accessing the special cases of raw pki endpoints?

[v6]

// , Went down this rabbit hole a bit last night. Would be nice if the CLI had a heuristic or pre-shared list to tell whether it's getting binary.

Does the CLI have a pre-shared list of which endpoints are raw?

If not, would you be interested in reviewing a PR for same?

[v6]

// , IMO a CLI's "awareness," in some ways, of the expected data format of the API we expect that CLI to consume could have some very positive pragmatic advantages for technical users.

But I'm not sure if this violates the design of Vault's API or Communicating Sequential Processes.