Vault: Vault may not be removing expired tokens from Consul

[justenwalker]

I noticed that with the app-id backend - it seems that there are a lot of tokens in consul after running for some time. We now have over 200k vault keys, which are primarily under vault/sys/expire/id/auth/app-id/login/<HASH> (78353 keys),
vault/sys/token/accessor/<HASH> (78356 keys),
and vault/sys/token/id/<HASH> (78356 keys)

Vault version: v0.6.0
Consul version: v0.6.4

Update: originally I thought these were empty, but it does appear they all have data, just that the consul ui will not display it.

[jefferai]

The Consul UI doesn't display it because it's encrypted, so it doesn't look like valid JSON.

Your clients are most likely logging in continuously instead of reusing tokens, so the tokens are no longer in use but are not purged until they expire. You should either reuse tokens or set the token lifetime short.

[justenwalker]

@jefferai Is there a way to know which applications are creating these tokens?

[jefferai]

The token metadata includes a hash of the user id and app id. You can use this to correlate entries. Combined with audit logs and token accessors you can easily revoke tokens for a given application.

[justenwalker]

Can the root token enumerate the currently issued tokens in the system? Or is there any documentation on how to correlate this hashes to a given app-id/user-id?

[jefferai]

You can use auth/token/accessors (https://www.vaultproject.io/docs/auth/token.html) in 0.6.1 to enumerate token accessors and from there manage the tokens.

[justenwalker]

thanks @jefferai - would be good to have a guide to doing this kind of introspection and revocation. Maybe i'll write up my experience revoking over 78k tokens XD

[mzupan]

@justenwalker did you ever come up with a way to revoke all tokens but the root token?

[juliangamble]

Hi @justenwalker - did you ever write that up?

[justenwalker]

Hey @juliangamble / @mzupan

First, you'll need to grab jq

When you get token info from an app-id you can match the application by it's metadata.

{
  "request_id": "b0f90afe-8d9d-6af6-4bf8-1adcd1cfd04b",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "accessor": "603a52e0-cdb6-4531-89c3-45c7ab034d5c",
    "creation_time": 1471789748,
    "creation_ttl": 2592000,
    "display_name": "app-id-My App",
    "explicit_max_ttl": 0,
    "id": "",
    "meta": {
      "app-id": "sha1:86c041fcef7a5606e63279e6b9bc10c8cc9cba49",
      "user-id": "sha1:dc555baa757a977a155d5011b5a221f47eb2e4f4"
    },
    "num_uses": 0,
    "orphan": true,
    "path": "auth/app-id/login",
    "policies": [
      "default",
      "some-policy"
    ],
    "renewable": false,
    "ttl": 20141467
  },
  "warnings": null
}

Here's a utility script that will do most of the work:

#!/bin/bash

if [ -r "~/.vault-token" ]; then
    token=$(cat "~/.vault-token")
fi

export VAULT_ADDR=${VAULT_ADDR:-"http://localhost:8200"}
export VAULT_TOKEN=${VAULT_TOKEN:-"$token"}

dump() {
  curl -k  -H "X-Vault-Token: $VAULT_TOKEN" "$VAULT_ADDR/v1/auth/token/accessors?list=true"  | jq -r '.data.keys[]' > accessors.list
}

revoke_app() {
  local accessor=$1
  local revoke_app=$2
  local appid=$(vault token-lookup -format=json -accessor $accessor | jq -r '.data.meta."app-id"')
  if [ "$appid" = "$revoke_app" ]; then
    echo "Revoke: $accessor"
    vault token-revoke -accessor $accessor > /dev/null
  fi
}

revoke_apps() {
  if ! [ -r accessors.list ]; then
    echo "Cannot find accessors.list; run dump first"
    exit 1
  fi
  while read ACCESSOR; do
    revoke_app "$ACCESSOR" "$1"
  done < accessors.list
}


info() {
  if ! [ -r accessors.list ]; then
    echo "Cannot find accessors.list; run dump first"
    exit 1
  fi
  while read ACCESSOR; do
    vault token-lookup -format=json -accessor $ACCESSOR
  done < accessors.list
}

case ${1:-"help"} in
  dump)
    echo "Dumping accessors.list"
    dump
  ;;
  info)
    echo "Dumping token information"
    info
  ;;
  revoke-app-id)
    if [ -z "$2" ]; then
      echo "Must provide an app ID hash as argugment 2"
      echo "Example: sha1:43a9e4dcda072319e26d79fda59bae2bf17af288"
      echo "Can get this from the 'info' command"
      exit 1
    fi
    revoke_apps "$2"
  ;;
  *|help)
    cat <<EOF
Usage:

  $0 dump                   # Dump accessors.list
  $0 info                   # Get token information on each accessor
  $0 revoke-app-id APP_HASH # Revoke AppID tokens matching the APP_HASH
  $0 help                   # Print this help
EOF
  ;;
esac