Vault: Cant read kv-v2 secrets with a JWT token

[rayisbadat]

Describe the bug
I have my gitlab runner CI/CD pipelines set to use vault cli to login via JWT tokens to fetch secrets. When configured with the kv version 1 secrets everything works fine. If i switch over to the kv version 2 secrets, it gets a 403 error trying to fetch the secrets. Nothing else is changing, except the secrets engine.

I am deploying with terraform, but confirmed it does the same if I create the secrets mount and values manually.

To Reproduce
Steps to reproduce the behavior:

• Set up a working CI/CD pipeline that uses vault cli to read the secrets
  
  
  
  • Ex:
    
    ```$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=${CI_PROJECT_NAME}-${ENVIRONMENT_SLUG} jwt=$CI_JOB_JWT)"
    
    $ export CI_REGISTRY_PASSWORD="$(vault kv get -field=CI_REGISTRY_PASSWORD operations/gitlab/infrastructure-vault-demos/${ENVIRONMENT_SLUG}/gitlab-ci )"
  
  

* setup a secret using the kv version 1 engine
* Launch job and watch it succeed
* Disable the engine, and recreate everything with kv version 2 and secrets
* Relaunch same jobs, and watch it 403



**Expected behavior**
vault cli should be able to access the secret with JWT tokens regardless of the version.

**Environment:**
* Vault Server Version (retrieve with `vault status`): vault 1.4.3
* Vault CLI Version (retrieve with `vault version`):1.4.2 and 1.5.0
* Server Operating System/Architecture:
Vault server running in k8s via  vault:1.4.3  docker image
Vault CLI running in apline based docker image

Vault server configuration file(s):

```/ $ cat /tmp/storageconfig.hcl
disable_mlock = true
ui = true
listener "tcp" {
  #Enable TLS
  tls_disable = 0
  tls_cert_file = "/vault/userconfig/vault-server-tls/tls.crt"
  tls_key_file = "/vault/userconfig/vault-server-tls/tls.key"
  tls_client_ca_file = "/home/vault/ssl/ca.crt"
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}
storage "raft" {
  path = "/vault/data"
}
service_registration "kubernetes" {}
# Enable a dns name to match self signed cert
cluster_addr = "https://127.0.0.1:8201"/ $ 

Additional context
success (kv v1):

$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=${CI_PROJECT_NAME}-${ENVIRONMENT_SLUG} jwt=$CI_JOB_JWT)"
$ export CI_REGISTRY_PASSWORD="$(vault kv get -field=CI_REGISTRY_PASSWORD operations/gitlab/infrastructure-vault-demos/${ENVIRONMENT_SLUG}/gitlab-ci )"
$ export CI_REGISTRY_USER="$(vault kv get -field=CI_REGISTRY_USER operations/gitlab/infrastructure-vault-demos/${ENVIRONMENT_SLUG}/gitlab-ci )"

error (kv v2):

$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=${CI_PROJECT_NAME}-${ENVIRONMENT_SLUG} jwt=$CI_JOB_JWT)"
$ export CI_REGISTRY_PASSWORD="$(vault kv get -field=CI_REGISTRY_PASSWORD operations/gitlab/infrastructure-vault-demos/${ENVIRONMENT_SLUG}/gitlab-ci )"
Error reading operations/data/gitlab/infrastructure-vault-demos/dev/gitlab-ci: Error making API request.
URL: GET https://vault.cloud.alcf.anl.gov/v1/operations/data/gitlab/infrastructure-vault-demos/dev/gitlab-ci
Code: 403. Errors:
* 1 error occurred:
    * permission denied

[jeffsanicola]

Could you provide the policy details?

KVv1 and KVv2 use different pathing which may be part of the issue.

[aphorise]

/secret/... in v1 vs /secret/data/... in v2

[rayisbadat]

I did not have the operations/data/... in the policy path, so i added it just now to try. No difference in behavior.

$ vault policy read gitlab-ci-dev
path "operations/gitlab/infrastructure-vault-demos/dev/*" {
  capabilities = ["read"]
}

#I just added this one to test
vault policy read gitlab-ci-dev_kvv2
path "operations/data/gitlab/infrastructure-vault-demos/dev/*" {
  capabilities = ["read"]
}

#Updated role for these two policies
$ vault read auth/jwt/role/gitlab-ci-job-tool-dev
Key                        Value
---                        -----
allowed_redirect_uris      <nil>
bound_audiences            []
bound_claims               map[project_id:527 ref:vault_test ref_type:branch]
bound_claims_type          string
bound_subject              n/a
claim_mappings             <nil>
clock_skew_leeway          0
expiration_leeway          0
groups_claim               n/a
not_before_leeway          0
oidc_scopes                <nil>
role_type                  jwt
token_bound_cidrs          []
token_explicit_max_ttl     129600h
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [gitlab-ci-dev gitlab-infrastructure-vault-demos-dev gitlab-ci-dev-kvv2]
token_ttl                  8760h
token_type                 default
user_claim                 user_email
verbose_oidc_logging       false

[jeffsanicola]

Not sure if it's a typo but your policy output command:
vault policy read gitlab-ci-dev_kvv2

doesn't match the policy being assigned:
token_policies [gitlab-ci-dev gitlab-infrastructure-vault-demos-dev gitlab-ci-dev-kvv2]

gitlab-ci-dev_kvv2 vs gitlab-ci-dev-kvv2
That would definitely cause your permission denied message.

[rayisbadat]

Sigh....Sorry for wasting your time and thanks for the help.
@jeffsanicola when i actually properly apply @aphorise advice of operations/data/* it all works. The reason it was working from my non jwt attempts was they were inheriting a operations/* that it wasnt.