Concourse: Vault Re-auth not following backoff logic

[christophermancini]

Bug Report

We setup a new staging cluster with v5 to ensure all is working as expected before upgrading our production cluster. Our security team informed us that our staging web node was hitting Vault with roughly 660k auth-requests over 2 hours every 4 hours. Our tokens are configured by default to expire after 4 hours. Our production environment (v4.2.1) is configured with the same parameters as our staging v5 environment, using the same Vault credentials and path.

Steps to Reproduce

Using Concourse binary w/config options:

CONCOURSE_VAULT_AUTH_BACKEND='approle'
CONCOURSE_VAULT_AUTH_PARAM='role_id:xxxx,secret_id:xxxxxx'
CONCOURSE_VAULT_CA_CERT='/etc/ssl/certs/cert.pem'
CONCOURSE_VAULT_PATH_PREFIX='/concourse'
CONCOURSE_VAULT_URL='https://myvaultapi:8200'

and vault policy:

# meta:{"policy": "concourse", "approle": true, "token_type": "batch"}

path "concourse/*" {
  capabilities = ["read", "list"]
}

Expected Results

To not attempt more than 60 * 60 * 2 vault api authentication requests.

Actual Results

Sample log output:

Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.171724805Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987682"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.161594742Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987681"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.149902501Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987680"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.139257673Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987679"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.128274794Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987678"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.117546628Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987677"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.107096377Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987676"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.096616981Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987675"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.085979553Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987674"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.074766455Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987673"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.062651456Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987672"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.051468331Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987671"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.034738488Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987670"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.025571724Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987669"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.015045438Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987668"}}
Mar 25 18:04:11 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:11.004478187Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987667"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.991264822Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987666"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.981773617Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987665"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.970592135Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987664"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.960504717Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987663"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.949790967Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987662"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.937586388Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987661"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.927461189Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987660"}}
Mar 25 18:04:10 concourse-web-stage-ca7c00b2f3 concourse[24945]: {"timestamp":"2019-03-25T18:04:10.916638204Z","level":"error","source":"atc","message":"atc.credential-manager.renew.failed","data":{"error":"Error making API request.\n\nURL: PUT https://example.com:8200/v1/auth/token/renew-self\nCode: 400. Errors:\n\n* batch tokens cannot be renewed","name":"vault","session":"8.12987659"}}

Version Info

• Concourse version: 5.0.0
• Deployment type (BOSH/Docker/binary): binary
• Infrastructure/IaaS: DO Droplets
• Browser (if applicable):
• Did this used to work?

[vito]

Yeah, I noticed this too while working on the Vault docs. The fix is in already and will be out in 5.1.0:

https://github.com/concourse/concourse/commit/37879afeb51d94d52cb9b216051d5a680f0dd839