Cert-manager: Service Account authentication to Vault

[mbrancato]

Is your feature request related to a problem? Please describe.
With Kubernetes auth having been added in #2040, it is restricted to known ServiceAccount tokens. I'm not certain of the use-case here as the secret required in spec.vault.auth.kubernetes.secretRef.name should have a random value appended to the service account name. That aside, I think what #558 was asking for was for the cert-manager Deployment to authenticate to Vault using its own SA token.

Was the choice not to support 'ambient' credentials in Kubernetes a design decision or just not implemented?

Describe the solution you'd like
Support for cert-manager to auth to Vault using its own service-account and without Issuer CRDs to need access to or know the name of secrets. The flow here would be that cert-manager would default to its own JWT token for auth to Vault. The Issuers would not need to specify spec.vault.auth.kubernetes.secretRef in the config.

Additional context
The current use-case requires knowing the SA token name. This is because the secrets.name is populated after creation of v1/ServiceAccount. The Issuer requires runtime knowledge and isn't purely declarative. An alternative here is specifying a service account name and allowing cert-manager to find the associated token.

Environment details (if applicable):

• Kubernetes version (e.g. v1.10.2): 1.16
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): n/a
• cert-manager version (e.g. v0.4.0): 0.11.0
• Install method (e.g. helm or static manifests): static manifests

/kind feature

[mbrancato]

After thinking on this more, I think it would be best if the Issuer took the service account name as an input and identified the secret name on its own.

[JoshVanL]

Hi @mbrancato!

I'm all for this. It is much better behavior to have a more declarative way of specifying the service account when the secret is at most times not deterministic.

The issue though is that service accounts are able to have more than one secret so cert-manager may not be able to know which one to use. My feeling for the best solution would be:

• Create a new kubernetes auth method spec.vault.auth.kubernetes.serviceAccount alongside spec.vault.auth.kubernetes.secretRef.
• If they are both or neither set then error not ready
• When the service account secrets list does not have a len of 1, error not ready
• Use the token in this secret as auth

In theory, we could also add _which_ secret of that service account to use, but at that point we already have this option with the secrets ref. We could also just use the first in the list, I believe this is what Kubernetes does when injecting a token into pods (?). Is this a decent idea?

IMO, I think I would rather try and keep with requiring only one secret listed. This seems safer and more obvious to users the behavior.

This will require extra reconciling over service accounts:

• If the service account is updated to add another secret to it's list, cert-manager needs to reconcile on issuers referencing this service account and mark not ready and error.
  
  
  
  • If the service account secret is changed, cert-manager needs to reconcile on issuers referencing this service account and get the new secret. This requires checking secrets if they have the service account annotation, and then checking that service account with the behavior of the previous point.
  
  

In terms of just using its own service account, I don't think that really makes sense. Most times, I would have thought a user would want to create a properly scoped service account separate from cert-manager for vault. I would assume it would be common to have multiple service accounts in one cluster requesting certs from vault with multiple issuers.

In any case, if you really want to be using cert-managers identity to request certificates, needing to use cert-managers own service account is available, it just needs to written explicitly. I would have thought this to be an anti pattern though.

What are your thoughts on this approach @mbrancato ?

FWIW unless I am mistaken, this is a non-trivial amount of work.

/cc @munnerz
/milestone Next

[munnerz]

/priority backlog

[jeremy-duckworth]

Hi, I'm part of a team that's using cert-manager + vault (bank-vault) as an issuer. We're using per-namespace, cert-manager issuers, and coupling it with namespace specific issuer policies stored in vault. To minimize the distribution of creds (e.g., approle) between vault and cert-manager, we've opted to use K8S service account auth -- and, as we're want to fully automate the deployment process, came across this issue.

Rather than writing integration code to do the secret name lookup and submit the issuer request, etc, I wanted to document a possible work-around here for others. In short, I've found creating a second API token (deterministically named secret), and associating it with an existing service account gets around the lookup problem. The trade-off here is storage and management of the additional secret, as necessary.

https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#to-create-additional-api-tokens

---
#
# Create a K8S service account for cert-manager <-> vault Issuer
# for the my-services namespace
#
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-manager-vault-issuer
  namespace: my-services
---
#
# Create a statically named secret for
# the cert-manager-vault-issuer SA that 
# will be used deterministically in the
# cert-manager issuer config.
#
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: cert-manager-vault-issuer-token
  namespace: my-services
  annotations:
     kubernetes.io/service-account.name: cert-manager-vault-issuer

[seanson]

This is the first Kubernetes service account implementation I've seen that uses a secretRef rather than pulling the service account token from /var/run/secrets/kubernetes.io/serviceaccount. It would be good to call out this and the workaround from @jeremy-duckworth in the documentation.

[jeremy-duckworth]

As an update/warning, if you are using Istio for mTLS mesh, please be aware of:

https://istio.io/docs/ops/configuration/mesh/secret-creation/

In short, Istio (at least in 1.4, and I believe in 1.5 as well) intercepts the kubernetes.io/service-account-token request and mutates it to a istio.io/key-and-cert secret type. I tried various methods to get past this issue, but it ultimately required namespace-level overrides that were not tenable due to other workloads in the issuer namespace. In the end, due to this behavior, I ended up using a configmap-based patch script and a helm post-install/upgrade hook to patch the issuers with the proper secret ref.

[povils]

What's the status of this feature request ?:)

[munnerz]

@povils there's nobody working on it at the moment as far as I am aware, and it's labelled as 'help wanted' 😄 if you'd like to get involved and work on it, @JoshVanL shared a proposal/some thoughts on it above: https://github.com/jetstack/cert-manager/issues/2345#issuecomment-553105450

[povils]

Hey, @munnerz I can pick it up)

[munnerz]

Awesome, please feel free to ask questions here or share any designs/ideas if you are uncertain on the API schema needed if API changes are needed!