Cert-manager: Cannot create certificate directly from ingress from different namespace than cert-manager & clusterissuer

Created on 15 Oct 2019 · 3Comments · Source: jetstack/cert-manager

Describe the bug:
When CertManager and ClusterIssuer are deployed in namespace cert-manager and ingress is in namespace apps, then cert-manager fails to create a certificate. The error from the certificate request is:

kubectl describe certificaterequests.cert-manager.io kafka-connect-cluster-tls-1673274601 -n apps

Failed to initialise vault client for signing: error reading Kubernetes service account token from cert-manager-token-n2llb: secret "cert-manager-token-n2llb" not found

The cert-manager-token-n2llb secret exists in the cert-manager namespace and I can authenticate to Vault using that token manually.

The ClusterIssuer is ready and the status is:

Message:            Vault verified
Reason:             VaultVerified
Status:             True
Type:               Ready

Ingress looks like this

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: internal-ingress
    cert-manager.io/cluster-issuer: vault-issuer
  labels:
    app: app
  name: kafka-connect-cluster-test
spec:
  tls: 
    - hosts:
        - kafka-connect-cluster.example.com
      secretName: kafka-connect-cluster-tls
  rules:
    - host: kafka-connect-cluster.example.com
      http:
        paths:
          - backend:
              serviceName: kafka-connect-cluster
              servicePort: kafka-connect

Expected behaviour:
Certificate should have been created

Steps to reproduce the bug:

Anything else we need to know?:

When creating a certificate manually in the cert-manager namespace everything seems to work:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: example
spec:
  secretName: test-tls
  duration: 24h
  renewBefore: 12h
  commonName: angelos.example.com
  dnsNames:
    -foo.example.com
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer

but when I apply the same certificate in the apps namespace it fails again with same error as described earlier

Environment details::

Kubernetes version: v1.13.10
Cloud-provider/provisioner: EKS
cert-manager version: v0.11.0
Install method: helm

Not sure if it's a bug or if I a have misconfigured something.

/kind bug

kinbug

Source

apetheriotis

👍10

Most helpful comment

I am having the same issue on my end but I am using secret via Vault so the error looks like this:

message: 'Required secret resource not found: secret "vault-approle" not found'

The vault-approle secret is within the cert-manager namespace which is different than the namespace I am trying to create the certificate in.

McShauno on 23 Oct 2019

👍2

All 3 comments

I am having the same issue on my end but I am using secret via Vault so the error looks like this:

message: 'Required secret resource not found: secret "vault-approle" not found'

The vault-approle secret is within the cert-manager namespace which is different than the namespace I am trying to create the certificate in.

McShauno on 23 Oct 2019

👍2

Confirm the Bug, I installed in a specific namespace "security", configured a ClusterIssuer with Hascicorp Vault as backend, and tried to emit a certificate in "default" namespace.

The log message is:
E1027 15:07:14.109045 1 vault.go:72] cert-manager/controller/certificaterequests-issuer-vault/sign "msg"="Required secret resource not found" "error"="secret \"cert-manager-vault-approle\" not found" "related_resource_kind"="ClusterIssuer" "related_resource_name"="vault-server-issuer" "related_resource_namespace"="" "resource_kind"="CertificateRequest" "resource_name"="vault-server-crt-3439034685" "resource_namespace"="default"

As sidenote if I move the secret to the default namespace, I can emit certificate from default namespace

After looking at the code, the vault client is initialized, using the namespace from CertificateRequest, while should be initialized from the namespace where the controller pod is running or better the one indicated by the --cluster-resource-namespace=$(POD_NAMESPACE) argument

https://github.com/jetstack/cert-manager/blob/716b11d8d9198fbe8ddb47dba0521b63a2e54941/pkg/controller/certificaterequests/vault/vault.go#L63

func (v *Vault) Sign(ctx context.Context, cr *v1alpha2.CertificateRequest, issuerObj v1alpha2.GenericIssuer) (*issuer.IssueResponse, error) {
    log := logf.FromContext(ctx, "sign")
    log = logf.WithRelatedResource(log, issuerObj)

    client, err := v.vaultClientBuilder(cr.Namespace, v.secretsLister, issuerObj)
    if k8sErrors.IsNotFound(err) {