Cert-manager: secret not found when using the cluster issuer

Created on 12 Jun 2018 · 22Comments · Source: jetstack/cert-manager

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
When switching from Issuer to ClusterIssuer

Re-queuing item "*" due to error processing: error getting clouddns service account: secret "*" not found

What you expected to happen:
A certificate to be issued

How to reproduce it (as minimally and precisely as possible):
following the docs
and then migrate the to using ClusterIssuers

Anything else we need to know?:
the secret existed in both the certificates namespace and in the default namespace

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-05-12T04:12:12Z", GoVersion:"go1.9.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.6-gke.1", GitCommit:"cb151369f60073317da686a6ce7de36abe2bda8d", GitTreeState:"clean", BuildDate:"2018-04-07T22:06:59Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration**:
Install tools:
Others:

config

---
# Source: cert-manager/templates/00-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: "cert-manager"

---
# Source: cert-manager/templates/certificate-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: certificates.certmanager.k8s.io
  labels:
    app: cert-manager
    chart: cert-manager-v0.3.2
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  scope: Namespaced
  names:
    kind: Certificate
    plural: certificates
    shortNames:
      - cert
      - certs

---
# Source: cert-manager/templates/clusterissuer-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterissuers.certmanager.k8s.io
  labels:
    app: cert-manager
    chart: cert-manager-v0.3.2
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: ClusterIssuer
    plural: clusterissuers
  scope: Cluster
---
# Source: cert-manager/templates/issuer-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: issuers.certmanager.k8s.io
  labels:
    app: cert-manager
    chart: cert-manager-v0.3.2
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: Issuer
    plural: issuers
  scope: Namespaced
---
# Source: cert-manager/templates/deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cert-manager
  namespace: "cert-manager"
  labels:
    app: cert-manager
    chart: cert-manager-v0.3.2
    release: cert-manager
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cert-manager
      release: cert-manager
  template:
    metadata:
      labels:
        app: cert-manager
        release: cert-manager
      annotations:
    spec:
      serviceAccountName: default
      containers:
        - name: cert-manager
          image: "quay.io/jetstack/cert-manager-controller:v0.3.0"
          imagePullPolicy: IfNotPresent
          args:
          - --cluster-resource-namespace=$(POD_NAMESPACE)
          - --leader-election-namespace=$(POD_NAMESPACE)
          env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
---
# this secret is present in both default and "somenamespacehere"
apiVersion: v1
kind: Secret
metadata:
  name: clouddns-service-account
type: Opaque
data:
  service-account.json: *****
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging

    # ACME DNS-01 provider configurations
    dns01:

      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:

        - name: prod-dns
          clouddns:
            # A secretKeyRef to a google cloud json service account
            serviceAccountSecretRef:
              name: clouddns-service-account
              key: service-account.json
            # The project in which to update the DNS zone
            project: *****
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ****
  namespace: somenamespacehere
spec:
  secretName: tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: '*.example.com'
  dnsNames:
  - www.example.com
  acme:
    config:
    - dns01:
        provider: prod-dns
      domains:
      - '*.example.com'
      - www.example.com

kinbug lifecyclrotten

Source

mgazza

👍2

Most helpful comment

I found that the secret had to be in the same namespace as the cert-manager Service in order for it to work.

Can we reopen this issue as a docs request for explaining this important detail?

samuela on 27 Nov 2018

👍19 🎉2

All 22 comments

Could you give a try to deploy your clusterissuer secret into the kube-system namespace, @mgazza?

darkyat on 12 Jun 2018

👍1

@darkyat thanks for the suggestion, unfortunately that had no effect. I also created the secret in that namespace too.
the secret now exists in default, "somenamespacehere", kube-system

However creating the secret in the namespace 'cert-manager' appears to have fixed the issue!
Can the scope of this bug be to please update the documentation regarding this?

mgazza on 13 Jun 2018

👍7 ❤1 🎉1

Creating the secret in the namespace 'cert-manager' has no affect
E0614 13:09:02.877284 1 sync.go:168] [cert-manager/testtls] Error getting certificate 'testtls': secret "testtls" not found

ivanovUA on 14 Jun 2018

@ivanovUA that is an unrelated issue, I also receive this when I delete the tls secret for the certificate. cert-manager creates a new one so this can be ignored.

This bug relates specifically to 'I0613 10:16:44.039666 1 sync.go:241] Error preparing issuer for...'

mgazza on 14 Jun 2018

I believe Let's Encrypt does not allow [email protected] as a valid email address.

I got a similar error:
Error initializing issuer: 400 urn:acme:error:invalidEmail: Error creating new registration :: invalid contact domain. Contact emails @example.com are forbidden

perspectivus1 on 17 Jun 2018

@perspectivus1 you do realise i replaced the actual domain name email address with example.com rather than their real counterparts?

mgazza on 18 Jun 2018

😄1

@mgazza I mean next:

I0619 14:53:15.674011       1 sync.go:241] Error preparing issuer for certificate ssl-test/testapp: acme: authorization for identifier domain.net is invalid
E0619 14:53:15.679462       1 sync.go:168] [ssl-test/testapp] Error getting certificate 'testapp': secret "testapp" not found
E0619 14:53:15.679526       1 controller.go:186] certificates controller: Re-queuing item "ssl-test/testapp" due to error processing: acme: authorization for identifier domain.net  is invalid

ivanovUA on 19 Jun 2018

I had the same issue. Moving the secret to kube-system namespace solved it for me (was default).

iceydee on 20 Jun 2018

👍1

@iceydee in your setup you used ClusterIssuer?

ivanovUA on 20 Jun 2018

Having the same issue - the ClusterIssuer and secret are all in the same namespace (kube-system), still getting the Error getting certificate 'ifconfig-tls': secret "ifconfig-tls" not found error. I'm wondering if something about cross namespace stuff is borking.

mlaccetti on 20 Jun 2018

👍2

Added certificate into same namespace where deployed cert-manager and added CAA dns records to the root domain, and now it works

ivanovUA on 21 Jun 2018

👍1

Am having the same problem. Using a ClusterIssuer, as long as I place it and the Certificate resource in the same namespace as the helm-chart for cert-manager it works. If I attempt to place it in default the error "[default/redacted] Error getting certificate 'redacted': secret "redacted" not found".

Edit: Was resolved by purging the chart and re-installing. It may have been due to first installing in the default namespace, not purging, then installing in the kube-system namespace also... Hope this tard mistake helps someone else.

f33rx on 29 Jun 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

retest-bot on 27 Sep 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

retest-bot on 27 Oct 2018

closing as i no longer have this issue

mgazza on 9 Nov 2018

I found that the secret had to be in the same namespace as the cert-manager Service in order for it to work.

Can we reopen this issue as a docs request for explaining this important detail?

samuela on 27 Nov 2018

👍19 🎉2

moving it to default namespace worked for me

hijak on 7 Feb 2019

I found that the secret had to be in the same namespace as the cert-manager Service in order for it to work.

Can we reopen this issue as a docs request for explaining this important detail?

+1. This bit me as well. The instructions tell to deploy cert-manager into "cert-manager" namespace: https://docs.cert-manager.io/en/latest/getting-started/install.html
but the clouddns instruction does not mention that the namespace must be the same for the secret: https://docs.cert-manager.io/en/latest/tasks/acme/configuring-dns01/google.html

jait on 3 Apr 2019

👍3

:+1: on adding the cert-manager namespace to the documentation for secrets

RickvdStaaij on 19 Apr 2019

👍2

I used GitLab to install cert-manager and adding the Cloud DNS secret to the gitlab-managed-apps namespace worked. The namespace for the ingress-nginx-ingress-controller service and certmanager-cert-manager-... pod.

jhselvik on 7 Jun 2019

I used GitLab to install cert-manager and adding the Cloud DNS secret to the gitlab-managed-apps namespace worked. The namespace for the ingress-nginx-ingress-controller service and certmanager-cert-manager-... pod.

i also installed cert-manager with gitlab but i dont seem to get it working...
what namespace are you using on the cert-manager?

i cant get rid of the following error:
6 backend_ssl.go:49] error obtaining PEM from secret kube-system/letsencrypt-staging: error retrieving secret kube-system/letsencrypt-staging: secret kube-system/letsencrypt-staging was not found
and
controller.go:989] ssl certificate "kube-system/letsencrypt-staging" does not exist in local store

MichelFR on 11 Jul 2019

I am using Vault CA in my AKS Cluster, on creating a ClusterIssuer I am getting a similar error.

Warning ErrInitIssuer 3m58s (x266 over 21h) cert-manager Error initializing issuer: error reading Vault token from AppRole: error reading Vault AppRole from secret: cert-manager-vault-approle/ingress: secret "cert-manager-vault-approle" not found