Azure-docs: Constantly getting staging cert

[jpetitte]

Hi there,

I've run this tutorial multiple times (on the same kubernetes cluster), and I followed the Cleanup Resources Section between each tutorial run. I also executed the following commands between each run:

kubectl get customresourcedefinitions
kubectl delete customresourcedefinition <thelistedresourcedefs>

to eliminate any custom resource definitions.

The first time through the tutorial, I used the staging certs as is default in the tutorial. Then, I changed every "letsencrypt-staging" to "letsencrypt-prod", and I changed

spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory

to

spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory

in cluster-issuer.yaml

Is the old staging cert getting cached or stored somewhere that I'm not cleaning up? I feel like I'm missing something, but I can't put my finger on it.

Edit1) My kubernetes version is: 1.9.11

Thanks!

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: ada58160-dc50-b977-d3bc-65f92406e16d
• Version Independent ID: 5c596e04-fddb-effe-3b72-575f4df42d4e
• Content: Create an HTTPS ingress with Azure Kubernetes Service (AKS) cluster
• Content Source: articles/aks/ingress-tls.md
• Service: container-service
• GitHub Login: @iainfoulds
• Microsoft Alias: iainfou

[mimckitt]

Thanks for the feedback! We are currently investigating and will update you shortly.

[jpetitte]

Just adding some more information as I get it.

It looks like the cert manager pod had problems (kubectl get pods --all-namespaces):

kube-system opining-fly-cert-manager-5d584686f-rz9xq 0/1 CrashLoopBackOff 79 6h

[jpetitte]

I ran through the tutorial for a 4th time this morning (just so I could try to watch when the cert-manager pod breaks), literally changing no configuration from the last runs, and it worked. The cert-manager pod also shows no errors.

It always disturbs me a little when I can't root cause something because I want to be able to fight the problem in the future, but I'll take it for now.

Any ideas as to what may have occurred?

[jakaruna-MSFT]

@jpetitte Sorry for the late reply.
I went through the doc. I configured my environment with staging certs. After the I created a new clusterissuer and certificate for prod and then created an ingress with the prod cluster issuer.
It still showed me the staging cert, because I didn't change the secret name (tls-secret) in this example.

That secret "tls-secret" is created when i created the staging certificate. Since the same name is referenced while creating the prod certificate also, Cert manager didn't create a new one and used the existing certificates.

I have changed the secret name to something else (eg tls-secret-prod) and then recreated the prod certificate. This time all worked as expected. Generally those secrets are created in default namespace. We can describe them to have a look at the certificates inside those secrets.
you can also describe the certificates to find out more about that certificates status kubectl describe certificate <certificate name>.

If we look at the document side, from the beginning if we pick staging or prod, Document will work fine.
If we configure for one environment and try to migrate to the other environment, Then we will end up with the same certificate. I think we can add a note to change the secret name when migrating.

I am note sure why cert manager crashed. We need to look at the logs to find that out.

I hope this is the issue occurred to you. Please let me know.

[jpetitte]

Hey @jakaruna-MSFT, no problem. I'm not sure if this was the issue, because I was cleaning out the cluster before each run through the tutorial. Maybe the secret was still in there. I just don't know about the lifecycle of secrets inside clusters.

I'll keep this in mind with any future issues I face, and I think you're right that a note should be added to the docs. Good find!

[Ojisah]

I had the same issue and the cause was that the tls-secret was not deleted by running the steps described in the "Clean up resources" section. After running kubectl delete secret tls-secret the problem was solved.

[jakaruna-MSFT]

@iainfoulds I think we can change the secret name to "tls-secret-staging" so that the users will change that to prod when they are trying out stage and prod on same cluster.
Also consider adding a step in "clean up resources" to delete the secret.
Please investigate further and make appropriate changes.
CC: @MicahMcKittrick-MSFT

[photomoose]

I'm also getting this problem. I have confirmed that the certificates I have in tls-secret-prod are indeed the real certificates for my domain, and confirmed that my ingress is using the correct hostname and secret. Yet I am still being served the "Kubernetes Ingress Controller Fake Certificate", which I don't know where it comes from as I have no tls-secret-staging or tls-secret secret.

[photomoose]

Update: I followed the instructions at https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/quick-start/index.rst and finally managed to get a valid production certificate.

[Marusyk]

How to solve that?

[mimckitt]

Hey @iainfoulds

@Marusyk is having a similar issue and I see this item is still open. Did we ever come up with a game plan for this doc/ issue? I can enable people for support if needed as well.

[iainfoulds]

@MicahMcKittrick-MSFT Let's Encrypt would fall under advisory, best-effort through CSS - https://docs.microsoft.com/en-us/azure/aks/support-policies#aks-support-coverage

[mimckitt]

Thanks @iainfoulds

So for anyone having issues with the prod certificate, we would suggest you reach out to Azure Technical Support. If you do not have the ability to open a technical support ticket, you can email me at [email protected] and provide me with your Azure SubscriptionID and link to this issue. I can then enable your subscription for that support request.

[tomingoglia]

@iainfoulds you should consider @jakaruna-MSFT suggestion again. It would be a simple change and would provide clarity for new users. Problems like this cause hours of frustration and headaches for your team to triage. I think you made the wrong call on this.

[iainfoulds]

Sorry you ran into problems, @tomingoglia. The docs were changed months to ago to name the secret tls-secret-staging and add clean-up steps. Sorry you've missed those steps.

The comment I'd made was that issues with using Let's Encrypt fall under best-effort support if you reach out for additional assistance. You're welcome to reach out to Azure support, and they'll do their best to provide assistance. That's not a doc issue I can resolve though.

CC: @mlearned who leads the AKS docs now.

[tomingoglia]

@iainfoulds I haven't gotten to this step yet...so I'm not having this issue. There are other inconsistencies that I am seeing in the docs vs. my results and that caused me to begin reading through these items. After reading your response, it indicated that you would not be making changes. Thanks for the quick reply! This may help clear things up for people reading through these issues.

[nageshajab]

@jpetitte Sorry for the late reply.
I went through the doc. I configured my environment with staging certs. After the I created a new clusterissuer and certificate for prod and then created an ingress with the prod cluster issuer.
It still showed me the staging cert, because I didn't change the secret name (tls-secret) in this example.

That secret "tls-secret" is created when i created the staging certificate. Since the same name is referenced while creating the prod certificate also, Cert manager didn't create a new one and used the existing certificates.

I have changed the secret name to something else (eg tls-secret-prod) and then recreated the prod certificate. This time all worked as expected. Generally those secrets are created in default namespace. We can describe them to have a look at the certificates inside those secrets.
you can also describe the certificates to find out more about that certificates status kubectl describe certificate <certificate name>.

If we look at the document side, from the beginning if we pick staging or prod, Document will work fine.
If we configure for one environment and try to migrate to the other environment, Then we will end up with the same certificate. I think we can add a note to change the secret name when migrating.

I am note sure why cert manager crashed. We need to look at the logs to find that out.

I hope this is the issue occurred to you. Please let me know.

worked for me when I created a new secret.