Caprover: Certificates auto renew, but are not picked up/served until apps redeployed

[dgpokl]

What is the problem?

• Caprover version: 1.6.1
• I can see that certbot is renewing certs perfectly, well in advance, but my sites are still serving up the stale and expired certs forever unless the apps have been redeployed after the renew took place.

Steps to reproduce the problem:

• Some time ago, I clicked the button to Enable HTTPS in Caprover and the certs got set up without any issue.
• I noticed today, checking in the browser that my certs are about to expire tonight. (I had a calendar reminder to check this that I made 90 days ago, when they did expire and caused an outage.)
• I logged into caprover server over ssh and ran this command: docker exec -it $(docker ps --filter name=captain-certbot -q) /bin/sh
• From that shell I ran certbot certificates
  Output:

/opt/certbot # certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: captain.mysite.com
    Domains: captain.mysite.com
    Expiry Date: 2020-08-09 02:49:39+00:00 (VALID: 67 days)
    Certificate Path: /etc/letsencrypt/live/captain.mysite.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/captain.mysite.com/privkey.pem
  Certificate Name: app1.mysite.com
    Domains: app1.mysite.com
    Expiry Date: 2020-08-02 08:25:39+00:00 (VALID: 60 days)
    Certificate Path: /etc/letsencrypt/live/app1.mysite.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/app1.mysite.com/privkey.pem
  Certificate Name: app2.mysite.com
    Domains: app2.mysite.com
    Expiry Date: 2020-08-02 08:25:46+00:00 (VALID: 60 days)
    Certificate Path: /etc/letsencrypt/live/app2.mysite.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/app2.mysite.com/privkey.pem
  Certificate Name: app3.mysite.com
    Domains: app3.mysite.com
    Expiry Date: 2020-08-17 13:49:39+00:00 (VALID: 75 days)
    Certificate Path: /etc/letsencrypt/live/app3.mysite.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/app3.mysite.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This was unexpected. Basically the certs are all great and appear to have been renewed well ahead of time. I checked the browser again and hard refreshed and saw that it is still serving up the stale certificate.

Next, I ran a caprover deploy of the same code revision I deployed 90 days ago and the cert served to the browser immediately updated to have an expiry of August 2 - exactly the date and time that appears above.

So it looks to me like, unless apps get redeployed, new certs, even though Caprover is renewing them for me, don't take effect.

I'm sure there's something I'm doing wrong, and am posting here after finding almost no discussion of ssl and caprover on the whole Internet. I'll be happy to read the docs on how to get the server to pick up new certs, if someone can point me to them. My issue is right now it's pretty bad because I don't even get reminders from LE since they 'know' I have a new cert already, but I actually need to manually redeploy every app to make sure that the certs update. And I kind of thought the whole point of certbot was basically to make sure all renewals happen even if the server admin doesn't have recurring calendar items to babysit the certs 😄

Answers to the following questions where applicable:

• Your OS and version? Ubuntu 18.04

• RAM? 4GB

  If applicable, content of captain-definition file:

• my apps are vanilla nginx containers serving static sites

[githubsaturn]

I think you tapped into a bug! This has been reported randomly in the past, but we never had a solid understanding of steps that could cause such an issue. Now we know! Thanks! We'll get this fixed soon.

[dgpokl]

I still have some apps in my caprover instance that I haven't restarted by the way, so let me know if I can perform any steps to help uncover the issue, or I could do a zoom with someone to poke around at it.

[githubsaturn]

Thanks! The issue is clear to me. We need to reload the nginx config upon certificate renewal.