Addons: LetsEncrypt not auto renewing certificates

[borpin]

Supervised Install.

I have just had a renewal notice for the certificate for the domain name I use for HA and setup using the LetsEncrypt Plugin. On the basis of getting the email, the renewal should have already happened (certbot does not wait that late).

It appears the auto-renew is not working.

When the addon is manually started, the renewal occurs.

letsencrypt log showing previous auto renewal attempt plus the renewal done when the addon was restarted.

2020-07-08 01:50:16,689:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 01:50:16,690:DEBUG:certbot.main:Arguments: ['-q']
2020-07-08 01:50:16,690:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 01:50:16,722:DEBUG:certbot.log:Root logging level set at 30
2020-07-08 01:50:16,723:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 01:50:16,734:DEBUG:certbot.renewal:no renewal failures
2020-07-08 12:52:11,104:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:52:11,105:DEBUG:certbot.main:Arguments: []
2020-07-08 12:52:11,105:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:52:11,138:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:52:11,141:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:No candidate plugin
2020-07-08 12:52:11,152:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2020-07-08 12:59:24,483:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:59:24,484:DEBUG:certbot.main:Arguments: []
2020-07-08 12:59:24,485:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:59:24,512:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:59:24,513:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log

addon log - log was empty before manual start.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] file-structure.sh: executing... 
[cont-init.d] file-structure.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[13:02:01] INFO: Selected DNS Provider: dns-cloudflare
[13:02:02] INFO: Use propagation seconds: 60
[13:02:02] INFO: Use CloudFlare token
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for hassio.borpin.net
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /data/letsencrypt/live/hassio.borpin.net/fullchain.pem
   Your key file has been saved at:
   /data/letsencrypt/live/hassio.borpin.net/privkey.pem
   Your cert will expire on 2020-10-06. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

syslog excerpts

Jul  8 01:50:15 hassio systemd[1]: Starting Certbot...
Jul  8 01:50:16 hassio systemd[1]: certbot.service: Succeeded.
Jul  8 01:50:16 hassio systemd[1]: Started Certbot.

/var/log/syslog:Jul  8 12:00:02 hassio CRON[15469]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)

[agners]

From what I understand, the supervisor has no feature which starts the Let's Encrypt add-on when necessary... By default the startup mode is set as once, which basically means "on user request". If you set it to "Start on boot", then it will get started on boot but you will have to reboot the system periodically...

From what I can tell there is no "periodically startup mode" for add-ons (see also startup section in
https://developers.home-assistant.io/docs/add-ons/configuration/). I guess a cron type of startup mode would be required for this case, which need to be implemented in the supervisor project.

[borpin]

@agners, no reason why there couldn't be AFAICS. I think if you modify the installed certbot config to find the right config file it will work, but I have not delved any deeper as yet.

I think the addon installs certbot to the underlying OS (although I may have done that in the past), but the standard config does not find the certificate config by default.

[lambtho12]

A simple (user-based) fix for this is to launch the add-on periodically to force it to renew the certificate if needed. I use the following automation to this end

# Start Let's Encrypt every night to force renewal of certificate
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt

Note that running this every day is probably overkill as certbot will renew it as soon as it is below 30 days left, but it does not really matters.

@ludeeus , maybe we can add this small snippet to the addon documentation while we wait for a more integrated solution?

[ludeeus]

That will only solve half the issue.
All services (like Home Assistant) will still use the old certificate.

[agners]

@ludeeus hm, I guess because the SSL certificate is already loaded at that time? Ideally, HA Core should have a reload function to just reload the SSL certificate. I guess for add-ons there is no common way to handle that, probably just restarting them is fine.

[ludeeus]

Yeah, all weservers load the certificate on startup, and that will continue to be loaded (even if the file changes) until the server restarts.

[borpin]

Under normal circumstances, certbot restarts apache on renewal. I wonder if the addon could create a flag/sensor that other services monitor?

With an update cycle for core as it is, and therefore regular restarts, it is less of an issue, but thinking on that, I'd have expected the addon to start and update so now surprised I saw the expiry notice. Odd.

[ludeeus]

That would be something that should be handled by a certificate manager in the supervisor and not by this addon.

[borpin]

That would be something that should be handled by a certificate manager in the supervisor

Not sure what this was a reply to, but if my comment, it should, but obviously isn't.

As I said, under normal circumstances, certbot (certificate manager) does it for you.

[ludeeus]

It was in regards to this part

I wonder if the addon could create a flag/sensor that other services monitor?

[borpin]

So back to my point, under normal circumstances, the tool doing the certificate renewal, i.e. managing the certificates, would do the server restart (as certbot does if the right plugin is used). Because the certificate manager is actually an addon, it cannot directly control the restart, but the addon certainly could tell supervisor/HA it needs to restart.

Supervisor does not have a certificate manager AFAICS.

[ludeeus]

The addon is not aware of which services that uses the certificates.

Supervisor does not have a certificate manager AFAICS.

Correct, and when someone adds that to the supervisor it will know which service uses which certificate and can handle that.

For now, your best option (since only you know where you are using the certificates), an automation as described here https://github.com/home-assistant/hassio-addons/issues/1445#issuecomment-668254065
With potentially another action for restarts of the addons (and/or core) where you use the certificates.

[lambtho12]

That will only solve half the issue.
All services (like Home Assistant) will still use the old certificate.

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

[ludeeus]

If you only use it for homeassistant, and you can have it "randomly" restart during the night yes :+1:

[agners]

@ludeeus @borpin most web servers do also support SSL certificate reloads via signals, this avoids full blown restarts. E.g. nginx supports reload which do not lead to any downtime: https://stackoverflow.com/questions/43088363/how-nginx-reload-work-why-it-is-zero-downtime. And that is also what certbot uses by default (see https://github.com/certbot/certbot/blob/09ab4aea01aaf95a2a830ad48271aa6bd11eef84/certbot-nginx/certbot_nginx/_internal/configurator.py#L1178).

[maidau]

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

@lambtho12 please excuse me, I'm new here :-)
Which yaml file should that code be inserted? I currently have the simple once a day time based restart code placed in system.yaml and it doesn't seem to be doing a thing (I've checked the Let's Encrypt logs).

[lambtho12]

@maidau This is an automation. So it should be in automations.yaml or something like that. You could also recreate it using the automation editor of the UI (under configuration/automation) instead if you prefer. See the automation documentation for more information.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[asychev]

The solution above will not work anymore because of https://github.com/home-assistant/core/pull/42338

[agners]

@asychev you can use a template to calculate the time until expiry:

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: {{ as_timestamp(states("sensor.cert_expiry")) - as_timestamp(now()) < 30 * 24 * 60 * 60 }}
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

[borpin]

So what it needs is the following;

Restart letsencrypt addon
Restart NGINX addon (new cert shows in browser padlock)
Restart HA to update sensor.

@agners - what is the right config to restart the nginx addon (and where could I look that up for future reference?)