Mailu: Letsencrypt certificate generation

[kaiyou]

Provide a way to auto-generate and renew the TLS certificate if none is provided or if it is expired.

[kaiyou]

After some thinking, there are two main options for automating Letsencrypt certificate generation:

• either create a new Docker image and an entry to Docker-Compose;
• or insert something in the admin container.

The first option is more Docker-friendly: certbot is a long running process and would fit in a container. But it would also require some mechanism to restart other containers properly, thus access to the Docker socket.

Also, it will become necessary at some point that users can restart containers and regenerate certificates through the admin interface. Although it is not natural to fork a long-time running process in a container, it still sounds like the best solution here.

[kaiyou]

This library sounds nice: https://github.com/agronholm/apscheduler

[kaiyou]

Another option would be to move all the tasks to a separate container running Celery (and maybe Beats). Loaded handlers would include every Freeposte related task and the container would be responsible for scheduling things.

The existing redis container would be used as a broker and the scheduler would handle all non-synchronous tasks. The sqlite database would still be shared for updates.

With such a scheme, all Docker-related operations would be moved as tasks. Remaining question: how should we share models between both containers?

[kaiyou]

For Milestone 1.4, we will go with the in-app scheduler on the admin interface side. The solution is quick and dirty but will do while awaiting a more structured architecture for the whole project.

[kaiyou]

Certificate generation using certbot on Python3 seems broken for now, just pushed this, awaiting review: https://github.com/certbot/certbot/pull/3757

[kaiyou]

Just pushed a first version of the letsencrypt certification generation, feel free to test and provide feedback once the builds are available.

[kaiyou]

Still an issue: when no certificate is available, nginx will not start and Certbot will fail. Some work is probably required on the Nginx configuration.

[pesimon]

@kaiyou how could I test this? thanks

[kaiyou]

If you already have a certificate setup and you set the ENABLE_CERTBOT variableENABLE_CERTBOTvariable in your env file, it should work fine onlatest``. Remaining problem is bootstrapping: nginx won't start if no certificate is available, so use a snakeoil certificate for now.

[trueserve]

This still has some problems.

1. Requires HTTP port to be open in order to update cert. Maybe have an option for custom ports if not wanting to use the HTTP with the reverse proxy? I do not want the HTTP server to be public, but I am fine with letting Mailu run certbot for me on this server.

2. Was able to restart mailu/admin and get generated certs, but the certs weren't being moved / linked into place. Looks like you may be trying to delete files before they exist? Copying the certs into place then restarting containers made the admin interface function normally (although it looks like nginx is working with default config now, another issue - EDIT: fixed by docker-compose down, docker-compose up -d).

3. The cert symlinks are absolute, not relative, so they fail to resolve on the host filesystem.

[2017-02-03 05:09:48 +0000] [17] [INFO] Starting gunicorn 19.6.0
[2017-02-03 05:09:48 +0000] [17] [INFO] Listening at: http://0.0.0.0:80 (17)
[2017-02-03 05:09:48 +0000] [17] [INFO] Using worker: sync
[2017-02-03 05:09:48 +0000] [20] [INFO] Booting worker with pid: 20
[2017-02-03 05:09:48 +0000] [21] [INFO] Booting worker with pid: 21
[2017-02-03 05:09:48 +0000] [22] [INFO] Booting worker with pid: 22
[2017-02-03 05:09:48 +0000] [23] [INFO] Booting worker with pid: 23
Job "generate_cert (trigger: date[2017-02-03 05:09:49 UTC], next run at: 2017-02-03 05:09:49 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/app/mailu/certbot.py", line 67, in generate_cert
    if certbot_install(hostname):
  File "/app/mailu/certbot.py", line 35, in certbot_install
    os.unlink(cert)
FileNotFoundError: [Errno 2] No such file or directory: '/certs/cert.pem'
Job "generate_cert (trigger: date[2017-02-03 05:09:49 UTC], next run at: 2017-02-03 05:09:49 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/app/mailu/certbot.py", line 67, in generate_cert
    if certbot_install(hostname):
  File "/app/mailu/certbot.py", line 35, in certbot_install
    os.unlink(cert)
FileNotFoundError: [Errno 2] No such file or directory: '/certs/cert.pem'
Job "generate_cert (trigger: date[2017-02-03 05:09:49 UTC], next run at: 2017-02-03 05:09:49 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/app/mailu/certbot.py", line 67, in generate_cert
    if certbot_install(hostname):
  File "/app/mailu/certbot.py", line 35, in certbot_install
    os.unlink(cert)
FileNotFoundError: [Errno 2] No such file or directory: '/certs/cert.pem'
Job "generate_cert (trigger: date[2017-02-03 05:09:49 UTC], next run at: 2017-02-03 05:09:49 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/app/mailu/certbot.py", line 67, in generate_cert
    if certbot_install(hostname):
  File "/app/mailu/certbot.py", line 35, in certbot_install
    os.unlink(cert)
FileNotFoundError: [Errno 2] No such file or directory: '/certs/cert.pem'
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

[kaiyou]

1. I will simply try and use a temporary self-signed certificate with only letsencrypt enabled in stead of HTTP
2. You are right about the nginx config that's a dummy mistake on my side
3. Is this really an issue? I will try and switch to relative symlinks however

[trueserve]

1. Then that also requires HTTPS to be open. In my case that will work, but what about the case where there is no public web interface? Perhaps an option for http or https (for nginx reverse proxy), or other port (certbot direct) in .env would be best.

2. The issue with the cert symlinks is minor, but considering these files are fixed in position and the targets are in directories beneath them, they should be relative. It's really an issue if other services (such as external webserver) use Mailu certbot to update certs and they want to use these symlinks. Honestly though I think most would implement this the other way around.

[kaiyou]

1. I see your point, I will create a separate issue for this as one could also wish to use TLS-SNI on the SMTP port for instance. Some thinking required, maybe in 2.0 we'll have the proper flexibility for this.

2. I will fix this one before closing, and finally tagging 1.4 :)

[r3c4ll]

Hi @kaiyou and everybody. Sorry if it's not the place to post this but my cert just expire today, whats is the best procedure to renew it?

Thanks in advance.

[r3c4ll]

Ready, I only had to restart the front container.

[crytos]

hey is the auto renew cert issue solved by now, just a newbie