Ejabberd: Let's Encrypt ACME integration broken & undocumented

[RubenHoms]

Environment

• ejabberd version: 18.12.1-2~bpo9+1 (debian package)
• Erlang version: Erlang/OTP 19 [erts-8.2.1] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
• OS: Linux (Debian)
• Installed from: distro package (stretch-backports repo)

Configuration

hosts:
  - "myserver.com"

acme:
   contact: "mailto:[email protected]"
  # ca_url: "https://acme-v01.api.letsencrypt.org"

Bug description

Since November 2017 there has been support for requesting Let's Encrypt certificates with the ACME protocol which is awesome. However, the only 'documentation' about this I could find is a diff in the original pull request. For the rest it was a case of trying to figure out through the issues tracker (#2418, #2487) how this functionality 'works'.

I have tried the following to get a Let's Encrypt certificate:

acme:
   contact: "mailto:[email protected]"
   ca_url: "https://acme-v01.api.letsencrypt.org"

And then run ejabberdctl get_certificates all which puts this in the error log file:

2019-07-25 12:01:47.446 [error] <0.480.0>@ejabberd_acme:create_new_account:285 Error: {error,badarg} creating an account for contact: <<"mailto:[email protected]">>

Then with some help provided in #2487 I edited the config to look like this:

acme:
   contact: "mailto:[email protected]"
   #ca_url: "https://acme-v01.api.letsencrypt.org"

Running the ejabberdctl get_certificates all puts this in the logs:

2019-07-25 12:12:22.393 [error] <0.480.0>@ejabberd_acme:get_config_ca_url:1191 No CA url has been specified in configuration
2019-07-25 12:12:37.567 [error] <0.480.0>@ejabberd_acme_comm:get_authz_until_valid:197 Maximum request limit waiting for validation reached
2019-07-25 12:12:37.567 [error] <0.480.0>@ejabberd_acme:create_new_authorization:316 Error: {error,{badmatch,{error,max_request_limit}}} getting an authorization for domain: <<"myserver.com">>

Then I found the sample config in the original pull request for this and added this section for an HTTP listener as instructed:

listen:
  -
    port: 80
    ip: "::"
    module: ejabberd_http

But the ejabberd service won't even start with the following error being shown:

2019-07-25 13:45:25.030 [error] <0.342.0>@ejabberd_listener:report_socket_error:417 Failed to open socket at [::]:80 for ejabberd_http: permission denied

Which is probably due to the systemd service running the process as user ejabberd which does not have root privileges to listen on this port. Only root privileged users can open ports below 1024.

I'm just thoroughly confused by this and would like to have some help on how to get this working and preferrably have someone write some documentation on this. Pinging the original author (@angelhof) and reviewer (@zinid) for some input on this.

[zinid]

Set another port, let's say 5280, and forward 80 to 5280 in iptables.

[zinid]

It's actually a problem in ACME protocol itself which requires privileged ports.

[RubenHoms]

Thanks for the swift response @zinid! While having a forward in iptables would (possibly) solve the issue, I don't think it would fix the underlying issue here, which is that there is no official documentation on how this should work. As you can see I could puzzle the pieces together with some effort but having a section on this in the documentation would make sure that people at least knows this exists and how it's supposed to work.

[RubenHoms]

I just tried the iptables forward trick you proposed but I'm getting no further with it. I just can't seem to get this working. The log output after the port forward is as follow:

2019-07-25 17:00:22.935 [info] <0.625.0>@acme_challenge:register_hooks:40 Registering hook for ACME HTTP headers
2019-07-25 17:00:37.275 [error] <0.625.0>@ejabberd_acme_comm:get_authz_until_valid:197 Maximum request limit waiting for validation reached
2019-07-25 17:00:37.275 [error] <0.625.0>@ejabberd_acme:create_new_authorization:316 Error: {error,{badmatch,{error,max_request_limit}}} getting an authorization for domain: <<"my.domain.com">>
2019-07-25 17:00:37.276 [info] <0.625.0>@acme_challenge:unregister_hooks:44 Unregistering hook for ACME HTTP headers

[licaon-kter]

Maybe try the staging LE server instead, as you've hit the limit now and you must wait.

[RubenHoms]

Thanks for the suggestion @licaon-kter. However if I set the ca_url to anything (staging, live or even local) and run ejabberdctl get_certificates all it returns with Error: create_new_account and the following is put in the error.log:

2019-07-26 09:45:04.906 [error] <0.1361.0>@ejabberd_acme:create_new_account:285 Error: {error,badarg} creating an account for contact: <<"mailto:[email protected]">>

Looking at the other threads with the same issues (#2418 #2487) and having tried almost every possibility I can't conclude any different than that the ACME module for requesting Let's Encrypt certificates is broken. I would look into fixing this myself but I don't know any Erlang.

I even uninstalled the Debian package and used the Linux bash script installed from the official download page but the results are the same.

If anyone would be willing to pick this up I could help out by writing some documentation for this. :+1:

[RubenHoms]

As a workaround I'm trying to use certbot to get a wildcard certificate for my domain and point ejabberd in that direction. I successfully managed to get the certificate but after configuring it in the ejabberd.yml config file I still get errors. This is what I've got:

hosts:
  - "my.domain.com"

certfiles:
  - "/opt/ejabberd/conf/fullchain.pem"
  - "/opt/ejabberd/conf/privkey.pem"

I moved the fullchain.pem and privkey.pem to the /opt directory and chown-ed them to ejabberd:ejabberd so it could be read by the process, yet this is the error I'm getting:

2019-07-26 10:45:01.620 [warning] <0.385.0>@ejabberd_pkix:log_warnings:397 Invalid certificate in /opt/ejabberd/conf/fullchain.pem: at line 32: certificate is signed by unknown CA

Any ideas what's causing this?

[RubenHoms]

Found out what happened. A wildcard certificate from Let's Encrypt (e.g. *.mydomain.com) will only work for subdomains and not the root domain itself. (so not mydomain.com)

What I did as a workaround for this issue is to create two certificates, one wildcard certificate and one for the root of my domain. I then smashed them together and pointed ejabberd at it. Like so:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns -d *.mydomain.com
certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns -d mydomain.com
# Copy the wildcard certificate and key to the jabber directory
cp /etc/letsencrypt/live/mydomain.com/fullchain.pem /opt/ejabberd/conf/fullchain-wildcard.pem
cp /etc/letsencrypt/live/mydomain.com/privkey.pem /opt/ejabberd/conf/privkey-wildcard.pem
chown ejabberd:ejabberd /opt/ejabberd/conf/*-wildcard.pem
# Copy the root certificate and key to the jabber directory
cp /etc/letsencrypt/live/mydomain.com/fullchain.pem /opt/ejabberd/conf/fullchain-root.pem
cp /etc/letsencrypt/live/mydomain.com/privkey.pem /opt/ejabberd/conf/privkey-root.pem
# Concatenate them into one certificate file and one private key file
cat /opt/ejabberd/conf/fullchain-root.pem /opt/ejabberd/conf/fullchain-wildcard.pem >> fullchain-all.pem
cat /opt/ejabberd/conf/privkey-root.pem /opt/ejabberd/conf/privkey-wildcard.pem >> privkey-all.pem
# Set correct permissions so ejabberd can read it
chown ejabberd:ejabberd /opt/ejabberd/conf/*-all.pem

Then in the ejabberd.yml config file:

hosts:
  - "mydomain.com"
certfiles:
  - "/opt/ejabberd/conf/fullchain-all.pem"
  - "/opt/ejabberd/conf/privkey-all.pem"

This solved the issue of not having SSL connections, but it's only a workaround and the ACME integration is still broken for me.

[UnixRonin]

I solved the issue of INSTALLING the certificate as follows:

# Check SSL certs and renew as necessary
37 3,15 * * *           certbot renew --quiet
# Copy manually to Jabber, certbot post-hook doesn't work reliably
30 6,18 * * *           [[ /etc/letsencrypt/live/www.caerllewys.net/privkey.pem -nt /etc/jabber/server.pem ]] && cat /etc/letsencrypt/live/www.caerllewys.net/privkey.pem /etc/letsencrypt/live/www.caerllewys.net/fullchain.pem > /etc/jabber/server.pem

However, this does not cause ejabberd to re-read the updated certificate. Is there a simple wya to do that WITHOUT restarting ejabberd?

Disclaimer: I'm on ejabberd-17.04-r2 because I'm still waiting for the Gentoo package to complete its pending update to 18.09.

[RubenHoms]

@UnixRonin Not sure if this re-reads the certificate, but you could try reloading ejabberd by running ejabberdctl reload_config

[UnixRonin]

@RubenHoms good thought, but I already tried that and it didn't work. For the moment I've worked around the problem by having the cron job also restart ejabberd if it updates the certificate.

[RubenHoms]

@UnixRonin Glad you got it working, too bad you have to restart ejabberd for it to work. Maybe make an issue for it so we can keep the issue you're having separate from this one.

From what I've seen thusfar the Let's Encrypt integration is still broken and people have their own ways to workaround it.

[zinid]

The ACME implementation has been fully reworked and the mentioned issues are fixed. The documentation is in progress, see here.

[RubenHoms]

Super happy with this, thanks alot @zinid! :smile: I'll leave the issue open until it is merged.

[UnixRonin]

And Gentoo has a new maintainer for net-im/ejabberd who is working on updating all of the relevant packages and stabilizing a current release, starting with the dependencies. This is progressing well.

[zinid]

I'll leave the issue open until it is merged.

Once 19.09 is out (this week), I'll merge that MD document into the official docs.ejabberd.im.

[zinid]

Also, if you have any questions on the documentation (or the implementation), leave them here, until it's not too late. So I will try to improve.

[RubenHoms]

Just had a read through it, and I think there is some room for improvement. What I would love to see is some expansion on this part:

Note that the ACME protocol requires challenges to be sent on port 80. Since this is a privileged port, ejabberd cannot listen on it directly without root privileges. Thus you need some mechanism to forward port 80 to the port defined by the listener (port 5280 in the example above). There are several ways to do this: using NAT or HTTP front-ends (e.g. sslh, nginx, haproxy and so on). Pick one that fits your installation the best, but DON'T run ejabberd as root.

I would love to see some examples of how to do this, or else a link to some documentation/blog post explaining how to do this with the tools you mentioned. I think this will make it a lot clearer.

Other than that, looks good to me! :smile:

[UnixRonin]

On 9/24/19 6:18 AM, Ruben Homs wrote:

Just had a read through it, and I think there is some room for
improvement. What I would love to see is some expansion on this part:

|Note that the ACME protocol requires challenges to be sent on port 80.
Since this is a privileged port, ejabberd cannot listen on it directly
without root privileges. Thus you need some mechanism to forward port 80
to the port defined by the listener (port 5280 in the example above).
There are several ways to do this: using NAT or HTTP front-ends (e.g.
sslh, nginx, haproxy and so on). Pick one that fits your installation
the best, but DON'T run ejabberd as root.|

This implies that ejabberd will take part in ACME verification directly,
presumably in a situation in which ejabberd is the principal SSL user.
Does the revised handling support the case in which another application
such as Apache is the ACME client, and is sharing the certificate with
ejabberd, which only needs in that case to be notified that the
certificate has been updated (or be able to recognize that for itself)?

--
Phil Stracchino
Babylon Communications
[email protected]
[email protected]
Landline: +1.603.293.8485
Mobile: [phone]

[zinid]

I honestly see no point in sharing ACME certificates between daemons: you can get many certificates for your domains, so every daemon may obtain its own certificate. Especially Apache, which has built-in ACME client as well.

[UnixRonin]

On 9/24/19 10:10 AM, Evgeny Khramtsov wrote:

I honestly see no point in sharing ACME certificates between daemons:
you can get many certificates for your domains, so every daemon may
obtain its own certificate. Especially Apache, which has built-in ACME
client as well.

....Huh. I can't say that it had ever occurred to me to have separate
SSL certificates for different services running on the same host. It
seems redundant.

--
Phil Stracchino
Babylon Communications
[email protected]
[email protected]
Landline: +1.603.293.8485
Mobile: [phone]

[sschimmel]

Have the fixes mentioned by @zinid been merged and released?

I just installed a new server on Ubuntu using the distro packages and I get the same error as @RubenHoms.

@ejabberd_acme:create_new_account:284 Error: {error,{badmatch,#{"VOfQtxVJ6gE" => "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417","keyChange" => "https://acme-v02.api.letsencrypt.org/acme/key-change","newAccount" => "https://acme-v02.api.letsencrypt.org/acme/new-acct","newNonce" => "https://acme-v02.api.letsencrypt.org/acme/new-nonce","newOrder" => "https://acme-v02.api.letsencrypt.org/acme/new-order","revokeCert" => "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"}}} creating an account for contact: <<"mailto:info@*******.com">>

I'd rather not use certbot as I have multiple domains and manually merging the certificates would break the auto renew. That can be fixed with some custom scripting if need be but I'd rather use the updated ejabber.

[zinid]

The error looks like you're running an old code.

[zinid]

And yes, the fixes are merged and released in 19.09.

[licaon-kter]

@RubenHoms

I just installed a new server on Ubuntu using the distro packages

You mean 18.12.1? Mmmmnope

[RubenHoms]

@RubenHoms

I just installed a new server on Ubuntu using the distro packages

You mean 18.12.1? Mmmmnope

@licaon-kter Yes I first tried with the packages from the distro repo but when that didn't work I grabbed the latest installer script from the download page which still didn't work. I haven't tried with the newly released version.

@zinid if the changes are merged and documentation as well let me know and I'll close the issue.

[zinid]

@RubenHoms @licaon-kter was replying to @sschimmel :)
Anyway, yes the changes are all there, the issue can be closed.

[RubenHoms]

My bad @licaon-kter, misread the context.

Closing this issue, because fixes are merged in 19.09 and documentation should be available.

@zinid could you link to the relevant documentation? I couldn't find any relevant commits in the docs.ejabberd.im repository.

[sschimmel]

When will the debian packages be updated? I did an apt install yesterday and that installed version 18.

[licaon-kter]

Which Debian version?

The site provided DEBs are up to date.

[sschimmel]

I'm running Ubuntu 18.04.3 LTS. Ran an apt update and apt upgrade to check if there's an update but everything is up to date. Ejabber version installed with apt install ejabberd is 18.01.

I checked packages.ubuntu and Bionic (18.04) has version 18 of Ejabber and Ubuntu Disca (19) has Ejabber 19 available.

Ubuntu 19 is not an LTS release so I'd rather stick with Ubuntu 18. It's not in the backports repository so question is if there's a dependency for ejabber 19 that's not available in Ubuntu Bionic.