Ejabberd: can't load letsencrypt certificate

[OneOfOne]

What version of ejabberd are you using?

community/ejabberd 18.04-1

What operating system (version) are you using?

Arch Linux 4.17.2-1-ARCH

How did you install ejabberd (source, package, distribution)?

distro package

What did not work as expected? Are there error messages in the log? What
was the unexpected behavior? What was the expected result?

Using - "/etc/letsencrypt/live/<domain>/*.pem" worked until after reboot, now it gives me failed to decode from DER format.

I tried cat /etc/letsencrypt/live/<domain>/{privkey,fullchain}.pem > ejabberd.pem and cat /etc/letsencrypt/live/<domain>/{fullchain,privkey}.pem > ejabberd.pem like some domains suggested but I still got the same error:

2018-06-25 20:38:21.079 [error] <0.377.0>@ejabberd_pkix:add_certfile:383 failed to read certificate from /etc/ejabberd/ejabberd.pem: failed to decode from DER format
2018-06-25 20:38:21.120 [warning] <0.377.0>@ejabberd_pkix:handle_call:254 No certificate found matching '<domain>': strictly configured clients or servers will reject connections with this host; obtain a certificate for this (sub)domain from any trusted CA such as Let's Encrypt (www.letsencrypt.org)

I'm not using the acme module because those certs are used by other services on the server.

[licaon-kter]

Leave only that fullchain file, don't cat stuff.

[OneOfOne]

@licaon-kter that returns a different error:

2018-06-25 21:10:25.161 [error] <0.403.0>@ejabberd_pkix:build_chain_and_check:415 Failed to build certificate chain for /etc/ejabberd/ejabberd.pem: no matching private key found for certificate in the chain

[zinid]

Insufficient access?

[OneOfOne]

No, I confirmed the user running ejabberd can access it, I even copied the files somewhere else and gave them 666 perm just for the kicks and it still errored out.

[zinid]

Also, failed to decode from DER format means that the content of that file is broken.

[OneOfOne]

@zinid i understand that, however those files are from letsencrypt and they are used by nginx.

What confuses me is that it is a PEM file not DER.

[zinid]

PEM is just a base64 encoded DER. So ejabberd decodes it from base64 and gets crap.

[zinid]

however those files are from letsencrypt and they are used by nginx

So everything is fine, but nothing works :)

[OneOfOne]

Running openssl pkcs8 -topk8 -nocrypt -in /etc/letsencrypt/live/<domain>/privkey.pem -out key.der -outform DER works without errors.

Is there anything I can try? this was all literally working for over 2 months until I rebooted earlier today.

With

certfiles:
   - "/etc/letsencrypt/live/<domain>/*.pem" # this worked for the last 2 months...

I get:

2018-06-25 21:37:01.442 [error] <0.377.0>@ejabberd_pkix:add_certfile:383 failed to read certificate from /etc/letsencrypt/live/<domain>/privkey.pem: failed to decode from DER format

[zinid]

And if you put the following in the config

certfiles:
  - "/etc/letsencrypt/live/<domain>/fullchain.pem"
  - "/etc/letsencrypt/live/<domain>/privkey.pem"

then what error you have?

[OneOfOne]

2018-06-25 21:47:17.777 [error] <0.377.0>@ejabberd_pkix:build_chain_and_check:415 Failed to build certificate chain for /etc/letsencrypt/live/<domain>/fullchain.pem: no matching private key found for certificate in the chain

[zinid]

Maybe the chain is fucked up? Some stray certificate is added? Try to reduce the chain to a single certificate (repeat this for every certificate in the chain).

[zinid]

Or just separate the chain in several PEM files and put in the config:

certfiles:
  - "/etc/letsencrypt/live/<domain>/cert1.pem"
  - "/etc/letsencrypt/live/<domain>/cert2.pem"
  - ...
  - "/etc/letsencrypt/live/<domain>/privkey.pem"

[OneOfOne]

still the same error with just a single cert (no intermediate certs).

The main thing is privkey.pem: failed to decode from DER format, but I don't know why it's not working now but it was working before reboot.

And it works fine with nginx :-/

[zinid]

Do you have two errors or what? The one with privkey.pem and the other with fullchain.pem?

[OneOfOne]

Sorry yes, I have 2, I just realize I didn't copy/paste it correctly earlier.

[zinid]

Then unpack privkey.pem to DER using openssl command then pack it back to PEM and retry.

[OneOfOne]

$ openssl pkcs8 -topk8 -nocrypt -in /etc/letsencrypt/live/<domain>/privkey.pem -out key.der -outform DER
$ openssl x509 -in key.der -inform DER -out key.pem -outform PEM
$ openssl pkcs8 -topk8 -nocrypt -in key.der -inform DER -out key.pem -outform PEM
$ diff /etc/letsencrypt/live/<domain>/privkey.pem key.pem

Same :-/ I'm a complete loss right now.

[zinid]

Run erl console, copy-paste the following and press Enter:

{ok, _} = application:ensure_all_started(public_key),
{ok, Data} = file:read_file("/etc/letsencrypt/live/<domain>/privkey.pem"),
[DER|_] = public_key:pem_decode(Data),
public_key:der_decode(element(1, DER), element(2, DER)).

You should get something like:

{'PrivateKeyInfo',v1,
                  {'PrivateKeyInfo_privateKeyAlgorithm',{1,2,840,113549,1,1,1},
                                                        {asn1_OPENTYPE,<<5,0>>}},
                  <<48,130,...>>,
                  asn1_NOVALUE}

If you get other stuff, show it here.

[OneOfOne]

{'RSAPrivateKey','two-prime',
1926983751828460458981779546............... 

[zinid]

Ah, so it's not packed into Key Info format, I think openssl can do that, could you pack it into KeyInfo format using openssl?

[zinid]

I don't remember how they are called exactly. One is called PKCS#1 and another is PKCS#8. This shit is freaking me out.

[OneOfOne]

I'm not sure how, any pointers? pkcs12 outputs a binary format, pkcs8 is the same.

Both gives me failed to decode from DER format.

[zinid]

There is definitely a bug in ejabberd: it tries to decode RSAPrivateKey from DER (here)
Kinda double decoding :) So as a work-around you should repack it into PKCS#1 (or whatever it's called, the format produced by openssl genrsa).

[zinid]

I'm not sure how, any pointers?

I think you should convert into pkcs1

[OneOfOne]

I finally got it to work, thank you for the help.

I hope that gets fixed in the next release.

$ openssl rsa -in /etc/letsencrypt/live/<domain>/privkey.pem -out key.rsa

Then in in /etc/ejabberd/ejabberd.yml:

certfiles:
   - "/etc/letsencrypt/live/<domain>/fullchain.pem"
   - "/etc/ejabberd/key.rsa"

[zinid]

Yeah, seems like this works too:

$ openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in /tmp/pkcs1.pem -out /tmp/pkcs8.pem

[zinid]

I will try to fix the issue by 18.06, thanks for the report.

[zinid]

Wait, what output do you get if you run the following in erl (using your "broken" key):

{ok, _} = application:ensure_all_started(public_key),
{ok, Data} = file:read_file("/etc/letsencrypt/live/<domain>/privkey.pem"),
public_key:pem_decode(Data).

(replace all data enclosed into angular brackets with <<...>>)

[zinid]

Also, what header do you have in your "broken" privkey.pem file? (the stuff started with "BEGIN ...")

[OneOfOne]

[{'PrivateKeyInfo',<<...>>,
not_encrypted}]

and -----BEGIN PRIVATE KEY----- (this is generated by certbot on Arch Linux).

[zinid]

Then I was wrong, the bug is in another place...

[zinid]

Would you help me to debug a little bit later? I would really appreciate that, because now I'm not sure why DER decoding of that stuff in <<...>> fails. I will be giving you Erlang commands and you will execute them and show the output, easy. This is not urgent.

[OneOfOne]

sure

[mytbk]

Is it related to erlang 21.0? I didn't have this problem before, but yesterday Arch updated the erlang packages.

[zinid]

For those with the same problem, copy the certificate key into /tmp/privkey.pem (for the convenience) and run the following inside Erlang shell:

[_|_] = rr(public_key),
{ok, _} = application:ensure_all_started(public_key),
{ok, Data} = file:read_file("/tmp/privkey.pem"),
[{'PrivateKeyInfo', Der, _}|_] = public_key:pem_decode(Data),
#'PrivateKeyInfo'{privateKey = Key} = public_key:der_decode('PrivateKeyInfo', Der),
public_key:der_decode('RSAPrivateKey', Key).

You will most likely get a crash, show it here. Remember to replace all "binaries" with <<...>> and strip all looooong intergers - this is the sensitive information.

[zinid]

Guys, this is the last issue in 18.06 milestone. If I get no info by Friday this will be postponed to 18.08.

[nosnilmot]

I can reproduce:

Eshell V10.0.1  (abort with ^G)
1> [_|_] = rr(public_key),
1> {ok, _} = application:ensure_all_started(public_key),
1> {ok, Data} = file:read_file("/tmp/privkey.pem"),
1> [{'PrivateKeyInfo', Der, _}|_] = public_key:pem_decode(Data),
1> #'PrivateKeyInfo'{privateKey = Key} = public_key:der_decode('PrivateKeyInfo', Der),
1> public_key:der_decode('RSAPrivateKey', Key).
** exception error: no match of right hand side value #'RSAPrivateKey'{version = 'two-prime',
                                                                       modulus = <snip>,
                                                                       publicExponent = 65537,
                                                                       privateExponent = <snip>,
                                                                       prime1 = <snip>,
                                                                       prime2 = <snip>,
                                                                       exponent1 = <snip>,
                                                                       exponent2 = <snip>,
                                                                       coefficient = <snip>,
                                                                       otherPrimeInfos = asn1_NOVALUE}

[zinid]

Can someone please try this patch?
Note: it should be applied on master branch.

[nosnilmot]

Can someone please try this patch?

That patch solves it for me

[OneOfOne]

https://bugs.archlinux.org/task/60154

Thanks.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.