Docker-mailserver: Using Let's encrypt certs obtained with Caddy, not working so far

[qdm12]

Context

I am trying to setup this project, everything seems to work but tls certificates.
I use Caddy to renew automatically Let's Encrypt certificates (private.key and public.crt).
I have digged into existing issues but haven't found an answer really.

The container has hostname mail and DNS configuration also uses mail for the FQDN.
I use the environment variables:

SSL_TYPE=manual
SSL_CERT_PATH=/tmp/ssl/cert/public.crt
SSL_KEY_PATH=/tmp/ssl/private/private.key

With my docker-compose.yml volumes as

- ../caddy/data/acme/acme-v02.api.letsencrypt.org/sites/mail.domain.tld/mail.domain.tld.crt:/tmp/ssl/cert/public.crt:ro
- ../caddy/data/acme/acme-v02.api.letsencrypt.org/sites/mail.domain.tld/mail.domain.tld.key:/tmp/ssl/private/private.key:ro

And verified with docker exec that these are present and right, and also copied properly to /etc/postfix/ by the startup script.

All ports are also port forwarded on my router, DNS settings should be correct as well.

Expected Behavior

1. Running docker exec mail openssl s_client -connect 0.0.0.0:25 -starttls smtp -CApath /etc/ssl/certs/ should give some certificates information and no error
2. Logging in with my Gmail Android app should not fail, using [email protected], IMAP, with incoming and outgoing username and server as [email protected] and mail.domain.tld respectively.

Actual Behavior

1. Gives the following message:

   CONNECTED(00000003)
   140342221178112:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1543:SSL alert number 40
   ---
   no peer certificate available
   ---
   No client certificate CA names sent
   ---
   SSL handshake has read 294 bytes and written 326 bytes
   Verification: OK
   ---
   New, (NONE), Cipher is (NONE)
   Secure Renegotiation IS NOT supported
   Compression: NONE
   Expansion: NONE
   No ALPN negotiated
   Early data was not sent
   Verify return code: 0 (ok)
   ---
   

2. I get the warning message in my app Email security not guaranteed... The highest security available may not be supported. In the Docker logs I get (all at the same second I try to login):

   mail postfix/submission/smtpd[2064]: connect from er4.x[192.168.2.1]
   mail postfix/submission/smtpd[2064]: SSL_accept error from er4.x[192.168.2.1]: -1
   mail postfix/submission/smtpd[2064]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:1419:
   mail postfix/submission/smtpd[2064]: lost connection after STARTTLS from er4.x[192.168.2.1]
   mail postfix/submission/smtpd[2064]: disconnect from er4.x[192.168.2.1] ehlo=1 starttls=0/1 commands=1/2
   mail postfix/submission/smtpd[2064]: connect from er4.x[192.168.2.1]
   mail postfix/submission/smtpd[2064]: lost connection after UNKNOWN from er4.x[192.168.2.1]
   mail postfix/submission/smtpd[2064]: disconnect from er4.x[192.168.2.1] unknown=0/2 commands=0/2
   

Possible Fix

• The certs are definitely picked up by postfix as if I give another domain certs, I will obtain a warning in my Gmail app about it being for another domain.
• The certificate (public.crt) contains two certificates (two BEGIN CERTIFICATE), is this expected
• The private key (private.key) is an EC key

Steps to Reproduce

Try using certificates obtained from Caddy and its Lets encrypt implementation.
There might be an issue with my configuration as well.

Your Environment

• Amount of RAM available:
• Mailserver version used: latest (pulled 2 days ago)
• Docker version used: 19.03.5-ce

[hoerbi1000]

@qdm12 i have the same issue with caddy and mailserver.

my docker-compose looks like this:

mail:
    image: tvial/docker-mailserver:latest
    restart: always
    hostname: mail
    domainname: domain.tld
    container_name: mail
    ports:
      - "25:25"
      - "143:143"
      - "587:587"
      - "993:993"
      - "4190:4190"
    volumes:
      - ./data/mail/data:/var/mail
      - ./data/mail/state:/var/mail-state
      - ./data/config:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - /home/app/main/data/caddycerts/acme/acme-v02.api.letsencrypt.org/sites/mail.domain.tld:/tmp/ssl:ro
    environment:
            #- ENABLE_SPAMASSASSIN=1
      - ENABLE_CLAMAV=1
      - ENABLE_FAIL2BAN=1
      - ENABLE_POSTGREY=1
      - ONE_DIR=1
      - DMS_DEBUG=0
      - ENABLE_MANAGESIEVE=1
      - SSL_TYPE=manual
      - SSL_CERT_PATH=/tmp/ssl/mail.domain.tld.crt
      - SSL_KEY_PATH=/tmp/ssl/mail.domain.tld.key
      - POSTFIX_INET_PROTOCOLS=ipv4
      - PERMIT_DOCKER=connected-networks
    networks:
     - mail_nw
    cap_add:
      - NET_ADMIN 

[erik-wramner]

This is not my area, I'm not using letsencrypt. However, the log says no shared cipher. Have you verified the connection with openssl?
openssl s_client -starttls smtp -crlf -connect yourhost:25
It may be that your client does not support modern ciphers.

And @hoerbi1000 you say SSL_TYPE manual but write that you are using letsencrypt. That may be right, but it sounds a bit odd?

[hoerbi1000]

@erik-wramner im saying SSL_TYPE manual, because i add my letsencrypt ceritificates generated by a other docker_container. so i bind the certificates in via ssl_type manual.

[qdm12]

Hi @erik-wramner thanks for your reply!

Running the command returns the same error message, even with TLS_LEVEL=intermediate:

docker exec mail openssl s_client -starttls smtp -crlf -connect mail.domain.com:25
140279144157248:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1407:SSL alert number 40
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 251 bytes and written 209 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1586015524
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

I also verified files are as expected with:

docker exec mail cat /etc/postfix/ssl/cert
docker exec mail cat /etc/postfix/ssl/key

Maybe the key/certificate format is not compatible with postfix? Would you know the openssl command to verify that perhaps? Thanks!

[casperklein]

You could try to get a certificate with a different tool, e.g. acme.sh and see if that works.
For me, that works flawless.

[qdm12]

Agreed, problem is my reverse proxy is already occupying ports 80 and 443 so it might be harder/more complex/more maintenance to have another LE instance just for my mail server 😢

It seems also that at least 2 other people had the same issue as me with Caddy obtained certificates and docker-mailserver, so it would be interesting to solve. I will dig into it more, although I'm not a postfix/openssl expert.

I also tried with openssl 1.1.1 but it still gives the same error.

One last question, if anyone out here is using custom certs & key, for me it looks like:

/etc/postfix/ssl/key:

-----BEGIN EC PRIVATE KEY-----
MH....
...
...rU7Vw==
-----END EC PRIVATE KEY-----

/etc/postfix/ssl/cert:

-----BEGIN CERTIFICATE-----
MIIEhzCCA...
...
...pyjCFtDq9Rk=
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIEkjCCA...
...
...Fu0Qg==
-----END CERTIFICATE-----

Note: Trailing new line and each certificates and key are on multiple lines.

Does that look fine with you? Maybe postfix doesn't support such key/certificates/both.

[casperklein]

Does that look fine with you?

Yes. Same here and no problems.

I've generated a certificate for testing: removed

Of course, wrong common name, but you can try and see if that loads properly.

[qdm12]

Differences I see with what you generated:

• I have an EC private key (only 3 lines) vs your RSA private key (26 lines)
• My first certificate is 24 lines vs 28 lines and my second is 28 vs 24 lines (maybe flipped around?)

I will first try with an RSA key, maybe the EC key does not work.

[qdm12]

I tried with the RSA key and certificates you gave, same error 😢

Running the command docker exec mail openssl s_client -starttls smtp -crlf -connect mail.domain,com:25, in the logs I can see:

21:55:50 mail postfix/postscreen[1067]: CONNECT from [10.0.0.42]:41794 to [10.0.0.42]:25
21:55:50 mail postfix/postscreen[1067]: WHITELISTED [10.0.0.42]:41794
21:55:50 mail postfix/smtpd[1076]: connect from mail.domain.com[10.0.0.42]
21:55:50 mail opendmarc[210]: ignoring connection from mail.domain.com
21:55:50 mail postfix/smtpd[1076]: lost connection after UNKNOWN from mail.domain.com[10.0.0.42]
21:55:50 mail postfix/cleanup[1082]: DE0311DD97: message-id=<[email protected]>
21:55:50 mail postfix/smtpd[1076]: disconnect from mail.domain.com[10.0.0.42] ehlo=1 starttls=0/1 unknown=0/3 commands=1/5
21:55:50 mail postfix/qmgr[968]: DE0311DD97: from=<[email protected]>, size=1128, nrcpt=1 (queue active)
21:55:50 mail postfix/local[1084]: warning: required alias not found: postmaster
21:55:50 mail postfix/local[1084]: DE0311DD97: to=<[email protected]>, orig_to=<postmaster>, relay=local, delay=0, delays=0/0/0/0, dsn=2.0.0, status=sent (discarded)
21:55:50 mail postfix/qmgr[968]: DE0311DD97: removed

Maybe someone can find something wrong in those logs

[casperklein]

So you are still getting no peer certificate available? That's weird.

This is how it looks for me (without an error), when using the test certificate:

docker exec mail openssl s_client -starttls smtp -crlf -connect localhost:25

depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = i59f4b706.versanet.de
verify return:1
250 DSN
DONE
CONNECTED(00000003)
---
Certificate chain
 0 s:/CN=i59f4b706.versanet.de
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgISBEDeHC/NCWp34CczN0ib71NgMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA0MDQxNjI4MTRaFw0y
MDA3MDMxNjI4MTRaMCAxHjAcBgNVBAMTFWk1OWY0YjcwNi52ZXJzYW5ldC5kZTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKuBKDM3CD6yTWNBk4Mw+MFc
HAiFByVhRqcwX90V44y6uur9nU8cvWhM9Uj+PsOOrXrkVpO0eYXFi45CdnK4t7UA
liG3XSifuK9VniXBmc8Nh9OEhti0wxtlYwUYlEY1l0FaTlWfvwiUG/UAfpb1EkCq
FfcXmUqijGdbBm56AB3eMge5CLF+Bg4odsrHJLYbsT/LK1OqRl8wz4o48h5lMEYG
rVzexPVxzBf3pbOIpVMEW98zQ3yQNi0Vc5K1pGilfqnmoUTAKzbLbHzg5BBVUd0h
ERYEa0PrNYI2+k2wAmejylWhz85rN7EgsqcZKtHlmEAh+H8Hd7C3bputOflA+SMC
AwEAAaOCAmowggJmMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQURsfsoAFe7lMLqaIx
9PABy3/32s8wHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYB
BQUHAQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2Vu
Y3J5cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2Vu
Y3J5cHQub3JnLzAgBgNVHREEGTAXghVpNTlmNGI3MDYudmVyc2FuZXQuZGUwTAYD
VR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYa
aHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHy
APAAdgDnEvKwN34aYvuOyQxhhPHqezfLVh0RJlvz4PNL8kFUbgAAAXFGPGGUAAAE
AwBHMEUCIDtQvsF++Q07FTN9qYBGHESAYCTs4ijKUcXTddBJ5OHgAiEApt3Sw1eC
aVMV1SmWlzci0aQInLfJFP5CIfL/fQmIiwoAdgCyHgXMi6LNiiBOh2b5K7mKJSBn
a9r6cOeySVMt74uQXgAAAXFGPGGKAAAEAwBHMEUCIQDahA0TICg6YaDSg5fOZTHa
4jlC2baUdUXDZtGxU3CrugIgMENMJqkNkh75Oa6xX9a5jlnI0e2wiN5C3TKnzjYp
yvkwDQYJKoZIhvcNAQELBQADggEBABUefAURl5EbJJ9DXNGt6WNnjo9OGTcMNT3F
K4u9Oyuxr4GGt8r+CaM8IQuwLQHA2qi4Frv2S/KOupkMNJpxmHV/c0UE8AbUkigA
neMYjsy+Z3Enz5TqJ+5NdTDYSjFzrtei/Q7ANPQWrCw0MhEEpmsP//ho7RptRJ1k
hXmDGmJf6vpAtx21B1Nt0iPeWStGmCXPo+MeCEJig0nAnUAkXFeLzt1f5tdzf2GG
k1gAZVUZV/fjuiZ3F7QbrubsNXxvchXFnx+7y+DrNQD+ehPY+2K1tqTbF8zfs0gd
wFsgEmYAzyZ1J8f53fzjPXuWgzynOkf7Q5M/VMm/0bETWN11o7g=
-----END CERTIFICATE-----
subject=/CN=i59f4b706.versanet.de
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3453 bytes and written 335 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: A613FD9A6C8BBB36A2F59E46DE80DA793BF2D23F456D669FF76E2DABE212F9D4
    Session-ID-ctx:
    Master-Key: 7A221338863C22C273F62701B5320BD67157DFB9955EFBDBCAA7852AE45CFA87604146C38EA9E080F6172F09DF739541
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - a0 ff 2f 1f 90 8b 86 19-08 ee 09 f0 d4 c8 14 20   ../............
    0010 - 1a 84 af 19 82 83 36 76-fb fd f4 18 31 d2 48 14   ......6v....1.H.
    0020 - ea 89 48 09 c7 c0 0e 3d-89 84 2a 9e d5 69 05 5b   ..H....=..*..i.[
    0030 - 82 f3 52 83 8d b3 d5 cd-04 73 c1 bb 86 b3 77 db   ..R......s....w.
    0040 - c7 d7 c9 7f c1 1e 3d d6-a4 62 9a bf cb f3 16 6e   ......=..b.....n
    0050 - c3 78 32 a9 cc f5 90 54-8e 94 cf c4 ce 04 a4 8c   .x2....T........
    0060 - 87 09 eb 24 ec f4 8a 5e-61 53 e5 64 2b d4 6a 96   ...$...^aS.d+.j.
    0070 - 37 89 7f f5 89 a7 88 af-7b 0c 3c 77 3b 41 f4 18   7.......{.<w;A..
    0080 - 14 26 31 e1 d1 79 25 8a-79 dd 52 63 3d 89 6b 99   .&1..y%.y.Rc=.k.
    0090 - 4d f5 87 3e 4b ff 29 61-1b 25 c2 ed 9b db e7 b8   M..>K.)a.%......
    00a0 - 90 1f 3f 8c 05 8c 26 ec-65 0f 0b 48 97 c7 f6 48   ..?...&.e..H...H

    Start Time: 1586041308
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Relevant docker-compose settings:

    volumes:
    - /test/public.crt:/tmp/ssl/public.crt:ro
    - /test/private.key:/tmp/ssl/private.key:ro

    environment:
    - SSL_TYPE=manual
    - SSL_CERT_PATH=/tmp/ssl/public.crt
    - SSL_KEY_PATH=/tmp/ssl/private.key

[casperklein]

That is my basic test setup: removed

1. extract to /test or adjust paths in docker-compose.yml
2. run docker-compose up
3. test: docker exec mail openssl s_client -starttls smtp -crlf -connect localhost:25

It seems also that at least 2 other people had the same issue as me with Caddy

IMHO it's a problem with the certificates obtained with Caddy.

[erik-wramner]

Is it possible to use RSA certificates with Caddy? They are far more common, so perhaps the use of an EC certificate is the problem? Postfix 2.6+ should support it, but one never knows.

[erik-wramner]

Any progress with this one? For example, would it be possible to test manually generated EC test certificates or to change Caddy to use RSA certificates? It would be nice to find the root cause here.

[qdm12]

Hello, sorry for the delay, it turns out there was a mismatch between the test key and cert given before.

For anyone coming by, provided you bind mounted the cert and key files in /tmp/ssl/cert/public.crt and /tmp/ssl/private/private.key,
I generated a matching rsa key and cert with:

docker exec -it mail openssl req -newkey rsa:2048 -nodes -keyout /tmp/ssl/private/private.key -x509 -days 365 -out /tmp/ssl/cert/public.crt

And now, after a container restart, the following gives me details about the certificate as expected:

docker exec mail openssl s_client -starttls smtp -crlf -connect localhost:25

Coming back to EC keys, I tried generating some with:

docker exec -it mail openssl ecparam -name secp521r1 -genkey -param_enc explicit -out /tmp/ssl/private/private.key
docker exec -it mail openssl req -new -x509 -key /tmp/ssl/private/private.key -days 365 -out /tmp/ssl/cert/public.crt

Then running

docker exec mail openssl s_client -starttls smtp -crlf -connect localhost:25

does not work, with output:

140385362141248:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1407:SSL alert number 40
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 251 bytes and written 209 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1588542422
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

and logs from the main program in the container:

mail postfix/smtpd[995]: SSL_accept error from localhost[127.0.0.1]: -1
mail postfix/smtpd[995]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:1419:
mail postfix/smtpd[995]: lost connection after STARTTLS from localhost[127.0.0.1]
mail postfix/smtpd[995]: disconnect from localhost[127.0.0.1] ehlo=1 starttls=0/1 commands=1/2

[qdm12]

It also seems related to #1369

[qdm12]

For people using Caddy and/or EC keys (@hoerbi1000), you need to use RSA keys.

I changed my Caddyfile to have

https://mail.domain.com {
    tls [email protected] {
        key_type  rsa2048
    }
}

such that RSA keys are obtained from Let's encrypt and auto-renewed.

In my docker-compose.yml for docker-mailserver, I bind mount them directly:

volumes:
  - ../caddy/data/acme/acme-v02.api.letsencrypt.org/sites/mail.domain.com/mail.domain.com.crt:/tmp/ssl/cert/public.crt:ro
  - ../caddy/data/acme/acme-v02.api.letsencrypt.org/sites/mail.domain.com/mail.domain.com.key:/tmp/ssl/private/private.key:ro

Then restarting the container and running the following works as expected:

docker exec -ail popenssl s_client -starttls smtp -crlf -connect localhost:25

Before closing the issue, one last little question: @casperklein it seems you are using EC keys, any idea why my self signed generated EC key mentioned above does not work?

Thanks!

[erik-wramner]

I've updated the wiki, feel free to elaborate.

[casperklein]

@qdm12 No, I am using RSA keys (https://github.com/tomav/docker-mailserver/issues/1440#issuecomment-609064683)

[qdm12]

Let's close the issue then. Do you mind if I create another issue to support EC keys? Thanks for the help all of you!

[casperklein]

I just found this: https://community.letsencrypt.org/t/postfix-doesnt-work-since-last-update-of-certificate-08-2019/100995/13

Is your OpenSSL version on the maschine you run your tests from maybe outdated?

[qdm12]

Indeed, @dolanor (https://github.com/dolanor) had the same issue. I think I reached out to him a few weeks ago but he didn't solve it either.

The openssl version from the docker-mailserver container is OpenSSL 1.1.0l 10 Sep 2019 (which I used to run all my tests in for consistency). It should be independent from the host OS/kernel right? On my host it's OpenSSL 1.1.1d 10 Sep 2019

[dolanor]

TL;DR: key_type rsa2048 works, make sure to have an updated docker image. I use the stable tag

Hi!
I tried the trick with key_type rsa2048 and it didn't work, so I got back to the paid certificate.
But, today, my paid certificate expired… So I had to find a solution again, pay and update every year, or let LE with Caddy deal with it. Reading my posts on the forum made me think my docker image hasn't been updated/pulled in a while. So maybe I had an incompatible OpenSSL lib on it. I updated it with docker pull.
After updating my Caddyfile, the certificate didn't update to rsa2048. I figured it was because it has been created recently, and didn't feel it needed updating. And caddy (v1) didn't get the difference in key_type between execution needed a request to LE. So I deleted the previously generated LE certs from /root/.caddy/acme folder and started caddy again.
I referenced the newly LE generated certs with rsa2048 from Caddy in my mail configuration, and BOOM, it works.

Thanks all!