Mailcow-dockerized: acme: HTTP & IP verification lists internal Docker IP - no SAN for MAILCOW_HOSTNAME

[lazyfrosch]

Prior to placing the issue, please check following: (fill out each checkbox with an X once done)

• [x] I understand that not following or deleting the below instructions will result in immediate closure and/or deletion of my issue.
• [x] I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
• [x] I have understood that answers are voluntary and community-driven, and not commercial support.
• [x] I have verified that my issue has not been already answered in the past. I also checked previous issues.

Summary

Today my system updated Docker to 20.10.0~3, which seems to implement changes regarding DNS resolution in Docker networks.

As you can see in the log below, Docker DNS seems to resolve MAILCOW_HOSTNAME, because it is used as hostname for two containers.

Since then, the ACME client as removed the main SAN for my mail system, since it can no longer verify it. I've already reset the certificates, just to be sure it was no fluke.

NOTE: I'm still into investigation, and will update the issue accordingly!

# docker exec -it mailcowdockerized_clamd-mailcow_1 bash
root@3b221440dfda:/# host mail.lazyfrosch.de
mail.lazyfrosch.de has address 172.22.1.10
mail.lazyfrosch.de has address 172.22.1.250
mail.lazyfrosch.de has IPv6 address fd4d:6169:6c63:6f77::d
mail.lazyfrosch.de has IPv6 address fd4d:6169:6c63:6f77::e

Logs

Thu Dec 10 10:57:10 CET 2020 - Found AAAA record for mail.lazyfrosch.de: fd4d:6169:6c63:6f77::d - skipping A record check
Thu Dec 10 10:57:10 CET 2020 - Confirmed AAAA record with IP fd4d:6169:6c63:6f77:0000:0000:0000:000d, but HTTP validation failed
Thu Dec 10 10:57:10 CET 2020 - Found AAAA record for candela.lazyfrosch.de: 2a01:4f8:c0c:2230::2 - skipping A record check
Thu Dec 10 10:57:10 CET 2020 - Confirmed AAAA record with IP 2a01:04f8:0c0c:2230:0000:0000:0000:0002

Reproduction

Check your docker-ce version and try to resolve your MAILCOW_HOSTNAME from within a container, like mailcowdockerized_clamd-mailcow_1.

System information

| Question | Answer |
| --- | --- |
| mailcow GIT Commit | 6664052e2ee8e7a6acaad698d4b9c3244c19d848 - no modifications |
| My operating system | Debian Buster |
| Is Apparmor, SELinux or similar active? | no |
| Virtualization technlogy (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported | KVM - Hetzner Cloud |
| Server/VM specifications (Memory, CPU Cores) | 16GB - 4 cores |
| Docker Version (docker version) | 20.10.0~3 |
| Docker-Compose Version (docker-compose version) | 1.25.4 |
| Reverse proxy (custom solution) | no |

[lazyfrosch]

Update: I've downgraded docker-ce and containerd.io:

Start-Date: 2020-12-10  12:40:45
Downgrade: docker-ce:amd64 (5:20.10.0~3-0~debian-buster, 5:19.03.14~3-0~debian-buster), docker-ce-cli:amd64 (5:20.10.0~3-0~debian-buster, 5:19.03.14~3-0~debian-buster)
End-Date: 2020-12-10  12:41:00

Start-Date: 2020-12-10  12:43:43
Downgrade: containerd.io:amd64 (1.4.3-1, 1.3.9-1)
End-Date: 2020-12-10  12:43:58

But, I also had to docker-compose down and docker-compose up -d in order for the network to work again:

Thu Dec 10 12:46:07 CET 2020 - Found AAAA record for mail.lazyfrosch.de: 2a01:4f8:c0c:2230::2 - skipping A record check
Thu Dec 10 12:46:07 CET 2020 - Confirmed AAAA record with IP 2a01:04f8:0c0c:2230:0000:0000:0000:0002
Thu Dec 10 12:46:07 CET 2020 - Found AAAA record for candela.lazyfrosch.de: 2a01:4f8:c0c:2230::2 - skipping A record check
Thu Dec 10 12:46:08 CET 2020 - Confirmed AAAA record with IP 2a01:04f8:0c0c:2230:0000:0000:0000:0002
Thu Dec 10 12:46:08 CET 2020 - Certificate /var/lib/acme/mail.lazyfrosch.de/cert.pem missing or changed domains 'mail.lazyfrosch.de candela.lazyfrosch.de' - start obtaining

My only guess is, that hostnames are now also automatically resolved in Docker's network.

Idea for a quick fix would be to not validate MAILCOW_HOSTNAME in acme, and just try to sign it with letsencrypt.

Any thoughts on my investigation so far? I'll resume investigation tonight.

[chriscroome]

I added MAILCOW_HOSTNAME to ADDITIONAL_SAN when I had an issue like this the other day and it appeared to solve the issue of MAILCOW_HOSTNAME not being included in the cert subjectAltNames, but I don't know if this was the right thing to do...

[andryyy]

Did you run a mailcow update? We fixed it yesterday. :)

[lazyfrosch]

Whoops, sorry long day and such... I should have checked the log more

@chriscroome in fact this was the first thing I've tried...

Problem is really that the MAILCOW_HOSTNAME can not correctly be resolved by DNS, or better, not checked by HTTP because it points to an internal container.

duplicate of #3893

@andryyy Lebkuchen? :wink:

[andryyy]

Mein Bauch sagt nein, mein Kopf sagt ja.

[lazyfrosch]

@andryyy done :wink:

[mikestp27]

Hi. I had the same problem and I just updated to the latest version and it's working fine now. Thank you. More about my issue here: https://community.mailcow.email/d/508-ssl-certificate-does-not-include-mailcow-hostname