Webmin: DNSSEC SOA expired. Webmin should renew prior to expiration date.

[chris001]

The automatic DNS checker tool on dnsstuff.com has uncovered a bug.

Isn't the webmin code supposed to check for soon to be expired DNSSEC SOA records for hosted domains, and renew them when the current time is less than X hours before expiration time of the record?

    DNSSEC SOA record date check
    DNSSEC SOA date has expired. This is bad because any signed data is now considered Bogus (RFC4033 section 5) and cannot be validated (RFC4641 section 4.1.1).

ns1.mydomain.com. has an expiration date of 20161207081119 | year=2016 month=12 day=07
ns2.mydomain.com. has an expiration date of 20161207081119 | year=2016 month=12 day=07

[chris001]

I suspect this bug (DNSSEC SOA records were allowed to expire) is behind the reason why google public DNS servers are refusing to return DNS results for the domain, google DNS follows the RFC and will not return data considered to be Bogus as defined by RFC4033 section 5.

[jcameron]

Yes, the BIND module in Webmin should automatically renew DNSSEC records by re-signing the zone.

If you SSH in as root and run /etc/webmin/bind8/resign.pl does it output any error message?

[chris001]

# /etc/webmin/bind8/resign.pl
Use of uninitialized value $ARGV[0] in string eq at /usr/share/webmin/bind8/resign.pl line 16.

[jcameron]

Try running it at /etc/webmin/bind8/resign.pl --debug

[chris001]

Today, dnsstuff.com is showing the DNSSEC SOA records test is matching, verified, and PASS.
Probably the /etc/webmin/bind8/resign.pl ran fine, despite how it complained Use of uninitialized value $ARGV[0] in string eq....

[jcameron]

Is there a cron job on your system that runs resign.pl ? You can check at System -> Scheduled Cron Jobs.

[chris001]

Yes, every day at 16:00, there's a cron job to run resign.pl.

[jcameron]

Odd that it did the re-sign when run manually but was never run by cron then ..

If you check the Cron log /var/log/cron (or similar), does it show resign.pl being run?

[chris001]

I'd like to show you the specific DNSSEC domain report from dnsviz, it contains details about exactly why the DNSSEC is fail for this particular domain. As it is, it's not too obvious whether this is a bug in webmin's re-signing, or a bug in what webmin chooses to re-sign (possibly webmin omitted to re-sign something it should re-sign)...
For privacy and security, rather than posting publicly the website name on github, would it be possible to email you directly, the link for the dnzviz DNSSEC report for this domain?

[jcameron]

Sure, you can email me at [email protected]

[chris001]

Thank you! Email sent!

[chris001]

The point where the DNSSEC fails, is in the delegation from mydomain.com to server1.mydomain.com - both of which DNS are managed/hosted by webmin bind9 as primary DNS.

Description:    Delegation from mydomain.com. to server1.mydomain.com.
Status: BOGUS
Errors: 

*    No valid RRSIGs made by a key corresponding to a DS RR were found covering the DNSKEY RRset, resulting in no secure entry point (SEP) into the zone. 
(IPv4 #1 of webmin bind9 dns server, 
IPv4 #2 of webmin bind9 dns server, 
external dns server #1, 
external dns server #2, 
external dns server #3, 
external dns server #4, 
external dns server IPv6 #1, 
external dns server IPv6 #2, 
external dns server IPv6 #3, 
external dns server IPv6 #4, 
IPv6 #1 of webmin bind9 dns server, 
IPv6 #2 of webmin bind9 dns server, 
UDP_0_EDNS0_32768_4096, UDP_0_EDNS0_32768_512)

*    The DS RRset for the zone included algorithm 7 (RSASHA1NSEC3SHA1), but no DS RR matched a DNSKEY with algorithm 7 that signs the zone's DNSKEY RRset. 
(IPv4 #1 of webmin bind9 dns server, 
IPv4 #2 of webmin bind9 dns server, 
external dns server #1, 
external dns server #2, 
external dns server #3, 
external dns server #4, 
external dns server IPv6 #1, 
external dns server IPv6 #2, 
external dns server IPv6 #3, 
external dns server IPv6 #4, 
IPv6 #1 of webmin bind9 dns server, 
IPv6 #2 of webmin bind9 dns server, 
UDP_0_EDNS0_32768_4096, UDP_0_EDNS0_32768_512)

[chris001]

@jcameron
Here's documentation with great detail (and many working scripts) on how implement DNSSEC in BIND 9 on centos and debian. There's considerable room for error! This page should help troubleshoot and fix the bug.
https://www.digitalocean.com/community/tutorials/how-to-setup-dnssec-on-an-authoritative-bind-dns-server--2

[jcameron]

Ok, I will look into this and update this bug once I have a fix..

[chris001]

I have the fix.
BIND 9 can be told to automatically resign records, and rotate in the new zone signing key!

Reference: https://www.eriklundblad.com/log/post/automatic-dnssec-signing-in-bind/

With BIND 9.9, DNSSEC has become easier to maintain and can almost be fully automated, including ZSK rollover. Configure smart signing and inline-signing in named.conf:

options {
  # dnssec keys directory:
  key-directory "/etc/pki/dnssec-keys";
  ...
};

And for the zone to be signed:

zone eriklundblad.com {
  type master;
  file "/var/named/master/eriklundblad.com”;
  auto-dnssec maintain;
  inline-signing yes;
};

auto-dnssec maintain enables automatic key management. The zone is automatically signed and keys are published. Also, the zone is re-signed on a regular basis, before the existing signature expires.

inline-signing yes signs the zone transparently, leaving the original zone file untouched by holding the zone with DNSSEC signatures in memory.

Reload the configuration:

$ rndc reconfig

For automatic ZSK rollovers, generate a new key which is a successor to the old key, and pre-publish it to the zone. First, set the old key to expire:

$ dnssec-settime -I +172800 -D +345600 Keriklundblad.com.+005+12332.key

-I inactivation interval is set to 2 days

-D deletion interval is set to 4 days

$ dnssec-keygen -S Keriklundblad.com.+005+12332.key -i 172800

-i pre-publication inverval is set to 2 days

-S successor key to the old key, Keriklundblad.com.+005+12332.key

The new key is picked up by BIND, which scans the key directory every hour, and is automatically published. After two days, the old key is inactive and the new key is used. After four days, the old key is removed from the zone. Add this to cron and run e.g. once every three months.

[chris001]

Webmin is showing this error, after you edit the zone file records, then click Save and Close:
DNSSEC signing after records change failed : dnssec-signzone: warning: /var/lib/bind/mydomain.com.hosts:51: signature has expired dnssec-signzone: fatal: 'server1.mydomain.com': found DS RRset without NS RRset
It appears dnssec-signzone has found 2 errors in the domain data files.
First, a signature has expired in the superdomain hosted by webmin.
Second, for the subdomain, there's a Resource Record set of Delegation Signer records, without a Resource Record set of Name Server records.

[chris001]

1. In trying to debug this DNSSEC issue I went to see if virtualmin.com is passing DNSSEC validation and DNSSEC's not enabled on the domain.
   http://dnssec-debugger.verisignlabs.com/virtualmin.com
   If you would, could you activate DNSSEC on virtualmin.com , then the validation errors seen here will be more immediately visible to you, and probably, easier/faster to fix.

2. The Let's Encrypt cert for virtualmin.com expired Dec 31 2016 and wasn't auto renewed:
   https://virtualmin.com:10000/
   
   So the DANE TLSA validation is impossible to verify on an expired cert, making the feature also impossible to test as a comparison basis.

[chris001]

Update: Went to registrar (godaddy) to add the 2nd DS record - it was previously not possible to add a second DS record because algorithm 8's digest type 1 was not supported at godaddy and now it is. It used to support only digest type 2. In adding this DS record, the godaddy control panel analyzed the zone (mydomain.com) and returned a warning, the zone does not contain a valid ZSK (zone signing key). Could this be a bug, webmin maybe isn't getting bind9 to generate the ZSK for the zone?

[chris001]

I've gotten a lot further into fixing this bug.

1. Fix the cron job by adding --debug to the daily command /etc/webmin/bind8/resign.pl --debug so that the resign.pl script will run to the end without halting on the very first error it encounters.
2. Make sure these settings are present in the options section of /etc/bind/named.conf.options:

        key-directory "/var/lib/bind";  // dnssec keys directory.  Keys are "K(domain).(serial).key"
        dnssec-validation auto;
        auth-nxdomain no;  //conform to RFC1035. Do NOT answer authoritatively if domain does not exist on this server, it might exist elsewhere.

1. Run service bind9 reload.
2. These errors print to the console:

Jan 15 21:09:27 server1 named[12678]: zone i-domain.com/IN (signed): reconfiguring zone keys
Jan 15 21:09:27 server1 named[12678]: dns_dnssec_findmatchingkeys: error reading key file Ki-domain.com.+007+16410.private: permission denied
Jan 15 21:09:27 server1 named[12678]: dns_dnssec_findmatchingkeys: error reading key file Ki-domain.com.+007+27260.private: permission denied
Jan 15 21:09:27 server1 named[12678]: dns_dnssec_findmatchingkeys: error reading key file Kb-domain.com.+007+55169.private: permission denied
Jan 15 21:09:27 server1 named[12678]: dns_dnssec_findmatchingkeys: error reading key file Kpm.g-domain.com.+007+46094.private: permission denied
Jan 15 21:09:27 server1 named[12678]: zone pm.g-domain.com/IN (signed): next key event: 15-Jan-2017 22:09:27.891
Jan 15 21:09:27 server1 named[12678]: zone b-domain.com/IN (signed): next key event: 15-Jan-2017 22:09:27.891
Jan 15 21:09:27 server1 named[12678]: zone i-domain.com/IN (signed): next key event: 15-Jan-2017 22:09:27.893
Jan 15 21:09:27 server1 systemd[1]: Reloaded BIND Domain Name Server.
Jan 15 21:09:27 server1 rndc[24846]: server reload successful

1. Take a close look at the K*.private files giving the permission denied error:
   # ls -la /var/lib/bind | grep "K*.private" | grep root | more
   All the K*.private ownership are root:root and -rw---- (0600) which explains why user bind running named cannot access them.

A Solution: The script which creates these files, presumably resign.pl, needs to set the K*.private files' owner to bind, or the group can be bind and permission relaxed to 0640 -rwr--- so that the bind9 process named can access them and thereby work properly.

[chris001]

# chown -R bind. /var/lib/bind
# chmod u+s /var/lib/bind
# chmod g+s /var/lib/bind
# service bind9 reload
# service bind9 status

It should show a clean reload now, no errors.

[jcameron]

OK, the current 1.831 release of Webmin should fix the resign.pl problem where it can fail if --debug is not set. And the next release will fix the problem of key files not having the right ownership for BIND to read.

[chris001]

I confirm from testing just now, after propagating the zones to the 2nd level external redundant DNS service (webmin is the primary authoritative DNS server), this fix fixes the bug 99%, enough for the domains' web mand email to work again.
In case you don't know, the google public DNS servers - and many tier 1 commercial ISPs e.g. comcast in the US - are now enforcing DNSSEC verification. This means, if your DNSSEC signatures fail cryptographic checks, their DNS servers discard the results and don't forward them to their subscribers/users. They interpret crypto fail as bogus probably fraudulent forged domains, and for the past couple of week so, their public DNS servers were NOT returning ANY DNS results for these mismanaged domains which appeared forged due to having either incorrect or expired hashes. This stricter policy has effectively removed websites and mail servers from the internet!! Nobody using their DNS (including the massively popular google public dns) can resolve your website or mail server names. From the vast majority of the internet, you've been getting no mail, no websites, no joy.

Here's a screenshot of the DNS test after applying my fix, detailed above. This passes the checks with good enough of a score to let DNS, web, mail, etc., work again.

[chris001]

Delegation Errors, SOA Warnings:
http://dnscheck.pingdom.com/?domain=virtualmin.com&timestamp=1484590412&view=1
Fine:
http://dnscheck.pingdom.com/?domain=webmin.com&timestamp=1484590550&view=1

Error checking b-domain.com on dnscheck.pingdom.com : No valid signatures for the SOA RRset for the zone was found - make sure the zone is signed with a valid and published key.
Where does the SOA resouce records RRset get signed? Does wembin create the RRSIG or does webmin expect BIND to automatically calculate the RRSIG for SOA records?

[jcameron]

I'm not sure what's up with the SOA record - Virtualmin just uses the dnssec-signzone command to generate the signatures. Is there something special about the way the SOA is signed?

[chris001]

Instead of manually signing zones, would it be possible to activate the automatic inline zone signing and key rollover, which is easily turned on in BIND 9.9 with a few settings added into the main BIND config file and into each zone file.

Details on how to do it here:
https://securityblog.switch.ch/2014/11/13/dnssec-signing-your-domain-with-bind-inline-signing/

This would eliminate the possibility of the bug currently showing up here from reoccurring. The expiration timestamp is 5 days ago, probably because of the permissions on the /var/lib/bind/*.hosts and dsset-* files were incorrect - too restrictive - and dnssec-signzone couldn't access the files, and wasn't made to handle such a corner case properly, got itself into a stalemate situation, the files have records with expired signatures, yet the tool cannot update the records because the records are expired.

Failed to sign zone : 
dnssec-signzone: warning: /var/lib/bind/e-domain.com.hosts:44: signature has expired 
dnssec-signzone: fatal: 'server1.e-domain.com': found DS RRset without NS RRset 

[chris001]

A couple more good references.
No more expired DNSSEC zones with BIND 9.9 Inline signing
https://t37.net/no-more-expired-dnssec-zones-with-bind-9-9-inline-signing.html

Inline Signing with NSEC3 (required by high security environments - govt, banks, corps, health care, legal) in BIND 9.9 A Walkthrough:
https://deepthought.isc.org/article/AA-00711/0/In-line-Signing-With-NSEC3-in-BIND-9.9-A-Walk-through.html

[jcameron]

BIND 9.9's zone signing feature sounds cool - however, unless a majority of Linux distros include it, I will likely stick with the "manual" resigning process for a while as it will work on more systems.

[chris001]

Good point. As far as I can tell, BIND 9.9 or 9.10 is included in CentOS/Redhat/Scientific, Ubuntu/Mint,/Debian. and FreeBSD, isn't it? How about a compromise, if BIND version is less than 9.9, keep running the manual resigning cron task, if BIND version is equal to or greater than 9.9, add the new settings to the BIND conf file and to each zone file, reload/restart the BIND service, and let BIND handle inline signing.

[jcameron]

I'll put this on my TODO list to investigate implementation - it's not an unreasonable idea.

[chris001]

Good, because I've experienced DNSSEC zones, hosted on BIND 9 and manually signed with DNSSEC by virtualmin, expire, and when they expire, the domain is considered spoofed/fraudulent, and consequently, all validating DNS servers such as google public DNS and comcast US public DNS, will return "non existent domain". The domain disappears from the internet, until you notice and dig up the IP address and ssh port of the server, ssh in and manually resign it on the command line. No email in or out, no virtualmin web interface, no DNS, no FTP, no nothing. With inline signing there's less moving parts involved, so, less chance for something to come unglued.

[jodenis]

@jcameron I experienced a DNSSEC expiration issue and found this thread. You are referring to a resign.pl command, but that is not part of the ubuntu/debian package. What command or mechanism resigns the zones for debian/ubuntu?

I am running webmin 1.852 and virtualmin 5.99

[swelljoe]

@jodenis why old versions? Can you try again with current versions of Webmin and Virtualmin? (I don't know that there's been any fixes specific to this area, but it's best to run the latest version, anyway.)

[jcameron]

Webmin does it's own re-signing of zones, and it should happen every 21 days by default (regardless of expiry time).

[jodenis]

@swelljoe I am updating to the latest, and see if that resolves issue(s). Knock on wood..

@jcameron, is the re-signing done via cron or internal to webmin? (I don't see any cron entries that would do this)

My goal here is just to prevent Google public DNS from doing a SERVFAIL on the domain if zone is expired or invalid. I found that several major providers are pointing their broadband customers to Google DNS for resolution.

I am putting a link to the article from Google for future folks researching this problem:
https://developers.google.com/speed/public-dns/docs/troubleshooting

[chris001]

The scary part of this bug was, the end result of expired invalid DNSSEC information in webmin's DNS, was to take email and websites offline, without any notification to the system admin.

Maybe webmin/virtualmin could add a DNSSEC domain health page, showing all hostnames hosted on the system, and the current validity expiration status of each hostname's DNSSEC info. Green light if the hostname has a chain of valid signatures up to the root and is therefore resolving on the internet / Red light if not, with text status explaining where the chain is broken, and buttons to renew all or renew individual crypto signatures within the system's ability to renew, obviously it may not be able to renew the DS records at the registrar, although there's a new protocol which does allow that to be automated by the system and no longer have to have the user copy paste their DS keys to the registrar.

[jcameron]

The renewal is done by a scheduled process that's internal to webmin - so if webmin itself were shut down, renewal wouldn't happen.

In your case, did the renewal happen eventually - just not soon enough?

[jodenis]

@chris001 Exactly what happened to me. DNS w/o DNSSEC was working fine, on not noticeable doing a dig with out the +dnssec flag. So your idea of some indicator within webmin would be a much needed indicator to identify issues and having a single source for your servers health.

@jcameron I second chriss001's line of thought on a indicator of some sort showing that DNSSEC is broken somewhere along the delegation.

And again for the future folks coming across this:
Also of note is to ensure that your Webmin->Servers->BIND DNS Server-> DNSSEC Key Re-Signing has enabled "Automatic re-signing".

[jcameron]

We could add a warning for DNS zones whose signing has expired, similar to what's done for webserver SSL certs.

[chris001]

Yes a warning would be a drastic improvement.

I can't think of a single scenario where you wouldn't want to know that one of the domains hosted on your server is no longer accessible, neither by email, nor by web, nor by ftp, from the vast majority of internet DNS servers, which drop domains ("NXDOMAIN Non existent domain" is the precise error they return) that are carrying an invalid or expired secure signature.

Even worse, if you're running an authoritative DNS server on wembin or virtualmin, and ITS DNSSEC records expire before renewal, then it and all of the domains it's responsible for, could potentially become inaccessible, and only because the records weren't re-signed automatically like they should've been. Good thing BIND 9.9+ has the built in feature to automatically sign and re-sign of all of the DNSSEC records. It's a daemon so it's always running so it's in the position to do this without fail.

[jcameron]

Ok, in the next Webmin release, expired DNSSEC domain signatures will display a warning on Webmin's system information page.