Pdns: Supermaster behind dnsdist is ignored

[dasJ]

• Program: Authoritative, dnsdist
• Issue type: Bug report

Short description

When getting notifications from a supermaster while being behind a dnsdist, PowerDNS ignores them because it's not authoritative for these zones.

Environment

• Operating system: Arch Linux
• Software version: 4.0.4
• Software source: OS Repo

Steps to reproduce

1. Have a supermaster configured
2. Replace the superslave by a new PowerDNS server behind a dnsdist
3. Use pdns_control notify example.com on the supermaster

Expected behaviour

The superslave creates the zones in the local domains table.

Actual behaviour

The superslave is not aware of the fact that it's a superslave:

Sep 01 21:17:44 localhost pdns_server[7435]: Sep 01 19:17:44 Received NOTIFY for example.com from 127.0.0.1 for which we are not authoritative
Sep 01 21:17:44 localhost pdns_server[7435]: Sep 01 19:17:44 Received NOTIFY for example.com from 127.0.0.1 for which we are not authoritative
Sep 01 21:17:45 localhost pdns_server[7435]: Sep 01 19:17:45 Error resolving SOA or NS for example.com at: 127.0.0.1: Query to '127.0.0.1' for SOA of 'example.com.' produced no answers
Sep 01 21:17:45 localhost pdns_server[7435]: Sep 01 19:17:45 Error resolving SOA or NS for example.com at: 127.0.0.1: Query to '127.0.0.1' for SOA of 'example.com.' produced no answers

Other information

dnsdist configuration:

addLocal('0.0.0.0:53', { doTCP=true, reusePort=true })
addLocal('[::]:53', { doTCP=true, reusePort=true })
addACL('0.0.0.0/0')
addACL('::/1')

setServerPolicy(firstAvailable)
newServer({
    address='127.0.0.1:5353',
    retries=5,
    name='local'
})

-- AXFR ACL
addAction(AndRule{
    OrRule{
        QTypeRule(dnsdist.AXFR),
        QTypeRule(dnsdist.IXFR)
    },
    NotRule(OrRule{
        -- localhost
        makeRule('127.0.0.1'),
        makeRule('::1'),
        -- A slave
        makeRule('10.0.0.1'),
        -- Puck
        makeRule('204.42.254.5'),
        makeRule('2001:418:3f4::5'),
    })
}, RCodeAction(dnsdist.REFUSED))

-- Notify ACL
addAction(AndRule{
    OpcodeRule(DNSOpcode.Notify),
    NotRule(OrRule{
        -- My superslave
        makeRule('10.0.0.2'),
        makeRule('fe80::2'),
    })
}, RCodeAction(dnsdist.REFUSED))

-- AXFR and Notify Target
addAction(OrRule{
    QTypeRule(dnsdist.SOA),
    QTypeRule(dnsdist.AXFR),
    QTypeRule(dnsdist.IXFR),
    OpcodeRule(DNSOpcode.Notify)
}, PoolAction(""))

Relevant parts from the PowerDNS config:

trusted-notification-proxy=127.0.0.1
master=yes
slave=yes
allow-axfr-ips=127.0.0.1
allow-unsigned-supermaster=no

Supermasters table in PostgreSQL:

    ip     |   nameserver    | account
-----------+-----------------+---------
 127.0.0.1 | example.com     | admin
 ::1       | example.com     | admin

[sshipway]

We have the same issue; given that this configuration is likely very common there should be specific support for this in pdns somehow.

Workaround 1 -

• Set up superslave to designate 127.0.0.1 as a supermaster (in the database supermasters table)
• Set up dnsdist on superslave to forward NOTIFY to pdns
• Set up dnsdist on superslave to forward IXFR, AXFR, SOA to supermaster

e.g.

newServer({address="127.0.0.1:5300", name="resolver", pool="resolver", mustResolve=false})
newServer({address="10.1.2.3:53", name="supermaster", pool="supermaster", mustResolve=false})
addAction(OpcodeRule(DNSOpcode.Notify),PoolAction('resolver'))
addAction(OrRule({QTypeRule(dnsdist.SOA),QTypeRule(dnsdist.IXFR),QTypeRule(dnsdist.AXFR)}),PoolAction('supermaster'))

Workaround 2 -

• Set resolver to listen on additional IP
• One of
  * Set NS record in zone on supermaster to specify the additional IP rather than the dnsdist one. (if you dont mind queries coming in here)
  * Set supermaster to also-notify the additional IP (if you want all secondaries to go here anyway)

[zeha]

Related: #8401 #8403

[dasJ]

Going to unsubscribe, I switched to a better DNS setup.

[influenist]

Since dnsdist, PDNS is totally none-sense...
We have bind masters which used to notify PDNS cluster. Now zones are not transfered and added to pdns because dnsdist filters this out and doest not pass...

addLocal('127.0.0.1:53')
setACL({'0.0.0.0/0', '::/0'}) -- Allow all IPs access

newServer({address='127.0.0.1:5300', pool='auth'})
newServer({address='127.0.0.1:5301', pool='recursor'})

recursive_ips = newNMG()
recursive_ips:addMask('127.0.0.0/8')
recursive_ips:addMask('xx/22')
recursive_ips:addMask('xx/22')
recursive_ips:addMask('xx/24')

addAction(NetmaskGroupRule(recursive_ips), PoolAction('recursor'))
addAction(AllRule(), PoolAction('auth'))

-- AXFR ACL
addAction(AndRule{
    OrRule{
    QTypeRule(dnsdist.AXFR),
        QTypeRule(dnsdist.IXFR)
    },
    NotRule(OrRule{
        -- Webservers
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
    })
}, RCodeAction(dnsdist.REFUSED))

-- Notify ACL
addAction(AndRule{
    OpcodeRule(DNSOpcode.Notify),
    NotRule(OrRule{
        -- Webservers
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
        makeRule('xx'),
    })
}, RCodeAction(dnsdist.REFUSED))

Any idea?

[Habbie]

A user tells me this works in 4.3, so I'm closing this.

[aldem]

@Habbie I hate to disappoint you, but it _does not_ work in 4.3 - same problem:

2020-06-25 18:12:49.868815 Received NOTIFY for test.zone from 127.0.0.1:60316 for which we are not authoritative, trying supermaster
2020-06-25 18:12:50.220433 Error resolving SOA or NS for test.zone at: 127.0.0.1: Query to '127.0.0.1' for SOA of 'test.zone' produced no answers

If that user could share his setup, it would be nice, but with dnsdist (1.4) in front of pdns auth 4.3 it still does not work. And I have trusted-notification-proxy=127.0.0.1 in my pdns.conf - does not help at all (though documentation does not explain what exactly it does nor how it is supposed to work at all).

[Habbie]

Hello! You replied to a closed ticket. Such replies often go unnoticed. Furthermore, given that other users report this works, I'd like to treat this as a support question. Canned reply follows:

Our GitHub issue tracker is for bug reports and feature requests. Your question looks like a support question. Support questions are handled in our other online communities: IRC and our mailing lists. Please see https://www.powerdns.com/opensource.html for information about those.

[influenist]

@hannie I can confirm it is not fully working yet. Will post details later.

[aldem]

@Habbie With all due respect, this is not a support request, this is a confirmation that initially reported bug still exists with version which you are named as "works with" thus the ticket deserves to be reopened.

From my point of view "a user tells me" is not something that justifies closing the ticket without any proof, and without inspecting that user's configuration and setup - it might _look like_ it works while in fact it is not. Perhaps this is a matter of configuration options (it is - see below) but you didn't mention it, and using workarounds like provided above does not really qualify as "it works".

Judging from log messages, it seems that pdns simply tries to fetch the SOA from source address of NOTIFY query, and since this is always dnsdist (which is often localhost) it could never work _by design_, unless... (inspecting the source...) looks like that pdns might extract the real source from EDNS subnet, and dnsdist even has option to forward client's source as ENDS subnet, but this requires few changes in dnsdist/pdns configuration:

• Adding useClientSubnet = true to addServer() options
• Adding one additional rule to adjust ECS netmask length (otherwise it will be "rounded" to /24 or whatever):
  
  
  
  • addAction(AllRule(), ECSPrefixLengthAction(32,128));
  
  
• Enabling ECS processing in pdns by adding edns-subnet-processing=yes to pdns.conf

This way, we actually could inform the backend server about original source, without any proxy protocol or anything like that (tested with pdns auth 4.3 and dnsdist 1.4).

But, there is a (still) big "but"... this does not work once zone is created as slave zone, as pdns sets master column to 127.0.0.1 (or whatever original _packet_ source was in use).

So, I would say that since the announced feature is not supposed to work "by design" (current design, that is) - it is still a bug. Sorry.

[Habbie]

Ok, I've reopened the ticket!

[rgacogne]

But, there is a (still) big "but"... this does not work once zone is created as slave zone, as pdns sets master column to 127.0.0.1 (or whatever original packet source was in use).

Sounds like https://github.com/PowerDNS/pdns/pull/9223

[Habbie]

Sounds like #9223

In that case, it should be fixed in 4.3.2 when we release it.

[influenist]

Just to confirm.
1 Any records which have been made priorly (before DNSDIST) are being updated;
2 Any record being manually added to the database get correctly pulled from master;
3 New zones without adding them are not being added by pdns. To me it looks like DNSDIST does not pass the notify as I do not see any error from PDNS (which is not listed as supermaster);

So above points 1-2 shows the config works for current zones (expect when you have a list of masters on the zone, then you run into the issue from point 4);

4) Some former zones get SOA error like this (a known issue):
Jun 26 16:11:21 ns1 pdns_server: Unable to retrieve SOA for xxxx.nl, this was the first time. NOTE: For every subsequent failed SOA check the domain will be suspended from freshness checks for 'num-errors x 60 seconds', with a maximum of 3600 seconds. Skipping SOA checks until 1593180741

I run with the following config:

setLocal('0.0.0.0:53')
setACL({'0.0.0.0/0', '::/0'}) -- Allow all IPs access

newServer({address='127.0.0.1:5300', pool='auth'})
newServer({address='127.0.0.1:5301', pool='recursor'})

recursive_ips = newNMG()
recursive_ips:addMask('127.0.0.0/8')
recursive_ips:addMask('xx/22')
recursive_ips:addMask('xx/22')
recursive_ips:addMask('xx/24')

webserver_ips = newNMG()
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')

-- AXFR ACL
addAction(AndRule{
    OrRule{
        QTypeRule(dnsdist.SOA),
        QTypeRule(dnsdist.AXFR),
        QTypeRule(dnsdist.IXFR)
    },
    NotRule(NetmaskGroupRule(webserver_ips))
}, RCodeAction(dnsdist.REFUSED))

-- Notify ACL
addAction(AndRule{
    OpcodeRule(DNSOpcode.Notify),
    NotRule(NetmaskGroupRule(webserver_ips))
}, RCodeAction(dnsdist.REFUSED))

addAction(OrRule{
    QTypeRule(dnsdist.SOA),
    QTypeRule(dnsdist.AXFR),
    QTypeRule(dnsdist.IXFR),
    OpcodeRule(DNSOpcode.Notify)
}, PoolAction('auth'))

addAction(NetmaskGroupRule(recursive_ips), PoolAction('recursor'))

[rgacogne]

This configuration does indeed not pass the original source IP from dnsdist to the auth using EDNS Client Subnet. The three steps listed in https://github.com/PowerDNS/pdns/issues/5664#issuecomment-650193287 are needed. I know our documentation does not make that clear, sorry about that, we are very open to suggestion on how to improve it!

[aldem]

Thank you guys, we are moving forward :)

@rgacogne Perhaps you could simply include the above mentioned steps somewhere in dndist docs, I believe that "Using EDNS Client Subnet" in "Advanced topics" is the right place, just mention "superslave" and "supermaster" so it could be found, and also include pdns auth option to enable (as it wouldn't work without it).

[influenist]

@aldem so in other words, slave setup configuration & zone creation based upon SUPERMASTER is not working since the introduction of dnsdist? Isn't this a major feature which should work after beta had been completed?

[influenist]

@aldem This setup is now correctly?

setLocal('0.0.0.0:53')
setACL({'0.0.0.0/0', '::/0'}) -- Allow all IPs access

newServer({address='127.0.0.1:5300', pool='auth', useClientSubnet=true})
newServer({address='127.0.0.1:5301', pool='recursor'})

recursive_ips = newNMG()
recursive_ips:addMask('127.0.0.0/8')
recursive_ips:addMask('xxxx/22')
recursive_ips:addMask('xxxx/22')
recursive_ips:addMask('xxxx/24')

webserver_ips = newNMG()
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')

-- AXFR ACL
addAction(AndRule{
    OrRule{
        QTypeRule(DNSQType.SOA),
        QTypeRule(DNSQType.AXFR),
        QTypeRule(DNSQType.IXFR),
        OpcodeRule(DNSOpcode.Notify),
        OpcodeRule(DNSOpcode.Update)
    },
    NotRule(NetmaskGroupRule(webserver_ips))
}, RCodeAction(DNSRCode.REFUSED))

addAction(OrRule{
    QTypeRule(DNSQType.SOA),
    QTypeRule(DNSQType.AXFR),
    QTypeRule(DNSQType.IXFR),
    OpcodeRule(DNSOpcode.Notify),
    OpcodeRule(DNSOpcode.Update)
}, ECSPrefixLengthAction(32,128), PoolAction('auth'))

addAction(NetmaskGroupRule(recursive_ips), PoolAction('recursor'))
addAction(AllRule(), ECSPrefixLengthAction(32,128), PoolAction('auth'))

[aldem]

@influenist: yes, technically you are right - though no one promised that it will work behind dnsdist "out of the box" :)

As to configuration - yes, seems correct (though you need only last action with ECSPrefixLengthAction() as addAction() accepts only one action), but you have to wait for 4.3.2 before you actually could use it because of #9223. And don't forget to add edns-subnet-processing=yes to your pdns.conf.

And the side note: unless you actively monitor slave zone creation/propagation, better don't rely on "supermaster" feature, if it fails or delayed (due to lost/interrupted/instable connectivity) you are in trouble.

[sshipway]

I spent a long time trying to get dnsdist->recursor+auth working on our test system; in the end I was unable to make it work with the supermaster feature and notifications. It would have been really useful to be able to have PowerDNS working as a single unit (as with v3.x). In the end I added a second IP to the DNS hosts and had recursor on one and auth on the other, removing the need for dnsdist.
Really, setting up this sort of configuration should be much simpler - for example, recursor being able to forward notifies on to auth, and to auto self-configure for the zones hosted by auth, etc. There can't be a small number of users who want to have a single DNS host for auth+recurse.

[influenist]

Notifications of in database zones run through the config with ease, yet notifications of none existing domains are being ignored. I do not even see an error like:

Received NOTIFY for test.xx from xx.xx.xx.xx for which we are not authoritative, trying supermaster

@aldem So you mean like this:

setLocal('0.0.0.0:53')
setACL({'0.0.0.0/0', '::/0'}) -- Allow all IPs access

newServer({address='127.0.0.1:5300', pool='auth', useClientSubnet=true})
newServer({address='127.0.0.1:5301', pool='recursor'})

recursive_ips = newNMG()
recursive_ips:addMask('127.0.0.0/8')
recursive_ips:addMask('xxxx/22')
recursive_ips:addMask('xxxx/22')
recursive_ips:addMask('xxxx/24')

webserver_ips = newNMG()
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')
webserver_ips:addMask('xx/32')

-- AXFR ACL
addAction(AndRule{
    OrRule{
        QTypeRule(DNSQType.SOA),
        QTypeRule(DNSQType.AXFR),
        QTypeRule(DNSQType.IXFR),
        OpcodeRule(DNSOpcode.Notify),
        OpcodeRule(DNSOpcode.Update)
    },
    NotRule(NetmaskGroupRule(webserver_ips))
}, RCodeAction(DNSRCode.REFUSED))

addAction(OrRule{
    QTypeRule(DNSQType.SOA),
    QTypeRule(DNSQType.AXFR),
    QTypeRule(DNSQType.IXFR),
    OpcodeRule(DNSOpcode.Notify),
    OpcodeRule(DNSOpcode.Update)
}, PoolAction('auth'))

addAction(NetmaskGroupRule(recursive_ips), PoolAction('recursor'))
addAction(AllRule(), PoolAction('auth'))
addAction(AllRule(), ECSPrefixLengthAction(32,128))

Or

AllRule(ECSPrefixLengthAction(32,128))

[Habbie]

Fixed in #9223