Pdns: Feature request: DNS Prefetching

[Habbie]

This is a feature request for DNS prefetching in PowerDNS Recursor. Attached is a proof of concept implementation. It seems to work. But it's hard to say whether it improves the average latency from the customer perspective. Prefetching simply ensures that popular domain names are always fresh in the Recursors cache.

The attached patch already includes the patch suggestion from [http://wiki.powerdns.com/trac/ticket/438] which is a requirement for the prefetch proof of concept patch.

[Habbie]

Attachment '' (prefetch.patch) https://gist.github.com/5466734

[Habbie]

Author: anon
Patch updated:
CNAME chains are now also handled by prefeching.

[Habbie]

Author: anon
Patch updated: Only pre-fetch if no "outqueries" was sent.

[Habbie]

Author: anon
prefetch-ratio isn't constrained between 0 and 90 with this patch; negative values are also allowed, with unknown but probably not good consequences.

[Habbie]

Author: anon
g_prefetchRatio is an unsigned integer, thus negative values are not possible I thought. And this line in the path should limit the value:

g_prefetchRatio=min(::arg().asNum("prefetch-ratio"), 90);

[r4um]

Is this still in roadmap ?

[paddg]

I have hard tried to prove that this feature improve average hitrate and latency significant on a very busy Recursor. But I failed.

[jemshad]

I would expect a considerable increase in hitrate when the hostnames to lookup are kind of static and even more useful when the application is very much latency sensitive - even milliseconds count.

[johnhtodd]

I'm a bit unclear on how this would work - is a pre-fetch performed only on RRs that have seen =>1 request during a TTL expiration cycle? There needs to be some way to allow records to expire, so I would expect that to be the case (I don't see any explanation in the config defaults, either.) Sorry if I could determine this by looking at the code in detail, but I don't see the location where that's determined. Also: is this still on the roadmap?

[jemshad]

I am not sure what you mean by TTL expiration cycle here. In the implementations available already, pre-fetch kicks in if a lookup happens for a record in cache in the last x% of the TTL expiry (x being configurable with 10 as default). Ex: For a record with 300s TTL, if a request comes again in the last 10 seconds of its expiry, then the record is pre-fetched in the background after serving the client with the current TTL.

[spirillen]

@paddg I've "tested" this on my two home linux boxes (bad adsl line) dnsdist+recurser vs unbound with prefetching enabled unbound keep all quries below 5ms where dnsdist+recurser use up to 1400ms on responses that have expired, now multiply this to a huge installation.... I'll see the benefits in @jemshad suggestion (one post up)

[paddg]

@spirillen, on the other hand, it is very unlikely that a popular domain name will be a cache miss if you share the cache with 100k other subscribers (huge). And prefetching only works with popular names anyway.

[spirillen]

@paddg that's somewhat true, But with the new TTL "standard" of 30|60 sec there most be somebody who is waiting for the re-heating the cache.... In the light of that new TTL "standard" I'll see the beneficial in have then script running in the background at the last 5 sec of a TTL to replace the the cache to ensure a high hit rate..

(NOTE:) This is a thought not tested or reading up on this.... But maybe one of the kind coders got add some wisdom

[paddg]

I agree, "active" prefetching would work a little bit better with that prefetching feature. But the term "new TTL standard" confuses me.

[spirillen]

@paddg the little play here is the CDN's who just like to spread a TTL on 36/60 secs and the number (to me) only seems to grow adding a lot of DNS traffic which in my mind could(should) be handled by round robin or anycast-IP. That why i called it "New standard" :smile:

[omoerbeek]

I studied the POC code and have some comments.

• The TTL is lowered to zero while prefetching. I think that means that new incoming requests asking for the same recordset will be stalled during prefetch.
• There are also RC entries that are used for resolving (like NS records) but that are not included in the answer. For these record the prefetching is not triggered at at all AFIKS.
• mthreads are kept busy for prefetching that might serve user requests otherwise.

I have some thoughts on how to potentially solve these issues. I might come up with a new POC.

[paddg]

Thanks for looking into it, @omoerbeek! I still think it could not hurt to have a prefetching feature. But please take the POC code not too serious. I am not a programmer and even I don't fully understand the code anymore.

[omoerbeek]

Whatever happens the POC code must be thrown away, the recursor changed too much.

[omoerbeek]

I built a quick POC of one of the building blocks to do prefetching. I did not do prefetching itself.
Please note this is pure experimental code and it has not been decided yet if we actually want this.

See https://github.com/omoerbeek/pdns/tree/rec-refresh-almost-expired

From the commit message:

This adds a code to the record cache to considers "almost
expired" records as expired. This has the consequence that
they will be refetched by SyncRes and thus refreshed.
Running a query with this enabled keeps your cache warm for
the records used to resolve the target.

This is intended as an experimental feature to build two other
experimental things:

• manual cache warming: by firing queries using this mode once
  in a while.

• auto refresh: in normal mode, if an almost expired (PC or RC?) record is encountered,
  queue the target for refetching in a background task queue. This
  queue should be processed with somewhat lower prio than ordinary
  queries. The tasks should also have a deadline: if not executed
  before the deadline, just drop it.

It has to be decided how to set this mode for external queries (I
do it now for queries arriving on a specific port with a silly 50%
setting).

If you want to play with this disable your packet cache.

[paddg]

Thanks @omoerbeek, I will test it, but how can I set refreshTTLPerc? I didn't find a setting for it.

[omoerbeek]

There is no setting. Currently I'm using a hack in pdns_recursor.cc to set it (to 50% even, so the refresh will kick in if half of the original TTL went by) if the port the query is coming in for is 5302. So for testing I send regular queries to an ordinary port (5301 in my test setup), while the "keep warm" queries go to port 5302.

This is part of the recursor.conf I'm using:
```
local-address=0.0.0.0:5301 0.0.0.0:5302 [::]:5301
````

[paddg]

@omoerbeek, I compared your prefetch patch to 4.4.0 with 4 tries:

1. 4.4.0
2. prefetch
3. 4.4.0
4. prefetch

I turned of the packet-cache (and the dnsdist cache).

It works! After 50% TTL, records get updated. I tried it with a real traffic recording.

My result:
No significant changes on Recursor metrics. Although I expected the latency to drop. Or CPU rise. Nothing. But maybe prefetching has advantages from subscriber perspective, I don't know.

[omoerbeek]

I think what you tested is just does refetching before the TTL expires all the time. That means that all the queries that hit the early expiry see extra latency for doing auth queries. I think we want to go to a solution where the refetch should be done in a background task, so all regular queries see a hot cache, even while the refetch is being done.

I have tested the by doing queries in normal mode (on port 5301). While I have a dig ; sleep loop queries the special port every 5 seconds with a set of names. That loop causes the prefetches to happen so the regular queries always see a warm cache. This is "a poor man's" approach to the a manual setup described in the first bullet above.

For the second bullet some more code has to be written. Especially the background task part takes a bit of work.

[paddg]

Yes, I did not expect the latency metric to include the prefetch times. Latency drop is the main argument for prefetching imo. We need to be able to measure this in order to judge the feature.

[omoerbeek]

We should be careful to distinguish latency seen by the client and rec-auth latency. There are two metrics: qa-latency and x-our-latency. The latter is documented to be unreliable and the first measures latency for non-PC cache hits, this latency mixes RC hits and non-RC hits (inducing rec-auth latency) cases. Rec-auth latency is not expected to change, but I expect the RC cache hit ratio to improve with prefetching.

All in all it would be best to measure latency in the client. In the end that's what matters.

[paddg]

All in all it would be best to measure latency in the client. In the end that's what matters.

Yes, we could measure latency on client. But that would make life complicated as this does only work if you do the testing yourself and not for real traffic. The qa-latency metric is the most important quality indication of a DNS resolver. We should be able to rely on it.

and the first measures latency for non-PC cache hits

I did not know that. I thought PC hits are at least counted as 0us, isn't it?

However, we should be fine if we turn off PC when testing this feature.

[omoerbeek]

Just pushed a missing piece: insertion into a task queue when an almost-expired entry is encountered and processing of the task queue in refresh mode. This is done atm from the handler thread (and this is likely something that should be changed)

To enable, set refresh-on-ttl-perc to e.g. 10.

Keep the packet cache disabled, since "almost expired" is not handled there yet.