Envoy: UDP based DNS server/filter - Feature Request

[sumukhs]

Similar to previously closed issue.

This feature request is to run Envoy as a DNS server. A new filter section for DNS can be added, which contains mapping from the dns name to the cluster name. The mapping can be configured statically via the config file or dynamically using xDS API.

Motivation:
We currently have a separate DNS server handling dns requests for our services and with the move to Envoy as our primary routing layer for service-service communication, we would like to move the dns handling as well to a centralized component that is already aware of how to route requests.

Since the filter framework works on a given listener, a UDP filter stack is required for this to work. I see some WIP for a UDP proxy. Would its completion enable this scenario more seamlessly?

@mattklein123

[mattklein123]

Since the filter framework works on a given listener, a UDP filter stack is required for this to work. I see some WIP for a UDP proxy. Would its completion enable this scenario more seamlessly?

Yes, we would need to complete UDP basic filter support for this to be implemented as an extension. cc @envoyproxy/quic-dev

[sumukhs]

Thanks @mattklein123 .

Is there an existing PR or work stream where we can contribute?
@appario and myself would be available to work on this effort.

[cmluciano]

@sumukhs I am currently working on plumbing UDP into upstream paths. I can cc on this PR once pushed (fixing some tests this week).

[sumukhs]

Thanks @cmluciano. Could you share the private repo/PR link? Is there a target date you are aiming for?

[sumukhs]

cc-ing others who have been involved in this area - @conqerAtapple @alyssawilk

I am trying to get up to speed on the history of how the UDP stack has been shaping up... I found design docs (including QUIC) and active work on the QUIC area these days.

I also see some closed PR's by @conqerAtapple in favor of smaller work items and I have seen the UDP listener is checked in now.

Is @cmluciano the only person actively working on the UDP stack outside of QUIC? I would like to get involved (with another co-worker at Microsoft) in the UDP effort to build the required pieces to shape up the filter stack so that we are able to write a DNS filter.

Can someone comment on what the road map is and give us more context. Happy to join the community call tomorrow morning (5/6) to talk more.

[mattklein123]

@sumukhs I would probably assume at this point that no one is working on basic UDP filters. I would love to see this happen. Maybe it would be easiest to setup a small meeting and we can discuss? Please email me at [email protected] to schedule. Thank you.

[sumukhs]

@mattklein123 - this is an initial draft of the proposed changes for supporting DNS type of filters.

https://docs.google.com/document/d/1uMGhZ4oizCE3nbmGtes20yb-m9cvWvMLfzanZXc81xM/edit?usp=sharing

[mattklein123]

@sumukhs looks like on the right track. I dropped some comments in. Very exciting stuff. cc @envoyproxy/quic-dev @envoyproxy/maintainers if anyone else wants to take a look at the doc. Thank you!

[sumukhs]

I have started working on a DNS filter based on the UDP listener added in #6912 . @mattklein123 - Do you think it might be useful to have this extension in envoy based on the interest here?

Here is a V1 proto I have for now - dnsproto. I will be happy to share the design doc if needed.

I read the requirements about needing a sponsor to author a filter - may be we can talk about it next week in the community discussion?

[mattklein123]

cc @lizan @PiotrSikora who had mentioned that Istio might want a DNS filter. ^ Sure let's discuss on the community call.

[bcelenza]

App Mesh also has a pretty big interest in this. I'd be happy to join and participate as needed.

[mattklein123]

If there are multiple users it would be great to get this upstreamed. @bcelenza I will connect you and @sumukhs over email.

[bcelenza]

@sumukhs and I spoke this morning, and what is in place may fit our particular use case. That said, I have an idea that may make the solution more flexible so that different record types could be responded to by Envoy.

Next step is for me to fork Sumukh's API changes and propose what I'm thinking of. If we agree those changes are generally useful to the feature, we can get them implemented and start looking at upstreaming the work with @mattklein123's help.

[bcelenza]

Hey @sumukhs, sorry for the delay on getting back to this.

I've spent some time looking through the DNS filter you've implemented and I think it would work for the majority of use cases I'm thinking of.

The one thing I'm interested in doing with this is adjusting the API definition slightly so that future options could be added, and the TTL specified per entry. For example, instead of:

message ServerSettings {
    repeated string known_domainname_suffixes = 1
    [(validate.rules).repeated = {min_items:1}];
    google.protobuf.Duration ttl = 2;
    map<string, string> dns_entries = 3;
}

It would be:

message ServerSettings {
    repeated string known_domainname_suffixes = 1
    [(validate.rules).repeated = {min_items:1}];
    map<string, DnsEntry> dns_entries = 2;
}

message DnsEntry {
    google.protobuf.Duration ttl = 1;
    string cluster_name = 2;
    // Potential future options:
    // repeated string A = 3;
    // repeated string AAAA = 4;
    // ... 
}

I could see a future in which Envoy is used for DNS queries not directly related to the clusters it is managing. Does this seem like a reasonable adjustment?

[sumukhs]

Hi @bcelenza , looks okay, but I am trying to understand why it is a oneof ? Maybe I am still missing the scenario.

[bcelenza]

@sumukhs Hmm, you're right, oneof doesn't actually make sense here. Adjusted my comment above.

[bcelenza]

Sorry, just realized I missed your comment on the scenario. I'm hoping to merely preserve the possibility to use Envoy as a general DNS responder for any record type, not necessarily tied to a known destination. I think only implementing cluster_name is the right thing to do now.

[sumukhs]

@bcelenza - Thanks for clarifying. I see how the TTL could be varying for every request. How does the cluster_name get used? Is it just information that the query would return as part of the DNS response? I can see how envoy can respond to queries about DNS entries that do not belong to the current cluster, but I am trying to understand if that information is part of the known_domainname_suffixes or is a separate cluster_name entity is needed. And if this field is present, how is being consumed or populated?

[bcelenza]

In this case cluster_name works the same as the original string value in the map for your implementation (here), it's just modeled with an explicit property name. So the DNS query response would still return all the records for the lb_endpoints. Alternatively a set of static A/AAAA/etc. records could be supplied via the API, but as mentioned earlier, that doesn't seem like an MVP for this right now.

On known_domainname_suffixes, that's a good question. Can you help me understand the usage of the field a little better? I see that we first check for the known domains, falling back to external resolution if not present, and for known names we'll return NXDOMAIN when an exact record doesn't match. Is there a reason we don't simply check the dns_entries map keys for the requested domain, and if not present, fall back to the external resolver?

[sumukhs]

Ah got it. Sorry, I had been on a pretty long vacation for 3 months and its taking time to context switch things back into place.

I confused the envoy naming convention of "cluster" to what we refer to as cluster in our team.

Good question regarding the known_domainname_suffixes. I added some comments in the code to clarify this, but clearly I have not done a good job :-)
Basically, in the response, I want to be able to differentiate between the 2 scenarios where a query for google.com arrives at the DNS server VS a query for "clusterA.microsoft.cluster" while the service is down arrives. In both cases, the DNS entry will be absent. However, for google.com, we want to forward it to an external server. For the clusterA, there is no point in forwarding the request to an external resolver as it is not expected to know.

In other words, the known_domain_names is the list of domains for which this DNS server is the authoritative name server.

[bcelenza]

Got it! That makes sense to me, thanks. :)

[bcelenza]

Hey @sumukhs, are you still working on this feature?

@abaptiste from the AWS App Mesh team would love to help out

[sumukhs]

Hi @bcelenza , I have been side tracked with some other work. It would be great to have @abaptiste continue the progress from what we discussed last time.

[bcelenza]

Great!

@mattklein123 Can we re-assign this to @abaptiste?

[abaptiste]

@mattklein123 Here's a document with a proposed design:

https://docs.google.com/document/d/1pTmd-ZMD-4VEdZOWeG4pgBBWUsp0D5z4wipK-FiXdiU/edit?usp=sharing

Please review and let me know if you have any problems accessing it.

[mattklein123]

@abaptiste do you mind providing comment access? Thank you!

[sumukhs]

@abaptiste - Have you taken a look at this implementation? https://github.com/sumukhs/envoy-dnsfilter

I do see some parallels with the design in your document, but wanted to touch base and see if we can come up with something that meets both our requirements. I had discussed this with @bcelenza and we had agreed on the direction based on the conversation in this thread.

One missing piece I noticed from the document review is the following config for "known domain name suffix" we had discussed earlier:

Good question regarding the known_domainname_suffixes. I added some comments in the code to clarify this, but clearly I have not done a good job :-)
Basically, in the response, I want to be able to differentiate between the 2 scenarios where a query for google.com arrives at the DNS server VS a query for "clusterA.microsoft.cluster" while the service is down arrives. In both cases, the DNS entry will be absent. However, for google.com, we want to forward it to an external server. For the clusterA, there is no point in forwarding the request to an external resolver as it is not expected to know.

In other words, the known_domain_names is the list of domains for which this DNS server is the authoritative name server.

[abaptiste]

@mattklein123 I updated the permissions on the doc.

@sumukhs I did look at your implementation.

Correct me if I am not understanding. It seems that we want Envoy to forward queries to an external server for names that are not virtual hosts or clusters.

Envoy should be able to determine whether it is the authoritative server for the query and divert the query externally if necessary.

[sumukhs]

@mattklein123 I updated the permissions on the doc.

@sumukhs I did look at your implementation.

Correct me if I am not understanding. It seems that we want Envoy to forward queries to an external server for names that are not virtual hosts or clusters.

Envoy should be able to determine whether it is the authoritative server for the query and divert the query externally if necessary.

Yes that's right. Does this implemention work for you guys? I recall @bcelenza had 1 requirement which wasn't addressed in the implementation. Could we work together on adding that to the current implementation?

[abaptiste]

Until @bcelenza comments, I have a couple questions.

• Why wasn't this implementation upstreamed and made in-tree?
• What's the benefit of defining a new listener instead of writing a filter for an existing UDP listener?

For the second question I get that UDP listeners didn't exist when the DNS Filter was started, but is there a clear advantage of going one way or the other?

[sumukhs]

The dns filter was lowered in our priority among other work items. The initial work was done to validate the proof of concept and it is at a stage where it is ready to get upstreamed if we agree on the goals.

For the second part, there is no new listener. I added the udp listener in envoy before starting to work on the dns filter. As you correctly point out, the dns filter is indeed a new filter for an existing listener that is if type UDP.

[bcelenza]

I see three requirements here that I think we need to address:

1. Whitelisted domains for which the proxy is allowed to respond to at all.
2. Fallback configuration for when the proxy receives a query for a whitelisted domain, but doesn't have a record.
3. Configuration for when the proxy receives a query for a non-whitelisted domain.

@sumukhs Does (2) sound like the right expression of intent you had with the original known_domainname_suffixes implementation?

I think (1) is a security requirement. Even though I may be the entity running the proxy, I may not be the one configuring it. I need a way to scope down what I'm allowing the proxy to attempt to respond to vs. what needs to be passed directly to a separate authoritative DNS server.

[sumukhs]

(1) addresses the intent. (2) is something I have not captured because my expectation was that this just means there is a transient fail over or some other situation causing the config to not contain and record - which means we must reply back with NX_DOMAIN and the client can retry later.

Agree with the security aspect of (1). Is there a reason why the dns filter repo I created is not forked and added with additional functionality? Or even better, just take the code and port it to the upstream rather than re-writing everything from scratch. There are sufficient unit tests and documentation in the form of comments in the code. I am also happy to go over it if needed to bring someone up to speed.

[bcelenza]

I believe we wanted to start from the latest master base in the main repo and build up from the official contributions of UDP, etc.

Is there a specific concern over using your implementation? For background: we've actually had multiple POCs around this that never ended up getting posted publicly. We're not worried about the cost of unit test coverage, etc., just to get assurance we're building from the right base.

[sumukhs]

No specific concern. Since the 2 of us had spent some cycles along that path with the config reviews and understanding what each of us wanted for the dns filter, taking the existing code to the requirements seemed like the path with least work. Also, given the implementation already took care of most of the points above, it is just a matter of adding 1 additional feature from what I recall you wanted to have:- the TTL and the ability to pre define configurations.

I am happy to work with an implementation starting at the master base also. Just wondering if it is the best use of everyone's time to start from 0 than picking it from a well established checkpoint we had agreed on earlier.

[bcelenza]

Thanks, that makes sense. We have a similarly working code base that's based off the most recent master branch, so we'll be starting there.

I'd like to make sure we codify the requirement appropriately, so @abaptiste let's get the whitelist revision reflected in the current design document.

[abaptiste]

@sumukhs, @bcelenza,@mattklein123 gents, can we resolve this issue since the DNS Filter is now merged into Envoy?

[mattklein123]

Thanks will close and we can open discrete issues for any new features, thanks!

[sumukhs]

Thanks for completing this work item @abaptiste - Greatly Appreciated!