Contour: AAAA querys in IPv4 mode between envoy and contour

[movikbence]

I am using contour as an IPV4 loadbalancer service and I noticed that in every 5 seconds a dns query is sent by envoy.
That is clear that the 5 sec query is happening due to the Strict DNS option in envoy bootstrap config.
"When using strict DNS service discovery, Envoy will continuously and asynchronously resolve the specified DNS targets."

I don't understand why are the querys are being sent on IPv6.

On slack there is a conversation started yesterday, here is the link for it:
https://kubernetes.slack.com/archives/C8XRH2R4J/p1612456585116300

Environment:

• Contour version: 1.11
• Kubernetes version: (use kubectl version): v1.15.2
• Kubernetes installer & version: v1.16.2
• Cloud provider or hardware configuration: bare metal
• OS (e.g. from /etc/os-release): suse

[sunjayBhatia]

As mentioned in Slack, its likely we could prevent this by ensuring the Envoy -> Contour cluster service (and probably other extension gRPC services) lookups happen over IPv4 only if desired via a config parameter etc. Currently the address family to lookup Contour and other extension services is not set so it should be defaulting to "auto" which looks up IPv6 first then IPv4.

[movikbence]

Hi

Has the investigation started on this issue?
Thanks in advance!

[youngnick]

Hi @movikbence, I'm sorry but it hasn't yet. Thanks for the ping, I will make sure we get someone onto it.

[movikbence]

Thank you for the update @youngnick !

[youngnick]

So, the AAAA query is being sent because that's Envoy's default. There is a parameter for each cluster - dns_lookup_family, that may be set to AUTO, V4_ONLY, or V6_only, with AUTO being the default.

In the AUTO case, Envoy will look for IPv6 addresses first with AAAA queries, then try IPv4 with A queries.

In order to fix this, we would need to add a way to specify the DNS family for the contour cluster in the bootstrap config output by contour bootstrap. Which doesn't seem like too big a change, to be honest.

[abhide]

@youngnick makes sense. Looked through contour bootstrap code.
Posted a PR: https://github.com/projectcontour/contour/pull/3376 .

[sunjayBhatia]

See https://github.com/projectcontour/contour/pull/3376#pullrequestreview-594262150 for some comments on what we should test to ensure this is fully finished

[youngnick]

Closed by mistake, keeping open to ensure we clear everything as per @sunjayBhatia.

[movikbence]

Hi

Is the fix included in contour version 1.13.0?

Thanks!

[sunjayBhatia]

@movikbence yes this commit https://github.com/projectcontour/contour/commit/64db9081049afb968b24be94541cadf074fd75ad is included in Contour 1.13.0+

[movikbence]

Hello

I tried if it works with the given option in the configmap.

I looked into the commit itself and I saw that if this is set it should be appear in envoy configdump as:

    "dns_lookup_family": "V6_ONLY",
    "upstream_connection_options": {
      "tcp_keepalive": {
        "keepalive_probes": 3,
        "keepalive_time": 30,
        "keepalive_interval": 5
      }

I deployed contour with v4 option and I was not able to find anything related to that in the configdump.
Then I tried with the 2 different option, still no luck.
After that I made a tcpdump, and I was still able to see the AAAA dns requests.

Can you please try to reproduce it, or just check if it can be seen in the configdump?
I attached the configdump.

Thanks in advance!

Br,
Bence
config.zip

[movikbence]

HI

Anyone had time to check it?
thank you

[youngnick]

Hi Bence, sorry to take a while to get back to you.

I had a look, and, when I edit the contour bootstrap cmdline in its initContainer in the Envoy DaemonSet to add

- --dns-lookup-family=v4

I see the dns_lookup_family="V4_ONLY" in the config.

I did make the mistake at first of setting the value to V4_ONLY, which is silently thrown away. That doesn't seem ideal, seems like we could do better there.

Could you check that:

• you are changing the value in the command line of the contour bootstrap command
• you are setting the value to v4, not V4_ONLY?

[movikbence]

Hi Nick,
No worries. If I use it in contour bootstrap cmdline it works for me as well, I thought it can be achieved if I use it in the configmap, because there is also an option with the same name (dns-lookup-family)

# Envoy cluster settings.
# cluster:
#   configure the cluster dns lookup family
#   valid options are: auto (default), v4, v6
#   dns-lookup-family: auto

That would make sense if it would work with it as well.

[youngnick]

The entry in the configmap causes Contour to use that DNS lookup family for all the dynamic clusters it creates (that is, the ones pointing to backends). However, the contour cluster that points Envoy back to the Contour control plane is static, and is created by the bootstrap, not the main Contour (as it's required to be able to connect to Contour so that the Envoy can be configured!).

So, to totally eliminate all AAAA queries, you need to use both the configmap item, and the bootstrap command line option.

[movikbence]

I see, thanks. Can you suggest an option for dns-lookup-policy inside the cluster if I don't know which type of IP will be used?
My helmchart is now IP agnostic, does not matter where I deploy it. I tried auto option, but it was auto by default, so no success with it.

[youngnick]

Sorry, the only way to stop Envoy sending AAAA queries is to set everything to V4 only, it seems. The AAAA-first behavior is pretty standard, so if you're not sure what IP space will be used, you're a bit stuck.

[movikbence]

Thank you for the update