Aws-app-mesh-roadmap: Bug: Cloud Map DNS negative caching TTL cannot be changed

[bcelenza]

Note: This bug affects customers who use Cloud Map with App Mesh. Although it is not an App Mesh-specific issue, the bug has been reported in this repository for visibility and resolution tracking.

Summary
When using a Cloud Map namespace that's associated with a public or private Route 53 hosted zone, the negative caching TTL is not adjustable. For public hosted zones, the TTL is 900 seconds. For private hosted zones with the use of the VPC DNS resolver (IP address ending in .2), the TTL is 300 seconds. See the Route 53 documentation for how this TTL is determined.

The negative caching TTL is used whenever a requested DNS name contains no records. When this occurs, the DNS server responds with NXDOMAIN, and that value is cached for the time indicated above. In the event that all records are removed for a given DNS name, clients querying DNS will continue to see NXDOMAIN responses until the cache has expired.

Steps to Reproduce

1. Create a Cloud Map namespace (docs) associated with a Route 53 private hosted zone.
2. Create a service entry in the Cloud Map namespace.
3. Register an instance to the service entry and give it an IP address. Set the TTL for this record to 10 seconds.
4. Using the dig utility on an EC2 instance within the VPC associated with the private hosted zone, query DNS for the record and assert that you receive the IP address you entered in step 3. (example command: dig my-service.namespace.cluster.local)
5. Remove the instance added in step 3 from Cloud Map
6. Repeat the query from step 4 until the record TTL (10 seconds) expires. Assert that you begin receiving an NXDOMAIN response.
7. Re-add the instance to Cloud Map.
8. Repeat the query from step 4, and assert that you continue experiencing the NXDOMAIN response until the negative caching TTL (300 seconds for a private hosted zone using the VPC DNS resolver) expires.

Are you currently working around this issue?
To mitigate the impact of the negative caching TTL resulting in no records being returned (and a potential outage):

1. Use a Route 53 private hosted zone and the VPC DNS resolver (IP address ending in .2). This ensures your negative caching TTL is 300 seconds (instead of 900 in a public hosted zone).
2. Ensure your services scale out before scaling in during deployments. This mitigates the possibility of all records being removed from the DNS record. See our best practices documentation for details on this.
3. If you're only using Cloud Map with App Mesh (and specifying Cloud Map as the service discovery type on your Virtual Nodes), consider increasing the instance record TTL. This will help mitigate a temporal issue in which all records are removed from the DNS name. Since Envoy proxies managed by App Mesh do not use DNS to resolve IP address for Virtual Nodes that use Cloud Map, the increased TTL will help ensure your applications have a resolvable DNS name without impacting which endpoints traffic is routed to.

[dastbe]

This is additionally a vector for issues seen in #151

[herrhound]

Today AWS Cloud Map released new default values for negative DNS caching. Now it is 15 seconds for namespaces with private DNS resolution (instead of 300 seconds) and 60 seconds for namespaces with public DNS resolution (instead of 900 seconds). These new defaults apply to new namespaces, created on or after 09/14/2020 only. We continue working on enabling modification of negative DNS caching settings for all existing namespaces as well.

[nbryant42]

I'm seeing an SOA TTL of 86400s (1 day) on an older zone, created via the ECS console, probably in 2018/19.

[nbryant42]

In fact, I'm still seeing a TTL of 86400s (1 day) on a brand-new zone created through the ECS console today.

[herrhound]

@nbryant42 -- could you please share your SOA record?

[nbryant42]

@herrhound sure. As obtained from dig qa-01.semsee SOA, the record is

qa-01.semsee.       15  IN  SOA ns-1536.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

[herrhound]

@nbryant42 - as per the RFC2308, when the authoritative server creates this record its TTL is taken from the minimum of the SOA.MINIMUM field and SOA's TTL. In your case the TTL of the SOA record itself is set to 15 seconds, so your client applications won't cache the negative DNS results for longer than that.

[nbryant42]

@herrhound Thanks. That makes sense, and seems consistent with the application behavior I've seen.

I still have two older zones that have a 900s TTL on the SOA record. It would be inconvenient to delete them and replace them, but guess I'll do that if I have to.

[herrhound]

@nbryant42 -- We are working on the API and console experience to update the SOA TTL for the existing hosted zones. I can't comment on the timeline, unfortunately. If it's critical for your workloads, recreating the Cloud Map namespaces will solve the issue today. Otherwise, you may save yourself the hassle.

[herrhound]

AWS Cloud Map now supports changing the SOA TTL for new and existing hosted zones: https://aws.amazon.com/about-aws/whats-new/2021/07/aws-cloud-map-supports-configuring-negative-caching-for-dns-queries/

Closing the ticket.