Ddev: DNS latency sometimes causes asking for actually unneeded change of hosts file

[mtelgkamp]

Describe the bug
Sometimes the dns seems to be too slow for the online detection, thus ddev asks to add an entry to the hostfile.
For me normally the dns resolution takes about 50ms, but sometimes it takes like 1500ms.

To Reproduce
Not reproducable every time, just sometimes it happens that DNS responds slow and then ddev asks to change hostfile. Aborting and restarting ddev works.

Expected behavior
It would be nice to have an option here

1. Add an entry to hostfile, I currently have no internet or my DNS does not resolve *.ddev.site to 127.0.0.1
2. Just continue, my DNS is slow sometimes but I know it will work

Version and configuration information (please complete the following information):

• Host computer OS and Version
  Ubuntu 20.04
• Docker version information
  Version: 19.03.6
  API version: 1.40 (minimum version 1.12)
  Go version: go1.12.16
  Git commit: 369ce74a3c
  Built: Thu Feb 13 01:26:23 2020
  OS/Arch: linux/amd64
  Experimental: false
• ddev version information
  DDEV-Local version v1.14.2
• config.yaml contents for the misbehaving project
  all projects affected
• Do you have any custom configuration (nginx, php, mysql) in the .ddev folder? If so, have you tried without them?
  all projects affected

Additional context
Slack communication on this topic: https://typo3.slack.com/archives/C8TRNQ601/p1591689467425000

[rfay]

Nice latency check there, dig $(openssl rand -base64 8).ddev.site | grep Query, and he sometimes comes up with >1s for that lookup, for unknown reasons. /cc @atomicptr

@mtelgkamp when you see this first time in the day, could you please and try again and see if that alone fixes it? Or maybe that's what you've been doing?

[atomicptr]

Hmm, seeing this I'd argue we should stop using DNS to test if we have internet.

Just increasing the timeout might also be a solution, tho not sure If I'd like that very much

[rfay]

How would you test for internet in a more effective way @atomicptr ? DNS is the the best technique I know of.

[atomicptr]

I'd argue doing a request to something is the best thing to do, but thinking about it this would only move the problem around because if the connection is degraded this will also take time.

If we increase the timeout to say 5s this also means that in a situation where you have no internet there will be a 5s delay to actually doing something, so I'm not too fond of this approach either...

[rfay]

Unfortunately, doing a request to almost anything ends up with firewall issues and network routing issues, and far more latency. If you do a request to a specific IP address then you skip the DNS lookup, but of course that's hugely fragile. If you do any other kind of request, then it's still dependent on the DNS lookup.

[rfay]

Note that it's the DNS lookup that we're actually interested in here. We want ddev to create a hosts entry if *.ddev.site cannot be resolved. That's basically it.

[atomicptr]

Unfortunately, doing a request to almost anything ends up with firewall issues and network routing issues, and far more latency. If you do a request to a specific IP address then you skip the DNS lookup, but of course that's hugely fragile. If you do any other kind of request, then it's still dependent on the DNS lookup.

Hm actually a very good point. I didn't think this through :).

Note that it's the DNS lookup that we're actually interested in here. We want ddev to create a hosts entry if *.ddev.site cannot be resolved. That's basically it.

Oh, well since his DNS is degraded sometimes it should create a hosts entry either way then, so the true issue is probably the fact that we're trying to do DNS resolves even though the requested site is already in the hosts file (because we ask for a random *.ddev.site entry). So we probably shouldn't do the DNS check in that case at all :thinking:

[mtelgkamp]

Oh, well since his DNS is degraded sometimes it should create a hosts entry either way then, so the true issue is probably the fact that we're trying to do DNS resolves even though the requested site is already in the hosts file (because we ask for a random *.ddev.site entry). So we probably shouldn't do the DNS check in that case at all.

Actually there is no entry in my hosts file and I do not want to have it. My dns works fine 99% of the time, but sometimes it seems to lag. Today seems to be a bad day for my internet because i also saw an Internet connection not detected message after I exited an ddev mysql connection... Is the internet connection checked for ddev mysql, too?
I prefer to have a clean /etc/hosts file and would like to have an option to skip the internet detection step somehow instead of having to abort the sudo password request and then start ddev again.

[atomicptr]

Just collecting some ideas, but I might agree that the current approach isn't optimal

[rfay]

@mtelgkamp one short-term approach to your situation is to wrap ddev in a script that just does your dig openssl rand -base64 8.ddev.site before starting, it would probably solve that. But I'm not suggesting that as a fix. You could also probably put that dig in your .profile as another solution.

Unfortunately, it's important to detect the ability to resolve, because otherwise the problems hit lots of people. They lose internet and then they launch their browser and get an incomprehensible error which we have no control of at all.

So I think one technique would be to make the lookup timeout configurable, it's pretty easy to do, and it doesn't have to be obvious to anybody that will never see it.

[mtelgkamp]

The dig command needs to have the backticks inside. If someone else wants to use it here is the version with backticks displayed:

dig `openssl rand -base64 8`.ddev.site

@rfay what do you think of my Expected behavior suggestion in the issue description where I wanted to give two options to continue if dns is unavailable? A commandline parameter (--skip-dns-test) or a configuration would be fine for me, too.

[rfay]

Unfortunately, if DNS is actually not available, there's no way you can access a site in the browser without /etc/hosts. So this could end up way too confusing for too many people. Your situation is an outlier I'm pretty sure @mtelgkamp and I'd love to know why it's doing that. It must be annoying to everybody on your network.

[mtelgkamp]

Okay, it really looks my DNS is very unstable today, so I will have a look if the issue with the first run still is a problem when using another DNS provider. In the meantime, it might be a good thing to make the timeout configurable and give a hint when the DNS does not resolve correctly.

[mtelgkamp]

First dig of the morning: 627msec

full output

dig `openssl rand -base64 8`.ddev.site

; <<>> DiG 9.16.1-Ubuntu <<>> Rj3dCA6UgyI=.ddev.site
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35510
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;Rj3dCA6UgyI=.ddev.site.                IN      A

;; ANSWER SECTION:
Rj3dCA6UgyI=.ddev.site. 60      IN      A       127.0.0.1

;; Query time: 627 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mi Jun 10 07:49:55 CEST 2020
;; MSG SIZE  rcvd: 67

Knowing this, I would be happy to have an option to configure the timeout.

[rfay]

Love your \<details\> \<summary>full output</summary\> - Best thing I've learned today!

No idea why your first-of-the-day resolution is so awful? Is it the particular nameserver? Could I ask what kind of nameserver you're using? Does it change if you use google (8.8.8.8) or cloudflare (1.1.1.1)? Are others using this nameserver, and do they have the same problem? Or I wonder if it's a new bug in Ubuntu 20.04?

I note that there have been many reports of this kind with Ubuntu over the years, and I've already heard of two with Ubuntu 20.04. There's a Stack Overflow question that has some general approaches, https://askubuntu.com/questions/1041526/very-slow-dns-lookup

[mtelgkamp]

I actually changed the nameserver so it maybe is a problem with Ubuntu 20.04. Maybe it is my way to configure the nameserver in Ubuntu, but I will have a look into this later.

[mtelgkamp]

I remembered to test this today. First dig of the day:
;; Query time: 1307 msec
And I already tried to load the Domain in the browser (Firefox) before, so the DNS entry should have been cached...
I will remove all manual changes to my DNS settings to figure out if the problem is there.

[mtelgkamp]

After changing my settings and an update on multiple DNS related packages of Ubuntu that have been released on 2020-0bind9-dnsutils:amd64 (bind9-dnsutils, bind9-host, dnsutils, bind9-libs) the problem did not occur again, so one of these seems to have done the trick. Thank you for having a look into the problem.

[rfay]

So weird. Let's keep this open at least for a while. Looking forward to your ongoing experience. I do wonder if there was something weird going on with ubuntu 20.04.

[rfay]

I ran into this while sitting on my porch with slow wifi, so even though it was sporadic, if I see it, others will. v1.15 will have global config to set the timeout higher.

[rfay]

@mtelgkamp ddev v1.15-rc1 will be out in an hour or two. Since you're a complete expert at this I'd very much appreciate it if you'd do some testing. I know you seem to have speeded up your DNS, but still, it would be great to have your feedback.

[rfay]

Thanks @mtelgkamp it's out there now, you can upgrade any of the standard ways including homebrew, https://github.com/drud/ddev/releases/tag/v1.15-rc1