Wpt: `wpt serve` immediately dies with error

[tabatkins]

I get the following error when I run wpt serve:

DEBUG:web-platform-tests:Going to use port 38675
Process ServerDictManager-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 550, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 162, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 132, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 256, in __init__
    self._socket.bind(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "./wpt", line 5, in <module>
    wpt.main()
  File "/usr/local/google/home/tabatkins/wpt/tools/wpt/wpt.py", line 129, in main
    rv = script(*args, **kwargs)
  File "/usr/local/google/home/tabatkins/wpt/tools/serve/serve.py", line 782, in run
    with stash.StashServer(stash_address, authkey=str(uuid.uuid4())):
  File "/usr/local/google/home/tabatkins/wpt/tools/wptserve/wptserve/stash.py", line 29, in __enter__
    self.manager, self.address, self.authkey = start_server(self.address, self.authkey)
  File "/usr/local/google/home/tabatkins/wpt/tools/wptserve/wptserve/stash.py", line 51, in start_server
    manager.start()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 528, in start
    self._address = reader.recv()
EOFError

I'm on Python 2.7.6.

[snuggs]

@tabatkins funny seeing you here....lol I Got the same.

[foolip]

Looks odd, but maybe it's the hosts file bits in https://github.com/w3c/web-platform-tests/blob/master/README.md#running-the-tests?

[foolip]

@jgraham have you seen this error before?

[jgraham]

No. It looks like it's dying trying to start the multiprocessing.Manager used to make the stash cross-process (so that websockets and http requests can share the same data). But I'm not sure exactly why. Maybe it's unable to access the required port? Is there some weird firewall configuration involved?

[foolip]

@snuggs, are you saying you've seen this very same error? What OS are you using? (Tab saw this on Goobuntu.)

[foolip]

@snuggs, did you have the hosts file set up when you saw this error?

[jgraham]

Looking at the code, I wonder if people changed any configuration? In particular I'm not sure if this works if you unset bind_hostname in the config.json file. Otherwise it looks like it tries to get a free port on the current host.

[tabatkins]

It's the lack of HOSTS file setup - I figured I didn't need to do that if I wasn't doing anything fancy, but once I did set them up as specified in the docs, everything worked fine.

Do you want me to edit the OP to reflect the real bug, then (wpt serve should catch this error and give an informative error message), or close this and open a fresh one?

[foolip]

@tabatkins that is very curious. Leaving this bug alone is fine, I'd like to remove my own /etc/hosts changes tomorrow and see if I can get the same error, and make it more informative.

[jgraham]

Oh so we have code to check that the hosts are configured correctly. But at some point that code got wrapped in the stash server context object, so now that check doesn't happen until later.

diff --git a/tools/serve/serve.py b/tools/serve/serve.py
index 74b9b7e55b..9a2825ac34 100644
--- a/tools/serve/serve.py
+++ b/tools/serve/serve.py
@@ -661,9 +661,6 @@ def start(config, ssl_environment, routes, **kwargs):

     ssl_config = get_ssl_config(config, external_config["domains"].values(), ssl_environment)

-    if config["check_subdomains"]:
-        check_subdomains(host, paths, bind_hostname, ssl_config, config["aliases"])
-
     servers = start_servers(host, ports, paths, routes, bind_hostname, external_config,
                             ssl_config, **kwargs)

@@ -721,6 +718,7 @@ def get_ssl_environment(config):

 def load_config(default_path, override_path=None, **kwargs):
     if os.path.exists(default_path):
+
         with open(default_path) as f:
             base_obj = json.load(f)
     else:
@@ -782,6 +780,10 @@ def run(**kwargs):
     if config["bind_hostname"]:
         stash_address = (config["host"], get_port())

+    if config["check_subdomains"]:
+        check_subdomains(config["host"], {"doc_root": config["doc_root"]},
+                         config["bind_hostname"], None, config["aliases"])
+
     with stash.StashServer(stash_address, authkey=str(uuid.uuid4())):
         with get_ssl_environment(config) as ssl_env:
             config_, servers = start(config, ssl_env, build_routes(config["aliases"]), **kwargs)

Should fix it. Going to sleep now, so if anyone else wants to make a PR I will be happy. Otherwise I might remember tomorrow.

[foolip]

I've confirmed that I get the same error as @tabatkins with the hosts removed. I'll try to make a PR following @jgraham's clues.

[foolip]

A mitigation in https://github.com/w3c/web-platform-tests/pull/8291.

[foolip]

Finding some more odd things trying to fix this. It's easy enough to just move stuff up to before the stash server is started, but I'm thinking that if the error messages are good, it doesn't matter.

And for the odd thing, @jgraham, it looks like the stash server will just use config["host"] before normalise_config is applied, so actually using external_host in config.json would lead to something slightly broken?

[jgraham]

I think making the check a precondition is the right solution rather than trying to fix all the places that might depend on the ports.

I agree we might need to move the config processing earlier in the file too.

[snuggs]

@jgraham @foolip @tabatkins sorry for the long delay. Speaking at a conference. Here's my setup.

[foolip]

@snuggs, is that what you put into your /etc/hosts or equivalent? The inclusion of the ports is suspicious, have you tried copy-pasting from https://github.com/w3c/web-platform-tests/blob/master/README.md#running-the-tests?

[snuggs]

@foolip i sure have! I just cannot get the hosts to work. Explicitly used loopback address. Docs seemed a tad vague. But you are clearly helping me along the way and it's greatly appreciated 🙏

[foolip]

@snuggs, what OS are you on?

[snuggs]

@foolip a (intentionally) fairly dated AIR. Hope this helps!

[foolip]

@snuggs were you able to update your /etc/hosts and have ping web-platform.test work in the terminal? If you've gotten that far, then any failure is probably due to something other than what @tabatkins ran into here.

[snuggs]

If you can give me the most up to date directions I can test it out for you. Last I remember was about removing ports. Much appreciated @foolip.

[foolip]

@snuggs, the current directions are to paste the block in https://github.com/w3c/web-platform-tests/blob/master/README.md#running-the-tests into your /etc/hosts. If it doesn't work then, try ping web-platform.test to see if the DNS is working, maybe also restart the browser you're trying it in.

[foolip]

Calling this roadmap since I intend to finish it.