Synapse: Federation fails when using IPv6 DNS servers

Created on 11 Aug 2016 · 22Comments · Source: matrix-org/synapse

Federation fails with:

2016-08-10 22:27:45,916 - synapse.http.matrixfederationclient - 183 - WARNING - POST-4- {GET-O-2} Sending request failed to matrix.org: GET matrix://matrix.org/_matrix/federation/v1/query/profile?field=avatar_url&user_id=%40benhylau%3Amatrix.org: InvalidA
ddressError - InvalidAddressError: IPv4 port write() called with IPv6 address

Removing my IPv6 DNS servers in /etc/resolv.conf fix federation:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# nameserver 2001:4860:4860::8844 <-- comment out
# nameserver 2001:4860:4860::8888 <-- comment out
nameserver 8.8.8.8

Seems to be a known issue that has come up in #matrix:matrix.org many times.

Source

benhylau

👍5

Most helpful comment

Yup. the official bug for this is https://matrix.org/jira/browse/SYN-254

ara4n on 27 Aug 2016

👍2

All 22 comments

Yup. the official bug for this is https://matrix.org/jira/browse/SYN-254

ara4n on 27 Aug 2016

👍2

...and now fixed on develop by #1696, we believe :)

ara4n on 20 Dec 2016

...except apparently it's still broken on develop :(

ara4n on 20 Dec 2016

@glyph Do you know the state of using IPv6 resolvers in twisted, if there's any work going on regarding it? Currently twisted's trunk still doesn't work with them.

kyrias on 20 Dec 2016

@kyrias I'm afraid I don't understand the question. Twisted's trunk only recently added support for them. What do you consider an "IPv6 resolver"?

glyph on 20 Dec 2016

@glyph An IPv6 resolver is a resolver queried over IPv6.

kyrias on 20 Dec 2016

@kyrias Oh, so just an IPv6 address for looking up A records, not a resolver that can resolve AAAA records? Huh. In what sense does that not work? Are you talking about something in twisted.names?

glyph on 21 Dec 2016

@glyph Yes.

>>> from twisted.names import client
>>> client.getHostByName('google.com')
<Deferred at 0x7f1818a1f290 current result: <twisted.python.failure.Failure twisted.internet.error.InvalidAddressError: >>

kyrias on 21 Dec 2016

@kyrias This is definitely a serious bug in Twisted. It's been filed here - https://twistedmatrix.com/trac/ticket/8877 - unfortunately that's still fairly recent. Sorry about that!

glyph on 22 Dec 2016

👍1

@glyph Weird: it's a bug with "normal" priority in Twisted...

swift11 on 29 Dec 2016

@swift11: Priority is not really used in our bugtracker. You can tell if something is _really_ high priority by whether it gets fixed or not :)

glyph on 29 Dec 2016

👍1

https://github.com/twisted/twisted/pull/649

reactormonk on 29 Dec 2016

Time to update dependencies, boys.

reactormonk on 18 Jan 2017

🎉1

The dependency won't be updated until there's a new twisted release. ;P

But awesome work fixing it!

kyrias on 18 Jan 2017

@reactormonk Awesome!! huge thanks for sorting it out :) Will close the bug once we've bumped the dep.

ara4n on 18 Jan 2017

Since 16.7 was full of ghosts, the next release will be 17.1 or 17.2.

glyph on 19 Jan 2017

It looks like 17.1 was just released yesterday, going to try and test it out.

EDIT: Doesn't seem to work, getting a 401 UNAUTHORIZED in homeserver.log when I try to join #matrix:matrix.org through my homeserver:

2017-02-12 10:38:10,226 - synapse.access.http.8008 - 59 - INFO - GET-25- 127.0.0.1 - 8008 - Received request: GET /_matrix/client/r0/directory/room/%23matrix%3Amatrix.org?access_token=<redacted>
2017-02-12 10:38:10,227 - synapse.http.outbound - 124 - INFO - GET-25- {GET-O-1} [matrix.org] Sending request: GET matrix://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23matrix%3Amatrix.org
2017-02-12 10:38:10,229 - twisted - 131 - INFO - - /etc/resolv.conf changed, reparsing
2017-02-12 10:38:10,230 - twisted - 131 - INFO - - Resolver added ('2001:4860:4860::8844', 53) to server list
2017-02-12 10:38:10,230 - twisted - 131 - INFO - - Resolver added ('2001:4860:4860::8888', 53) to server list
2017-02-12 10:38:10,232 - twisted - 131 - INFO - - DNSDatagramProtocol starting on 43979
2017-02-12 10:38:10,232 - twisted - 131 - INFO - - Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x41b7ad0>
2017-02-12 10:38:10,233 - synapse.metrics - 162 - INFO - GET-25- Collecting gc 0
2017-02-12 10:38:10,252 - twisted - 131 - INFO - GET-25- DNSDatagramProtocol starting on 39985
2017-02-12 10:38:10,252 - twisted - 131 - INFO - GET-25- Starting protocol <twisted.names.dns.DNSDatagramProtocol object at 0x41aadd0>
2017-02-12 10:38:10,253 - twisted - 131 - INFO - GET-25- (UDP Port 43979 Closed)
2017-02-12 10:38:10,253 - twisted - 131 - INFO - GET-25- Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x41b7ad0>
2017-02-12 10:38:10,271 - synapse.http.endpoint - 248 - INFO - GET-25- Connecting to 83.166.64.33:8448
2017-02-12 10:38:10,274 - twisted - 131 - INFO - GET-25- Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x41bbf38>
2017-02-12 10:38:10,275 - twisted - 131 - INFO - GET-25- (UDP Port 39985 Closed)
2017-02-12 10:38:10,275 - twisted - 131 - INFO - GET-25- Stopping protocol <twisted.names.dns.DNSDatagramProtocol object at 0x41aadd0>
2017-02-12 10:38:10,296 - synapse.storage.TIME - 207 - INFO - GET-25- Total database time: 0.000% {update_presence(0): 0.000%, update_cached_last_access_time(0): 0.000%, store_device(0): 0.000%} {}
2017-02-12 10:38:13,494 - synapse.http.outbound - 209 - INFO - GET-25- {GET-O-1} [matrix.org] Result: 401 Unauthorized
2017-02-12 10:38:13,495 - root - 180 - WARNING - GET-25- Error retrieving alias
2017-02-12 10:38:13,496 - synapse.http.server - 126 - ERROR - GET-25- 401: Unauthorized
Traceback (most recent call last):
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 116, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 255, in _async_render
    callback_return = yield callback(request, **kwargs)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/rest/client/v1/directory.py", line 49, in on_GET
    res = yield dir_handler.get_association(room_alias)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/handlers/directory.py", line 177, in get_association
    retry_on_dns_fail=False,
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/federation/transport/client.py", line 185, in make_query
    timeout=10000,
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 390, in get_json
    timeout=timeout,
  File "/home/jython234/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
  File "/home/jython234/synapse/lib/python2.7/site-packages/synapse/http/matrixfederationclient.py", line 219, in _create_request
    response.code, response.phrase, body
HttpResponseException: 401: Unauthorized

jython234 on 12 Feb 2017

@jython234 let us know how you get on :)

ara4n on 12 Feb 2017

It turns out my error was because of incorrect reverse proxy setup, not IPv6. I fixed it and all is well now! This should be fixed, I submitted a pull #1909 which simply bumps the dependency.

jython234 on 12 Feb 2017

so this is fixed by upgrading Twisted to 17.1.0. We should recommend that in the README, but this is fixed.

richvdh on 31 Mar 2017

🙌 🎉 The system works!!!🎉🙌

glyph on 31 Mar 2017

readme-recommendation PR is https://github.com/matrix-org/synapse/pull/2088 ftr. and \o/ for the system working

ara4n on 31 Mar 2017

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Tell people to give synapse the full certificate chain, not just the single certificate

babolivier · 3Comments

restricting registration with email regex prevents adding/verifying nonconforming addresses after the account exists

williamkray · 3Comments

`notif_template_html` and `notif_template_text` settings should have sensible default values

ukcb · 4Comments

Automatically invite/join new users to community

AnwariasEu · 3Comments

synapse fails to start under Twisted >= 18.4

richvdh · 3Comments