Node: order of getaddrinfo results not respected

Created on 20 Apr 2016  Â·  59Comments  Â·  Source: nodejs/node

  • Version: v5.10.1
  • Platform: Linux archer 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64 GNU/Linux
  • Subsystem: dns/cares_wrap

The results returned by getaddrinfo are reordered, giving preference to ipv4 addresses. The explanation given in #4693 seems unsatisfactory. Not implementing happy eyeballs is a reasonable design decision, of course, but why does that mean ipv4? It should mean _taking the first entry_. At the moment node is deliberately defying the system configuration.

dns

Most helpful comment

Happy Eyeballs - for those who haven't heard the term

All 59 comments

As I understand it, without the reordering, ipv6 addresses could be the first entry, which would break anywhere that doesn't have an ipv6 stack. The current implementation is intentional in order to give preference to ipv4. /cc @bnoordhuis

Is this still a problem when using the latest v6 release candidate (aka is this fixed by #6021)?

The DNS hints aren't related to the reordering.

@bnoordhuis would the dns.ADDRCONFIG flag mitigate the problem?

As I understand it, the proper place to configure preferences is via gai.conf.

Snippet from my gai.conf:

# precedence  <mask>   <value>
#
#    Defaults:
#
#precedence  ::1/128       50
#precedence  ::/0          40
#precedence  2002::/16     30
#precedence ::/96          20
#precedence ::ffff:0:0/96  10
#
#    For sites which prefer IPv4 connections change the last line to
#
#precedence ::ffff:0:0/96  100

would the dns.ADDRCONFIG flag mitigate the problem?

Only partially. AI_ADDRCONFIG means 'return AAAA entries only when there is a configured IPv6 interface' but the presence of such an interface doesn't say anything about IPv6 traffic actually being routable. (That's true for IPv4 as well, of course; my point is that AI_ADDRCONFIG isn't a fix per se.)

We made the decision to prefer IPv4 over IPv6 three or four years ago when IPv6 support was much more spotty than it is today. It's something we can reconsider if the majority thinks it's a good idea.

There are likely two things that can happen here:

  1. We can improve the documentation to indicate why the results are ordered the way they are, and
  2. We can add an option that would tell the impl not to re-order the output. The current default behavior would remain.

This is clearly a violation of the principle of least astonishment and I don't see how adding documentation is going to help. Hardly anyone affected by this will be using the dns module directly. It took me half a day before realizing my issue was with the dns module. Since then I have put a workaround in place and am no longer affected. Still, I think you should slowly phase out this odd logic. IMHO, over time, keeping it will hurt more than it helps.

from what i gather, this is possibly the source of my issue with npm install failing (a possible regression of https://github.com/npm/npm/issues/6857)

This time i'm running alpine (hence, muslc) under docker, again, IPv6-only network.

~ # node -e "dns.lookup('registry.npmjs.org', (...args) => console.log(args))"
[ null, '151.101.112.162', 4 ]
~ # 

note that musl implements rfc 3484/6724.

this works fine in ruby, btw:

# pry -r 'open-uri'
[1] pry(main)> open("https://registry.npmjs.org/")
=> #<StringIO:0x0055ecc25d4fd0
 @base_uri=#<URI::HTTPS https://registry.npmjs.org/>,
 @meta=
  {"server"=>"CouchDB/1.5.0 (Erlang OTP/R16B03)",
   "content-type"=>"application/json",
   "cache-control"=>"max-age=300",
   "content-length"=>"194",
   "accept-ranges"=>"bytes",
…etc…

from what i gather, this can be fixed with:

diff --git a/lib/dns.js b/lib/dns.js
index cbb994b8f2..c161a89a66 100644
--- a/lib/dns.js
+++ b/lib/dns.js
@@ -140,7 +140,7 @@ exports.lookup = function lookup(hostname, options, callback) {
     if (all) {
       callback(null, []);
     } else {
-      callback(null, null, family === 6 ? 6 : 4);
+      callback(null, null, family === 4 ? 4 : 6);
     }
     return {};
   }

That fix would default the family to 6 iff the caller isn't asking for all families. However, most of the time the caller should be asking for all families. Even if that was not the case, changing defaults is a mean thing to do to clients.

Generally any caller should ask for all families and then try the results in-order. If the caller were to implement happy eyeballs they would ask for all and concurrently try the per-familiy subsequences in-order.

IMHO the proper way to fix this is to apply #4693. Whether to happy eyeball or not is orthogonal.

Happy Eyeballs - for those who haven't heard the term

every month or two, i try a fresh major release of nodejs, only to find this issue is still there.

(@jasnell commented on 22 Apr 2016) There are likely two things that can happen here:

  • We can improve the documentation to indicate why the results are ordered the way they are, and

    • We can add an option that would tell the impl not to re-order the output. The current default behavior would remain.

i highly agree with @benschulz here:

(@benschulz commented on 25 Apr 2016 • edited) This is clearly a violation of the principle of least astonishment and I don't see how adding documentation is going to help. Hardly anyone affected by this will be using the dns module directly. It took me half a day before realizing my issue was with the dns module. Since then I have put a workaround in place and am no longer affected. Still, I think you should slowly phase out this odd logic. IMHO, over time, keeping it will hurt more than it helps.

and believe that #4693 is the correct way forward — in 7.x+

Should this remain open? (I'm guessing the answer is "yes" but would be happy to find out I'm wrong.)

This is your project and you are free to take it in any direction you please. If you think the status quo is good enough or even right, you should close the ticket. All I can tell you is that I had a really rough time with this. But that's just one data point, I imagine there are several more people affected.
Fermi estimate: Two people have raised the issue; if we conservatively posit that .1‒1% of those affected manage to figure out the root cause and find this report, it'd be around 200‒2000 people affected in total.

Hope that helps. :P

Edit: Added missing "affected" qualifier. =)

@bnoordhuis looking at your comments:

https://github.com/nodejs/node/issues/6307#issuecomment-212569239

We made the decision to prefer IPv4 over IPv6 three or four years ago when IPv6 support was much more spotty than it is today. It's something we can reconsider if the majority thinks it's a good idea.

https://github.com/nodejs/node/pull/4693#issuecomment-172807221

We don't do happy eyeballs. If you change the order, that would have to change. That's a big can of worms.

I understand that you're against implementing happy eyeballs, but not necessarily against just letting the results come back in their original order. Is your second comment saying that ceasing to modify the order will force us to implement happy eyeballs?

I understand that this may be a massive breaking change, and thus a non-starter, but if it's possible to make that change we can at least consider it (even if it's just an option for now, and doesn't become the default till Node 18.x).

Is your second comment saying that ceasing to modify the order will force us to implement happy eyeballs?

Not exactly. Happy eyeballs is a transition mechanism. It's not needed in a world where IPv6 connectivity is good enough. Do we live in that world yet? Maybe, maybe not; not enough data.

Happy eyeballs is a transition mechanism. It's not needed in a world where IPv6 connectivity is good enough.

That's the part of the argument I don't understand. I set my system up to resolve names to IPv6 addresses and, if that yields no results, to IPv4 addresses. What's the issue? (I consider names with invalid AAAA entries an issue with the name, not my system configuration.)

A valid AAAA entry does not mean there is a network path to that machine.

That was historically a big barrier to IPv6 adoption and is what happy eyeballs tries to address: try IPv6 and IPv4 in parallel and use whatever manages to establish a connection.

A valid AAAA entry does not mean there is a network path to that machine.

So anyone affected has an IPv4 and an IPv6 address and their ISP is unable to route all IPv6 packets? Couldn't they do the reverse of what I do, i.e. prefer IPv4 over IPv6 at the OS rather than application level?

I appreciate the time taken for the various comments, thank you.

Yes, but that takes time and effort to set up. We want to avoid a situation where a node.js release breaks the out-of-the-box experience for a large group of users, especially when it might be difficult to debug for them.

then, the best way forward, as mentioned numerous times in this issue, would be to implement happy eyes in a future version of node.js

then, the best way forward, as mentioned numerous times in this issue, would be to implement happy eyes in a future version of node.js

I don't see anyone in this thread suggesting implementing happy eyeballs in Node.js

Not sure if this is the right bug, but IPv6 support seems to be seriously broken since 0.10... until and including v6.11.2. No version (also not 4.8.2 or 4.8.4) are capable of working in IPv6 only networks (as originally described in https://github.com/bitinn/node-fetch/issues/66). It seems that npm is only trying IPv4, where everything else works.

This breaks various software building on top of node and will cause a variety of problems when bigger vendors like Apple will soon require IPv6 networks to function.

Is there any chance that there will be a proper IPv6 support in nodejs in the near future?

I am sorry, I actually meant to refer to https://github.com/npm/npm/issues/6857

@telmich just for clarity, there is a separation of responsibility between node and npm (although there is obviously strong coupling), and node does support other package managers.
My question is, can you make it clear whether node isn't working for you in an IPv6 only environment, or is it npm that fails?
If it is the former, can you open a new issue? Optimally with a clear description of the scenario, and a minimal reproducing code?
If it is the later, a better recourse would be to open an issue in the npm repository, or evaluate an alternative package manager.

@benschulz and @igalic will a configuration mechanism that will reorder/restore-the-order of getaddrinfo result fulfill the requirements of your use case?
Such a mechanism could be an environment variable (like NODE_OPTIONS), a global variable (like require.extensions) or a cli flag (like --zero-fill-buffers)

env options or global config vars would work.
cli options would not, cuz you don't know how many layers of software you're working under…

the (often unknown (to an end-user)) layers of indirection are also why it takes so long for people to pup up here after having exhausted all other venues.
when you read thru people's debugging history of this bug it's always,

  • blame the setup
  • verify other software
  • suspect npm (or equivalent)
  • npm devs tell you it's a dns (_core module_) issue
  • validate issue (wow, it's so obvious now!)

this behaviour of the software is really unintuitive.

but then people finally get here (at least those that persisted thru all these steps) and find a bug where we're arguing whether in a world where we've run out of IPv4 addresses, we should support an IPv6 only setup. for over a year now.

For us it was even more painful, because we migrated rocketchat to the IPv6 network and it partially works (as in: it can receive connections!), but it cannot connect to elsewhere.

I think this should really not be a configuration option, as most other software in the world has solved it properly now, without the need for quirk or tuning.

Implementing happy eyeballs should not be hard and would solve the problem for everyone instantly.

@telmich re configuration option: https://tickets.puppetlabs.com/browse/SERVER-393
despite the JVM defaulting to prefer IPv6 for binding in dual-stack setups (see https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html), it doesn't seem to do the same for outgoing connections

Implementing happy eyeballs should not be hard and would solve the problem for everyone instantly.

You can do this in your application right now, you don't have to wait for node to do it for you.

Matter of fact, I don't think node should because it's absolute hell to debug when it goes wrong. A cure worse than the disease.

@telmich FWIW, I don't think your issue is this issue. I can testify from first-hand experience that node can work just fine in an IPv6-only environment. If you have a test case of sorts, please file a new issue.

@bnoordhuis Well, the authors of NPM seem to disagree with you. If you look at https://github.com/npm/npm/issues/6857 it shows that npm does not work in IPv6 only networks, because of core dns issues which brings me to #6307 - wait, this is here again.

I have the feeling that you are holding up to a theory ("hell to debug") that is not true in practice. I highly recommend to have a look at the code of curl or chromium, both supporting happy eyeballs, both doing it in an easy to debug manner, by printing out the ip when inspecting the connection.

This is exactly what I as a user would expect, whereas nodejs' "use only IPv4" is for all practical reasons stopping to use it in IPv6 environments.

You might also anticipate that IPv6 is actually the new standard that is pushed out aggressively, having grown to 30% of the network traffic in Switzerland.

In the end, it is very easy: you can claim that IPv6 is not worth/interesting/whatever and people who try to use it in newer data centers will learn that nodejs is the IPv6 "bad kid" and basically doesn't work and eventually move on to something else.

Or you anticipate that IPv6 is actually already there and the lack of proper IPv6 support is hitting your users right now (and for almost 3 years, for the matter) and implement a proper solution. Be that happy eyeballs or something else fancy (which I highly disrecommend from seeing that chances of doing it wrong are 99%), is then up to you.

Well, the authors of NPM seem to disagree with you.

Fine, but the authors of npm don't know node as well as I do.

In the end, it is very easy: you can claim that IPv6 is not worth/interesting/whatever

Have you actually _read_ through this issue? I'm pushing for abandoning the current hack we have.

I wonder whether the best way forward is an opt-in which can be turned into opt-out with some major version change and ultimately be removed (hopefully).

I was under the impression a dns.lookup() option was added some time ago that returns the results as-is but seems I was mistaken. I opened #14731 to add one.

Looking at #14731, I wonder if you suggest to use a random address to connect to, @bnoordhuis?
If that is the case, you will have randomly success on IPv6 only networks, instead of always success, compared to happy eyeballs.

Correct me, if I am wrong.

Should this be closed since #14731 has landed? You can access the results from getaddrinfo directly (no re-ordering) by passing the { verbatim: true } option.

/cc @bnoordhuis

Yes, it's fixed now. I'll close.

Not sure if it's related, but I'm using the IPv6 only wifi access point at FOSDEM and "nmp install" failed with

24 http request GET https://registry.npmjs.org/dredd
25 silly fetchPackageMetaData Error: connect ENETUNREACH 151.101.0.162:443 - Local (0.0.0.0:0)

For the record. RFC 1123 stipulates that addresses should be tried in order and that NETWORK errors should be handled for multi-homed hosts. With the introduction of IPv6 and DNS64 there are lots of multi-homed hosts. Happy Eyeballs is just a mechanism for dealing with a common NETWORK error. It is not and never will be a "Transition Mechanism".

Happy Eyeballs wouldn't be a thing if IPv6 didn't exist, or had worked great out of the gate; it was brought into this world to get over IPv6's early adoption hurdle. Privately, I call it "a godawful hack", but in polite company, "transition mechanism" seems apt.

addresses should be tried in order

Wouldn't you agree that conflicts with Happy Eyeballs' "first pass the post" principle? Most applications, Node.js included, call getaddrinfo(AF_UNSPEC); they accept any mix of IPv4 and IPv6 addresses.

Firstly, Happy Eyeballs does in order connection attempts. There are very few cases where a the first connection attempt would have succeeded after the second connection attempt succeeds.

Having multiple TCP connection attempts in flight existed before Happy Eyeballs existed. The only thing Happy Eyeballs did was give the practice a IETF stamp of approval. There was never any requirement for a connection attempt to timeout before starting a new connection attempt to a multi-homed server. A large part of the load balancer business is a direct result of bad multi-home connection attempt handing by TCP clients. But if you want to keep propping up a business model that is wasting 10's of millions of dollars a year rather than do the right thing and support simultaneous connection attempt.

Multi-homed servers are here to stay as are networks where routes to some of the addresses don't work. You can learn from other peoples experiences or continue to use naive connection attempt models.

Guys. It's 2018. It's too late to still DISCUSS on how to fix IPv6 connection establishement issue. Move on, fix it.

  1. Not all "guys" here.
  2. PRs welcome.

I think this is the wrong issue for this discussion. As I said before, whether to do Happy Eyeballs—or some other connection establishing scheme for that matter—is orthogonal.
This issue is about the DNS module. All the DNS module does is resolve names. It should respect the system config when doing so, which is what this issue is trying to address, and I think the right steps are being taken. I hope the opt-in turns opt-out in the near future, but that is a breaking change…

By the way, I appreciate frustration—I "wasted" a day on this issue, after all. But please make an effort to check your attitude before commenting. Especially if you want to see your issue addressed. Snark won't help you, even when you're technically right.

Well opt-out is what is REQUIRED by RFC 1123 regardless of whether is it breaks stuff. The existing stance is causing people to request than name servers return NODATA responses for A record requests despite there being A records at the name.

I hope the opt-in turns opt-out in the near future

Node.js 11, most likely.

Is there any progress on this issue?

@telmich #20710 - turned out getting it past the CI was a lot more complicated than expected. If someone wants to dust off that PR, you're welcome to it.

Any news on this?

It's shame that in 2020 the verbatim is still not set to true by default!
With false NodeJS simply doesn't work in IPv6 only networks...
I had to make a custom patched version of NodeJS to have it working in my IPv6 only network: https://aur.archlinux.org/packages/nodejs-ipv6/

All of what's need is just to switch verbatim to true and inverse the verbatim option so that it force every library to follow the correct order:
````patch
--- lib/dns.js.ipv6 2020-01-26 19:09:18.114911160 +0100
+++ lib/dns.js 2020-01-26 20:31:28.720237093 +0100
@@ -95,7 +95,7 @@
let hints = 0;
let family = -1;
let all = false;
- let verbatim = false;
+ let verbatim = true;

// Parse arguments
if (hostname && typeof hostname !== 'string') {
@@ -109,7 +109,7 @@
hints = options.hints >>> 0;
family = options.family >>> 0;
all = options.all === true;
- verbatim = options.verbatim === true;
+ verbatim = options.verbatim !== false;

 validateHints(hints);

} else {
````

We have actually started to migrate away from nodejs in various projects due to the nodejs incompatibility with IPv6 only networks.

@unixfox would you be willing to open a new issue or PR to track your suggested change?

I believe the issue last time with switching the default was getting the test suite to pass: https://github.com/nodejs/node/pull/20710

I even mention that two comments up... I might take another stab at this but I'm not going to sink days into getting it past the CI.

My patch was indeed inspired by the PR #20710.
I'm not experienced enough to make a PR because it seems that it's quite difficult to make the CI pass the tests with these "two lines" changed.

Instead of creating a new issue for that, shouldn't this issue be reopened because #20710 hasn't landed into the core of NodeJS?

I'm also looking forward to a contributor experienced enough with the core of NodeJS to try again to make the PR #20710 passing the CI tests :smiley:.

It'd be better to open a new issue; this one is long enough already.

31567

In case somebody runs IPv6-only networks and came across exact that issue here and needs a fix asap: One working workaround is to NXDOMAIN A records in your DNS64 service. Without any IPv4 address, it won’t connect and will try the IPv6 address.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

filipesilvaa picture filipesilvaa  Â·  3Comments

loretoparisi picture loretoparisi  Â·  3Comments

seishun picture seishun  Â·  3Comments

Icemic picture Icemic  Â·  3Comments

jmichae3 picture jmichae3  Â·  3Comments