Stack: Stack resolves addresses through DNS queries rather than NSS on Ubuntu 16.10

[Blaisorblade]

This breaks Stack because Ubuntu uses a broken DNS proxy, part of systemd. Some of the reports in #2536 appear to boil down to this issue. Since it's systemd's fault, other distros might soon be affected.

I haven't reproduced this myself—but the best evidence comes from this strace log:
https://github.com/commercialhaskell/stack/issues/2536#issuecomment-285327722. I might still be missing something.

Using DNS rather than NSS is not fully compliant with what Ubuntu and systemd expect. Stack seems to be using standard APIs, so I suspect weird linking trouble (NSS must always be dynamically linked).

Using DNS is bad because Ubuntu adds a systemd-provided DNS proxy on IP 127.0.0.53, and that does not follow CNAME records correctly (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1647031). Amazon S3 uses CNAME so stack can't connect to S3. I expect connections to other CNAME-using hosts to also break, but #2536 suggests only S3 is affected.

A more complete analysis is in https://github.com/commercialhaskell/stack/issues/2536#issuecomment-286753723, but I suspect this summary should cover Stack's fault.

[Blaisorblade]

Update: stack is statically linked against MUSL, as I discovered chatting with @snoyberg.

This comment from Rust guys confirms that will not respect NSS configs.

It's unclear how unsupported this configuration is on Ubuntu, because other teams also avoid NSS, including Go (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320/comments/1). And Ubuntu might have fixed https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1647031 in updates (I'll ask #2536 reporters).

[borsboom]

You can try downloading a dynamically linked stack executable from https://github.com/commercialhaskell/stack/releases/download/v1.3.2/stack-1.3.2-linux-x86_64.tar.gz. This is executable was built on Debian 7 and is dynamically linked with glibc and libgmp.

[decentral1se]

Seems to be a number of network related issues! @Blaisorblade, next steps here?

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1647031 ended up Invalid!?

[Blaisorblade]

Searching my nick finds an answer: linking against musl (as we do) "is generally broken and cannot be supported". I understand on 16.10 the bug is invalid because all configurations that trigger it are unsupported. I'd check if 16.10 is still supported, and for how long, and maybe recommend users to upgrade if they can if later versions avoid the issue as they should.

For minimal context: musl is an alternative to glibc, the library providing core services, and apparently on Ubuntu 16.10 they're less compatible than elsewhere. Stack precompiled binaries use musl for $reasons (not sure myself). Hope this summary is enough for non-Unix geeks.

Dynamic linking against libc (regardless of how Haskell libraries are linked) avoids the problem, but nobody tested @borsboom 's binary, probably because the people who had the issue are not here but in #2536 (and not all of those relate to this issue, but the Ubuntu 16.10 people are). Using that binary and upgrading by compiling new versions from sources should work.

[dchagniot]

Already written in #2536 , the dynamic linked executable solved my issue:
"ConnectionFailure getAddrInfo: does not exist (Name or service not known)"

I used the installer script (get.haskellstack.org) in order to get stack (Arch linux) and it
will install the static version. I don't understand why more people don't run
into this issue.

[borsboom]

We'll likely to switching back to dynamic binaries as the default for the next major release of Stack.

[nh2]

Ubuntu 16.10 is end-of-life'd now.

What is the status of _current_ Ubuntus and distros?

Some of the linked Launchpad issues say that systemd-resolved not following CNAME is fixed now, and the systemd ticket https://github.com/systemd/systemd/issues/4254 confirms that.

I'm not clear on the status of https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320; it suggests problems with providing NSS services through systemd-resolved and problems with DNSSEC are still there.

[snoyberg]

This issue can be closed, as the problem has been resolved upstream.