Nixpkgs: test systemd-networkd-ipv6-prefix-delegation broken

[lheckemann]

The bug

The test mentioned in the title has been failing to build since #95922, but the underlying issue existed beforehand and was only exposed by a bugfix in iputils.
Last tested on a332da8588aeea4feb9359d23f58d95520899e3c.

Incorrect behaviour, identical before and after the iputils bump

When the router machine attempts to ping 2001:db8::, it gets replies from itself. Why is completely unclear to me, but the route it obtains for this address looks suspicious to me:

# ip r g 2001:db8::
anycast 2001:db8:: from :: dev lo table local proto kernel src 2001:db8::ffff:0:0:5eb6 metric 0 pref medium

The source address looks right. The use of anycast and the lo device does not.

The difference

The new failure is due to a bug fix in iputils's ping: "discard packets with wrong source address (comply RFC 1122, commit: 5e052ad)" in iputils (release notes)

The solution

I have no idea. I'm not familiar enough with prefix delegation matters to understand why this is happening, nor how to fix it. cc @andir, author of the test.

[primeos]

When the router machine attempts to ping 2001:db8::, it gets replies from itself. Why is completely unclear to me, but the route it obtains for this address looks suspicious to me:

I haven't had a closer look at the test yet but I assume that router get's an IP in that subnet for a local interface. In that case it would be due to the Subnet-Router anycast address (only get's added if forwarding is enabled, which is why that address can be used to reach a router/gateway from a client).

Should we revert the iputils bump in the meantime?

[primeos]

Ok, so from the test log the issue is (just to post it here as well):

router: waiting for success: ping -6 -c1 2001:DB8::
router # ping: connect: Network is unreachable
[...]

And in nixos/tests/systemd-networkd-ipv6-prefix-delegation.nix:

    # wait until the uplink interface has a good status
    router.wait_for_unit("network-online.target")
    router.wait_until_succeeds("ping -6 -c1 2001:DB8::")

    # shortly after that the client should have received it's global IPv6
    # address and thus be able to ping the ISP
    client.wait_until_succeeds("ping -6 -c1 2001:DB8::")

Since we usually shouldn't disable forwarding on the router (even only for eth1 (ISP interface)) it would seem like the best solution to assign a different static IPv6 address to eth1 of isp as the test comments suggest that 2001:DB8:: is meant to ping the ISP.

[primeos]

cc @andir as the test author

[lheckemann]

@primeos

Ok, so from the test log the issue is (just to post it here as well):

router: waiting for success: ping -6 -c1 2001:DB8::
router # ping: connect: Network is unreachable
[...]

not quite — the "Network is unreachable" error is expected for the first attempt, since the prefix delegation/router advertisement/whatever hasn't taken place yet. The problem is that later attempts fail silently (with no error, just as if the echo request were dropped), since a route is available and a packet is sent, but no reply arrives (at least not from an expected source address).

What also "fixed" the test was making the prefix length of the address for the isp machine 32 instead of 64 (making the route setting setuid wrapper script obsolete). I'm not sure exactly why this works — I think it may be because it then sends router advertisements which router accepts, giving it the knowledge of where to reach 2001:db8::. While semantically (probably) incorrect, it may be an adequate solution given that the isp machine is more of a test fixture than a test subject (the test being about networkd's prefix delegation)?

(also, I already cc'd Andi in the issue description ;) )

[primeos]

not quite — the "Network is unreachable" error is expected for the first attempt, since the prefix delegation/router advertisement/whatever hasn't taken place yet.

Ah, that makes sense. I thought this was due to your mentioned iputils change and was already confused why they would return a wrong error message and why the output suddenly stops... :facepalm:

The problem is that later attempts fail silently (with no error, just as if the echo request were dropped), since a route is available and a packet is sent, but no reply arrives (at least not from an expected source address).

And that's due to the iputils change? If that's the case and they don't return an error then we should open an upstream issue as well.

What also "fixed" the test was making the prefix length of the address for the isp machine 32 instead of 64 (making the route setting setuid wrapper script obsolete).

That's indeed strange. IIRC only IPv6-Addresses with /64 prefixes should be assigned to interfaces so maybe the kernel doesn't add the Subnet-Router anycast address if a different prefix length is used? (nvm, router should still have an /64 address so that wouldn't explain it - but maybe radvd does something strange then or doesn't even send RAs since SLAAC only works with /64 prefixes?)

---

Edit: Regarding the 64-bit boundary: RFC 7421 and RFC 4291:

For all unicast addresses, except those that start with the binary value 000, Interface IDs are required to be 64 bits long and to be constructed in Modified EUI-64 format.

Though in practice this is sometimes/(often?) ignored and non 64-bit prefixes usually work (with some exceptions/limitations).

---

(also, I already cc'd Andi in the issue description ;) )

Sorry, I was a bit short of time and only had a brief look (felt a bit responsible).
Anyway, huge thanks for looking into this :heart: :) Also @vcunat for noticing the test regression and pinging us.

[vcunat]

In case you're bored, the today's staging-next merge introduced at least one new channel-blocking regression.

[lheckemann]

And that's due to the iputils change? If that's the case and they don't return an error then we should open an upstream issue as well.

No, I think iputils is doing everything right — pinging an address and not getting a reply (from an expected address) should make it time out eventually and report 100% packet loss. Now I'm confused though, since the test logs don't look like it's doing that, even though it _did_ when I ran it manually iirc.

[Emantor]

Looking at the routing table of the ISP:

::1 dev lo proto kernel metric 256 pref medium
2001:db8::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium

It's clear that the ISP machine does not know how to answer pings from any prefix within 2001:db8::/36 without it being exactly 2001:db8::/64 by chance. The ISP delegates prefixes from 2001:db8::/36 but does not have a routable entry for any but one /64 prefix. IMO the correct fix here is to use 2001:db8::/36 to have a route for all delegated prefixes.

[andir]

Looking at the routing table of the ISP:

::1 dev lo proto kernel metric 256 pref medium
2001:db8::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium

It's clear that the ISP machine does not know how to answer pings from any prefix within 2001:db8::/36 without it being exactly 2001:db8::/64 by chance. The ISP delegates prefixes from 2001:db8::/36 but does not have a routable entry for any but one /64 prefix. IMO the correct fix here is to use 2001:db8::/36 to have a route for all delegated prefixes.

The on-commit hook of the dhcpd instance is supposed to add all required routes to the routing table.

I'll have a look at why this is failing.

[primeos]

@andir have you read https://github.com/NixOS/nixpkgs/issues/96188#issuecomment-679266668? If that's the case than it should make sense to e.g. assign 2001:db8::1 instead of 2001:db8:: to eth1 of isp so that router can actually ping isp (instead of itself).

I'm not sure though if this fixes the ping problem that got introduced by the iputils bump as well but it seems like something that we should fix in any case to make sure that the test works as expected.

[andir]

@andir have you read #96188 (comment)? If that's the case than it should make sense to e.g. assign 2001:db8::1 instead of 2001:db8:: to eth1 of isp so that router can actually ping isp (instead of itself).

I'm not sure though if this fixes the ping problem that got introduced by the iputils bump as well but it seems like something that we should fix in any case to make sure that the test works as expected.

Yeah, that is what I've concluded as well. I've still no idea why the other wouldn't work but that is a valid solution. I am knocked out most of this week due to some medical conditions. If someone creates a PR I'll do my best to review / merge it.

[primeos]

We've already merged #96250 and the test passes now o/ ;) But thanks for your update and offer to help :)

I hope you get well soon :sunflower: and wish you the best of luck :four_leaf_clover:. And thanks a lot for all of your awesome work :heart:.