Tailscale: Routing loops/sticky sockets: don't route tailscaled traffic through tailscale routes

[danderson]

Classic VPN problem: VPN installs routes, the VPN's daemon's own traffic starts going over the VPN, and everything craps out.

There's various hacks to make this work, which are well documented at https://www.wireguard.com/netns/ . We need to do something like this in tailscaled on linux.

Weirdly, there is one method that's not listed on the Wireguard page, and which I think is basically perfect for our needs: policy routing's suppress_ifgroup option. man ip-rule says:

              suppress_ifgroup GROUP
                     reject routing decisions that use a device belonging to
                     the interface group GROUP.

Translated a bit: if we make an interface group tailscale, and add tailscale0 to that group, we have a way to express "route these packets as if tailscale0 didn't exist", which is exactly what we want.

All that remains from there, is to mark packets that originate from tailscaled, and push them through a slightly different set of routing policies. Aside from that, we can continue business as usual and have everything work out. Sketch of proof below.

Before tailscaled, the network stack looks something like this:

# ip rule
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# ip route show table main
default via 192.168.16.1 dev eth0 proto dhcp metric 600
192.168.16.0/24 dev eth0 proto kernel scope link src 192.168.16.10 metric 600 

Tailscaled comes up, connects to control, receives a configuration that pushes 0.0.0.0/0 to some other node. It does the following to configure networking:

1. Create interface tailscale0, via tuntap.
2. Run ip link set dev tailscale0 group 5189, which places the interface in the 5189 interface group (careful: ip link add also has a group option, but that one mucks with multicast groups)
3. Run ip rule add priority 5189 fwmark 5189 table main suppress_ifgroup 5189
4. Run ip rule add priority 5190 fwmark 5189 table 5189
5. Run ip route add unreachable 0.0.0.0/0 table 5189
6. Bring up tailscale0, add preferred default route: ip route add default via 100.101.102.103 dev tailscale0 metric 10
7. tailscaled sets SO_MARK=5189 on all sockets it makes, both listeners and dial-outs.

Network stack now looks like this:

# ip rule
0:  from all lookup local
5189:   from all fwmark 0x1445 lookup main suppress_ifgroup 5189
5190:   from all fwmark 0x1445 lookup 5189
32766:  from all lookup main
32767:  from all lookup default

# ip route show table main
default via 100.101.102.103 dev tailscale0 proto kernel metric 10
default via 192.168.16.1 dev eth0 proto dhcp metric 600
192.168.16.0/24 dev eth0 proto kernel scope link src 192.168.16.10 metric 600 

# ip route show table 5189
unreachable default

Packet flow for a random ping packet:

• Rules 5189 and 5190 don't match (wrong fwmark). Execute route lookup in table main.
• Match default via 100.101.102.103 dev tailscale0, push packet into tailscale0.
• tailscaled receives packet, does fun stuff to it, eventually transmits a wireguard packet out its UDP socket.
• Rule 5189 matches because UDP socket set fwmark 5189. Execute route lookup in table main, suppressing ifgroup tailscale.
• Match default via 192.168.16.1 dev eth0, push packet out eth0.

Misc points:

• Rule 5190 is there to prevent a fallthrough into a general routing table lookup, which would create a routing loop. If the suppressed lookup fails, we want that to be a hard "address unreachable".
• 5189 is the keys above "tail". We will probably have to exercise more care in selecting the fwmark, since some sysadmins use it as a bitmask.
• Partially cleans up after itself on crash. Routes via tailscale0 are removed, which is enough for the network stack to revert to VPN-less behavior. IP rules and the 5189 routing table linger harmlessly. Obviously we clean those up on normal exit.

Concerns:

• We have to SO_MARK every socket in tailscaled. Unmarked sockets will use VPN routes. We could make the IP rules match on a UID range, if we get tailscaled running as its own user with capabilities granting privileges. However this would break just running tailscaled as root without a service daemon to set things up right.
• Maybe this doesn't actually work as documented? I'm worried that it doesn't appear in the WireGuard rundown of options, even though it's very close to the SO_NOTOIF patch Jason tried to upstream.

┆Issue is synchronized with this Asana task by Unito

[danderson]

Another option for marking the packets, which doesn't require marking all sockets (which might get dicey with 3p libraries): use a gid.

When installing tailscale, create a tailscale-bypass group. Run tailscaled with tailscale-bypass as a supplementary group. Add a netfilter rule: iptables -t mangle -A OUTPUT -m owner --socket-exists --gid-owner 1234 --suppl-groups -m comment --comment "tailscale" -j MARK --set-mark 5189

Rest of the flow is as before. Unfortunately, the knob we _really_ want (mark based on PID) doesn't exist, which is why we'd have to resort to a UID (hard to run as root) or GID (fine because we can make it a supplementary GID).

People doing manual install or using legacy init systems would have to create the group by hand, or if tailscaled can't find it, the marking won't occur and some routing configurations might break. We can warn about that, or just refuse to install subnet routes if the group isn't present.

[bradfitz]

People doing manual install or using legacy init systems would have to create the group by hand, or if tailscaled can't find it, the marking won't occur and some routing configurations might break.

You objected to auto-creating /var/lib/tailscale so you'll probably object to this, but: when run as root we could auto-create a group for them unless they opt-out somehow.

[bradfitz]

FWIW, I'm not very worried about the SO_MARK work. It's not too many.

[danderson]

I considered auto-creating, but that feels especially rude compared to filesystem twiddling. I think of creating groups as something package managers do, not something that random binaries help themselves to. I'd do it if SO_MARK is untenable, but I'd feel dirty.

If SO_MARK is feasible (and we're okay with saying we will never use a 3p package that doesn't let us pass in Dial and/or Listen functions), I think that's a fine way to go. We can keep groups in reserve if that doesn't work out.

[bradfitz]

I considered auto-creating, but that feels especially rude compared to filesystem twiddling.

I'd be happy if tailscaled at least failed on start-up with a message saying that its required group (with a default flag value of, say, tailscale (bypass sounds scary as a user)) doesn't exist, and included a command I could copy/paste to create it on Linux. (I don't create create groups and add users to groups regularly enough to have it memorized and I can't imagine I'm alone in that.)

[apenwarr]

There are a lot of ways gid marking could go wrong, I don’t really
recommend it. And when it fails it would be extra hard to trace why. It has
echoes of dependency injection, or spooky action at a distance.

Generally an explicit flag on each socket is much more straightforward and
does what you want. Plus a regular user can run a test program with the
fwmark set without having to mess around with sudo or their gid.

On Fri, Mar 6, 2020 at 11:03 Brad Fitzpatrick [e-mail]
wrote:

I considered auto-creating, but that feels especially rude compared to
filesystem twiddling.

I'd be happy if tailscaled at least failed on start-up with a message
saying that its required group (with a default flag value of, say,
tailscale (bypass sounds scary as a user)) doesn't exist, and included a
command I could copy/paste to create it on Linux. (I don't create create
groups and add users to groups regularly enough to have it memorized and I
can't imagine I'm alone in that.)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tailscale/tailscale/issues/144?email_source=notifications&email_token=AAAFA4GWYGO5ZSVCGY2BSGDRGENEHA5CNFSM4LCC2PQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOB32WQ#issuecomment-595836250,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAFA4EMYP6N3OFLF5GG46LRGENEHANCNFSM4LCC2PQA
.

[danderson]

Yup, only risk of the fwmark is if there are sockets we don't control in the process. I don't think we have any of those, and we can be disciplined about not letting any sneak in.

[apenwarr]

Whoops, this is still an open bug on Windows.

[danderson]

This is now fixed in unstable. All of Tailscale's traffic is bound to the non-Tailscale default route interface for the machine.

This means that machines sitting on a subnet exported by a subnet router should now work correctly - i.e. traffic will flow over Tailscale to the subnet router, rather than the previous behavior where tunnels would fail to come up due to a routing loop.

Things not implemented by this bug:

• Disabling the subnet route when on the same subnet. That's a separate bug that has some security implications requiring careful design.
• Full "privacy" default route forwarding. With this change you can forward 0.0.0.0/0 correctly, but it'll only relay IPv4. Nodes with IPv6 connectivity cannot route the v6 stuff over Tailscale yet.

[AlekSi]

Full "privacy" default route forwarding. With this change you can forward 0.0.0.0/0 correctly

Is there some documentation for that?

[danderson]

It's the same as any other subnet routing: https://tailscale.com/kb/1019/subnets . Just use 0.0.0.0/0 as the subnet route.

But again, note that this will not forward IPv6 traffic, so you will have a split: IPv4 traffic will relay over Tailscale, but IPv6 will not use the VPN.

[AlekSi]

That page says "Default routes, 0.0.0.0/0, are not currently supported." :)

[danderson]

That's true for the current stable release. It works on unstable builds though :). We'll update the KB when we release it to stable.