Zerotierone: Support "remote gateway" to route all traffic (default route) through ZeroTier networks

[adamierymenko]

A ZeroTier network could advertise, as part of its configuration, one or more gateways. A user could -- optionally -- after joining a network choose to send all traffic (default route) over that gateway. This would support conventional VPN "tunnel everything" use cases. Been requested before, wanted to officially put into backlog.

[keesbos]

Optionally disabling other network interfaces? That would prohibit hopping. Or maybe only permit networks from same controller?

[adamierymenko]

Not sure I understand... this would route 'default' (with the exception of ZeroTier traffic itself) via a gateway on a virtual network. It wouldn't _necessarily_ affect other networks, and I don't think this would be desired behavior.

Example: let's say I'm a member of a network with a 10.1.0.0/16 IP scheme and another with 28.0.0.0/7 (Earth). If I enabled the use of a gateway on 10.1.0.0/16, I'd want my default traffic e.g. to www.google.com to go through that gateway. But I'd still want the 28.0.0.0/7 network to behave normally. That's a more specific route -- default is catch-all.

[keesbos]

Indeed. Default gateway won't have any effect on other interfaces. For normal "reasonably" concerned security officers this could be enough. Probably good for 99% percent of the use cases where the default gateway is applied.

In a paranoid security environment, the scrutiny officer will want to control more than that: Make sure that whoever connects to your network will _never_ allow others via that node to penetrate the network. E.g. I'm connected to the management network of a organisation with a high risk profile and have also a connection to "earth". That's something that might be unwanted...

[janjaapbos]

That really is depending on your control of the OS. In any case you can disable bridging for the management network. Still, the connected OS can do IP nat for other networks that it is connected to. I'm not sure if you can restrict that at the zerotier device level.

[heri16]

I think this feature is a bad idea as it would place zerotier right in the crosshairs of government censorship. If the government is serious about clampdown, I don't think zerotier traffic can evade the sophisticated great firewalls at the state-level. It is very easy to add in a static route on any operating system, rather than make zerotier not function in many countries.

[adamierymenko]

This is going to be optional, but it likely will make an appearance. It's a highly desired feature among certain classes of enterprise customers that might want to use this to replace conventional hub-and-spoke VPNs.

The reason it hasn't made an appearance yet is that it's a pain in the behind to implement. You have to -- on _each operating system where it is supported_ -- send ZeroTier traffic in such a way that it bypasses the "fake" VPN default route and uses the physical one. But other traffic uses the VPN.

That's easy on hub and spoke VPNs since you have a single static IP endpoint to add a specific route for. But this is a P2P system which means you have multiple ever-changing routes that sprawl all over the Internet.

The solution on some platforms could be using SOCK_RAW and bypassing the kernel's routes and other filters completely. This would also make ZeroTier traffic skip iptables though, which some users might not like -- though other users might consider that a feature. It would be controversial. It would also be highly platform specific and probably impossible on Windows since anything novel related to networking is impossible on Windows without being paid for in the wicked currency of misery and suffering. A hack that should be kept from the eyes of the innocent is usually required.

[whosawhatsis]

Not to pile more onto your plate, but a nice feature for this to support would be the ability to route/not route certain ports to the gateway. For instance, if I'm on an open wifi network, it would be nice to be able to route all of my port 80 traffic (http) through an encrypted tunnel to my home network, but I'd rather have port 443 traffic (https) go directly for latency and bandwidth reasons.

It's probably a terrible idea that would add a ton of complexity, but on the off chance that it's not (or that it is, but is valuable enough to others to implement anyway)...

[hpk42]

I'd be happy already if only linux nodes were supported as an "exit gateway". However, I didn't understand yet if the implementation complexity is on the side of the node who wants to route default traffic through a remote "exit" zerotier node or if it's at the side of the exit node which routes traffic to the wider internets. Could you clarify?

[janjaapbos]

The complexity is at the side that wants to route through the default gateway.

[faddat]

This is another example of something that definitely would help us to make our networks more robust at Klouds. Thanks a lot for all the hard work.

[adamierymenko]

Just to make a note:

There is a well documented hack used by many VPNs for doing this. It turns out you can add two new routes such as (for IPv4):

route add 0.0.0.0/1 [VPN remote gateway IP]
route add 128.0.0.0/1 [VPN remote gateway IP]

This works because more specific routes trump less specific routes. These two routes together constitute a default route since 0.0.0.0/1 gets everything with a 0 first bit and 128.0.0.0/1 gets everything with a non-zero first bit. The OS will pick these over 0.0.0.0/0 because /1 > /0 -- it's more specific.

This also avoids messing with the OS-chosen default route, which is awesome!

The difficulty will be in making sure that our traffic uses the physical default route. We should be able to accomplish this by refactoring OneService.cpp to bind not to wildcard but to each specific physical network card in the system. This also has the added benefit of making recursion (ZT-over-ZT) impossible in an elegant way. (though we already fixed that anyway)

This solution should work everywhere: Mac, Linux, and Windows. If we want we could instead add four default routes in /2: one for (binary) 00, 01, 10, and 11. This would make us trump other VPNs.

(I now cringe every time I use the word trump in a sentence.)

[adamierymenko]

Same trick also works for IPv6.

[adamierymenko]

See also #323 -- we should be able to have networks advertise default routes _or_ routes to more specific subnets.

[adamierymenko]

Done in dev -- ZeroTier can now manage routes. This will require it to be enabled on a per-joined-network basis since we don't want casually joined or less trusted networks to modify the routing table outside setting of their own IPs.

[sbilly]

https://www.zerotier.com/community/topic/5/bridging-ethernet-to-zerotier-virtual-networks-on-linux/

[heri16]

When will this feature be released to Android and iOS?

[Ramblurr]

The link posted by @sbilly is dead (redirects to support index page), anyone have an up to date link?

[leleobhz]

Some people told here about the /1 routes but this do not fix the zerotier output. Openvpn and similar software add a route to /32 passing by the old router, but zerotier does not uses just one address, making this useless.

I think the best way to do, at least in windows is zerotier creates another route table and propagate the default route to it. After this, a ip rule to make zerotier pid check only the new route table and after this, publish routes to main. This allow zerotier to works well and allow a default gw over zerotier. This is useful even when not using default gw, because this allows a bigger control over what passes over zt or not, maybe even allowing bgp over it.

[leleobhz]

Also, if this is not possible yet, can this bug be reopened? I think it's not clear about the default gateway, just clear about random routes, what does not include default gw because zt need the route for external network

[louchenyao]

ZeroTier certainly knows which coordinate servers it is communicating to. So ZeroTier can simply add these servers IP to route table to bypass its own traffic to the original gateway.

ZeroTier do this thing is easier to user. Because the remote IP is not fixed ZeroTier's mesh topology.

[leleobhz]

@Chenyao2333 Ok, how do ZeroTier handle dhcp routes that can change? How zerotier handles vertical handover with IP addressing loss while ZT is running?

This is the main questioning in this issue.

[leleobhz]

Example: ZT inserts a route to the resolved peer with the network gateway, but user changes from wifi to 4g network, so upstream network default gateway changed. How will zerotier handle it?

[louchenyao]

@leleobhz It's the problems the other VPN also faced. The only different thing is the other VPN may only need to connect a few fixed IPs, but ZT connects to a bunch of float IPs.

ZT can periodically check route table to find and what's the default gateway.

Or do you consider just let ZT exposes some API to allow users to write some scripts to handle these dirty works?