Zerotierone: [Feature Request] my.zerotier.com hability to set route metric to static routes

[leleobhz]

Hello folks!

Today in my.zerotier.com, we are able to setup static routes in the managed routes screen. But this place does not allow setup route metric. This way, as example, if I connect to ZT network that have a route to the network i'm joined, the route will not work because ZT usually haves a lower metric than local interface.

Can you please add this feature?

Thanks!

[glimberg]

I'm not quite sure what you're asking here.

Do you have a physical network and a ZeroTier network where the IP address ranges overlap? Because if that's the case, then that's a good way to have problems. If this is the case, your best bet is to make sure that your IP address ranges don't overlap.

Additionally, not all platforms allow setting route metrics (iOS and Android for instance).

[novirium]

I _think_ this is the same as the issue I'm having it my situation: I've got a ZT network being used to connect several nodes including roaming laptops together, and have a gateway node set up in the office with a managed route to direct traffic to the office subnet from the ZT one. This works great until a laptop (running Linux here) connects to the office network directly, and the managed route added by ZeroTier has higher priority than the direct LAN one. In this situation, internet access is via the LAN subnet - which the zerotier route is overriding, resulting in a complete loss of internet connectivity (except to other hosts on the ZT subnet, which still work).

Running ip route shows that the managed route added by ZeroTier has no metric, and so is defaulting to the highest priority. Manually changing this to a higher metric solves the problem.

I'm not sure what the best approach for this situation is, but interestingly it doesn't seem to be a problem on Android.

[glimberg]

@novirium Well that's a weird one. ZeroTier definitely should be setting a metric such that the ZeroTier route is a lower priority than the physical network. What Linux distribution & version of the distro are you using?

And no, it wouldn't be a problem on Android. Android & iOS are entirely different beasts.

[laduke]

Try making the ZeroTier managed route less specific than the LAN's.

[novirium]

I'm not seeing any metric on the added route on both Arch Linux and Debian Stretch (it's probably worth a separate issue, but on the Stretch host the local subnet route is via a bridge, and ZeroTier is removing that route and replacing it entirely). Both are running ZeroTier One 1.2.8 .

For clarity, the local subnet is 10.0.3.0/24, and the zerotier network is 10.5.5.0/24. The gateway node (10.5.5.51) is connected to both, and I have the managed route 10.0.3.0/24 : 10.5.5.51 added in Zerotier Central.

On the Arch host, ip route before connecting to the ZT network:

default via 10.0.3.1 dev enp0s31f6 proto dhcp src 10.0.3.171 metric 202 
10.0.3.0/24 dev enp0s31f6 proto dhcp scope link src 10.0.3.171 metric 202 

And after connecting to the ZT network:

default via 10.0.3.1 dev enp0s31f6 proto dhcp src 10.0.3.171 metric 202 
10.0.3.0/24 via 10.5.5.51 dev zt1 
10.0.3.0/24 dev enp0s31f6 proto dhcp scope link src 10.0.3.171 metric 202 
10.5.5.0/24 dev zt1 scope link

On the Debian Stretch host (a stock ProxMox install), before connecting to the ZT network:

default via 10.0.3.1 dev vmbr0 onlink
10.0.3.0/24 dev vmbr0 proto kernel scope link src 10.0.3.50

And after connecting to the ZT network:

default via 10.0.3.1 dev vmbr0 onlink
10.0.3.0/24 via 10.5.5.51 dev ztuku45mat
10.5.5.0/24 dev ztuku45mat scope link

[laduke]

Hi @novirium Is a zt adapter part of the vmbr0 bridge? In that case, sometimes you want to set allowManaged to false for that zerotier network.
zerotier-cli set $NETWORKID allowManaged=0

Does changing the managed route to 10.0.3.0/23 : 10.5.5.51 work around the metric issue?

[novirium]

Sorry @laduke , I'd missed your meaning before. Yes, specifying a less specific route does work and is a fairly elegant workaround, thanks.

I suspect the bridge thing is a separate issue, as the ZT adapter isn't part of the bridge, and with the different managed route now (10.0.3.0/23), the route isn't being added at all (no change at all in the routes when joining the ZT network).

That aside, _should_ zerotier one be adding a metric to its managed routes?

[leleobhz]

Hello folks!

About prefix engineering, it isn't a option: Does not make any sense a prefix engineering with the following routes being installed:

192.168.0.0/24 via 10.0.0.1
192.168.1.0/24 via 10.1.0.1

As example of 2 networks from same network, If I have different routes for each one from a ZT network, run a /23 route may lead a network to be wrongly routed.

About metric: Let's take DHCP example from Linux:

 root  ~  ip r l | grep dhcp
default via 192.168.43.1 dev wlp1s0 proto dhcp metric 600 

DHCP Clients can set routing metric to prefer a link besides other (Preferring Wired Ethernet before 802.11 and before LTE - that can be always on). So what may stop metrics to be set? I see lack of API from SO a reason, but I don't believe all modern SO cannot set it.

So, routing engineering for my case does not help because I need fallback routes that are equal - and need to be - because the near prefix belongs to other network.

[SimmyD]

Hi All,

did a solution come from this? I have the exact thing with my raspberry pi.

Home network - 192.168.87.0/24
ZT network - 192.168.28.0/24

have a route in ZT for 192.168.87.0/24 pointing to 192.168.28.250 (linux server acting as ip router on 192.168.87.0/24)

Once the raspberry pi is connected to the home network but also to ZT i can only connect to it via the ZT address not the Home network address from devices on the home network.

[mhandugan]

Same issue here. The zt metric is higher than wired Ethernet, but lower than Wi-fi. So, with a route to my home LAN on zt, I cannot access Wi-Fi interfaces on-LAN.
Metric for Linux eth0: 202
Metric for zt: 204
Metric for wlan0: 303

Syn/acks to a local IP go bouncing through zt and fail to make it through the statefull firewall (not sure why they don’t make it eventually).

I think I just need to go unmanaged for all hosts on the LAN, and only use zt ip/routes for iOS/Android.

[bjeanes]

Also same issue. It does not result in a loss of connectivity for me, but it does make all traffic from ZT network members which are also on the physical network route their traffic to the ZT gateway, which also routes to itself but eventually forwards on the packet. So for me, this just results in a higher perceived local network latency.

$ ip addr show wlp2s0
2: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:b6:d0:98:63:73 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.143/24 brd 10.10.10.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 53214sec preferred_lft 53214sec
    inet6 fe80::94a5:e05f:907:bb76/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
$ ip route show
default via 10.10.10.254 dev wlp2s0 proto dhcp metric 600 
10.10.10.0/24 via 10.144.119.0 dev ztks5srcp7 
10.10.10.0/24 dev wlp2s0 proto kernel scope link src 10.10.10.143 metric 600 
10.144.0.0/16 dev ztks5srcp7 proto kernel scope link src 10.144.17.130 linkdown 
$ tracepath -b 10.10.10.10
 1?: [LOCALHOST]                      pmtu 1500
 1:  tumtum (10.144.119.0)                                 4.509ms 
 1:  tumtum (10.144.119.0)                                 3.242ms 
 2:  _gateway (10.10.10.254)                               4.975ms 
 3:  10-10-10-10.local (10.10.10.10)                       4.082ms reached
     Resume: pmtu 1500 hops 3 back 2 

As you can see, the 10.10.10.0/24 via 10.144.119.0 dev ztks5srcp7 route does not have a metric set and seems to sit at a higher precedence than the 10.10.10.0/24 dev wlp2s0 route.

[novirium]

Got curious about this again this morning, and correct me if I'm wrong (I'm by no means familiar with Linux kernel programming), but it looks like no metric is being passed _at all_ on Linux when a route is being added. I've attempted to follow the logic through below:

A generic, default metric of 5000 is set here, and passed in to EthernetTap, which hands it out to whatever OS implementation is used. As far as I can tell, Mac and others like BSD run a command to bring the interface up, which uses the "metric" value passed down from EthernetTap when creating the interface.

On Linux though, the metric needs to be set when routes are added (there's not a simple way to set a default metric on an interface - you can't do it via ifconfig or the RTNetlink system that is used by ZT on Linux for interface creation). The command that's run to add routes on Linux is defined here, and currently doesn't have any support for setting the route metric.

Tentatively, a "metric" string could simply be added on the end of the commands run here, but at the moment the metric value isn't passed down to the ManagedRoute instance, and so some other structure would also need to be added to enable it to be set as a value on a ManagedRoute. I imagine this would have complications with similar per-route support on other OSes, so maybe instead for now the metric value could be directly pulled from the n.tap.metric when the ManagedRoute is created here? That way the metric set in EthernetTap is still the source of truth, and treated as a per-interface value.

It looks like this would require changes touching a few places down through the source, and I'm by no means familiar enough to be able to implement this personally. Being able to set a high metric would be an incredibly useful feature on Linux though, and solve a lot of weird little issues people have had using ZT as a way to provide access to local networks with portable devices.

[TheBestPessimist]

Bumping this as i sufffer the same problem: 2 laptops connected to the same wireless and with zerotier running. When i transfer a 1gb file, the speed is very slow (that of my isp). when i disable zerotier on any of those laptops, the transfer speed goes back to normal lan (wireless) speed.

[ihipop]

same issue。I think a router metric to static routers is a better Solution

[ihipop]

same issue。I think a router metric to static routers is a better Solution

I install ZeroTier One in my UBNT router,which has a 192.168.1.1/24 sub-net on eth1 at home，this device also has a ZeroTier private address 10.10.10.10
I add a static router 192.168.1.1/24 via 10.10.10.10 on my.zerotier.com so that my android device can visit 192.168.1.1/24 sub-net on eth1 at home ,but when I run ZeroTier One in my UBNT router,It will add a router 192.168.1.1/24 sub-net via 10.10.10.10 too,which make the router 192.168.1.1/24 sub-net on eth1 invalid and LAN device can not visit internet via UBNT router

I think a router metric to static routers on my.zerotier.com is a better Solution

[pulento]

Got curious about this again this morning, and correct me if I'm wrong (I'm by no means familiar with Linux kernel programming), but it looks like no metric is being passed _at all_ on Linux when a route is being added. I've attempted to follow the logic through below:

A generic, default metric of 5000 is set here, and passed in to EthernetTap, which hands it out to whatever OS implementation is used. As far as I can tell, Mac and others like BSD run a command to bring the interface up, which uses the "metric" value passed down from EthernetTap when creating the interface.

On Linux though, the metric needs to be set when routes are added (there's not a simple way to set a default metric on an interface - you can't do it via ifconfig or the RTNetlink system that is used by ZT on Linux for interface creation). The command that's run to add routes on Linux is defined here, and currently doesn't have any support for setting the route metric.

Tentatively, a "metric" string could simply be added on the end of the commands run here, but at the moment the metric value isn't passed down to the ManagedRoute instance, and so some other structure would also need to be added to enable it to be set as a value on a ManagedRoute. I imagine this would have complications with similar per-route support on other OSes, so maybe instead for now the metric value could be directly pulled from the n.tap.metric when the ManagedRoute is created here? That way the metric set in EthernetTap is still the source of truth, and treated as a per-interface value.

It looks like this would require changes touching a few places down through the source, and I'm by no means familiar enough to be able to implement this personally. Being able to set a high metric would be an incredibly useful feature on Linux though, and solve a lot of weird little issues people have had using ZT as a way to provide access to local networks with portable devices.

So basically it is not only a management interface so also a "client" feature also.

I think this feature is a must IMHO 😃