Zerotierone: Question: Multipath: Enabling on box with multiple WAN interfaces

[jdrews]

I am trying out multipath on 1.4.0. Based on the manual it says:

will automatically detect available physical interfaces and begin allocating across all paths.

I'm a little stuck on how to get ZeroTier to recognize the multiple WANs available to it on one linux machine. Any pointers?

How does ZeroTier determine that a physical interface is "available" for use in multipath?

Use Case

Site A
3 WAN connections, all of which are plugged into a Debian 10 box via separate NICs.
The Debian box has ZT 1.4.0 with local.conf "multipathMode": 2.

Site B
1 WAN connection and a Debian 10 box.
The Debian box has ZT 1.4.0 with local.conf "multipathMode": 2.

Details

1. Site A and Site B are in the same ZeroTier network and can ping each other on ZT network IPs.
2. Also enabled FRRouting at both sites and running OSPF network across them.

Goal
Able to load balance traffic across all 3 WANs at Site A to aggregate and improve bandwith at Site A.
Site B acts as the 0.0.0.0/0 default router for all traffic from Site A.

[jdrews]

Playing around with this some more. I thought maybe ZeroTier needed to see multiple routes in ip route to a peer, each going out a different WAN interface. So I constructed multiple routes via my 3 WAN interfaces, all with different metrics.

That didn't seem to work as I still see "multipath": null, and "multipathMode": 2, in zerotier-cli -j status on both sides.

[jdrews]

I drew it out as pictures are worth more than words.

In this case Site A and Site B are Debian 10 boxes.
Site A has multiple WAN connections, albeit through routers that hand out private subnet IPs.
Site B has a public IP directly attached to it.

[jdrews]

I also confirmed via zerotier-cli -j peers that it's only seeing one path on either side.

[jdrews]

A bit of feedback here: It's really hard to get multipath working correctly over multiple unique WAN connections. I've got it to detect two of the WAN connections, but the last one is very elusive. I've had to fudge with the routes a ton.

It would be useful if we could tell ZeroTier what interfaces+gateways it should use for the paths.

[2600box]

I'm in the same boat. Can you let me know what you did with your routes to get the two recognised?

[jdrews]

In my case, I configured Site B to have 3 WAN IPs. I then configured Site A's routing table to push traffic through one of the three WANs for one of the three Site B IPs.

[dhbaird]

This affects me too. I have devices with only 1 internet connection, that I would like to connect to both of 2 connections available at another site, and haven't figured out how to get this case working. (I did try @jdrews solution mentioned in the above comment, and it indeed works for me with 2+2 connections and I manually configure the routes on each side; But what about a 1+2 configuration and without manual route configuration?)

[justinh-rahb]

Any solid instructions for this? Plenty of mention of this awesome new feature but absolutely nothing on how to actually make it work...

[jbfuzier]

Same here :(

[jocelynke]

Any news ?

[o-s-e]

Hi, is there any update? I tried to enable it, but it doesn't work:
zerotier-cli -j status { "address": "d6214ff7fc", "clock": 1602235526986, "config": { "physical": null, "settings": { "allowTcpFallbackRelay": false, "bind": null, "controllerDbPath": null, "multipathMode": 2, "portMappingEnabled": true, "primaryPort": 9993, "rabbitmq": null, "softwareUpdate": "disable", "softwareUpdateChannel": "release" } }, "multipath": null,

[laduke]

New release coming shortly.
https://github.com/zerotier/ZeroTierOne/commit/fe1ce885a04ae68894b8a8fc1892db212addbf6e

[joseph-henry]

Hello everyone,

There's been a beta release of our new multipath functionality (tagged as 1.5.0), in addition to @laduke's comment above you can find the documentation here: https://zerotier.atlassian.net/wiki/spaces/SD/pages/568459265/Multipath

Let us know if you have any questions, feature requests or discover any bugs. Thanks.

[wyattanderson]

I have a functioning DIRECT connection between two ZeroTier peers, each running v1.5.0. Is there anything more I need to do to enable multipath beyond the following local.conf?

{
    "settings": {
        "allowTcpFallbackRelay": false,
        "portMappingEnabled": false,
        "defaultBondingPolicy": "balance-aware"
    }
}

This is the configuration on both peers. Peer A has three interfaces available, each on separate subnets. Peer B has only one interface. This the output from zerotier-cli listbonds on either peer:

    <peer>                        <bondtype>    <status>    <links>
      NONE                              NONE        NONE       NONE