Zerotierone: Reduce data use for VL2 status packets: credentials, multicast subscribe, etc.

[heydonovan]

Came across a similar issue in my research: https://github.com/zerotier/ZeroTierOne/issues/694

We deploy ZeroTier (1.2.12) across a fleet of Windows machines, which have limited data (4G/LTE). After receiving numerous overage fees, I was tasked with investigating why the machines were using so much data when our application fetches very little. We installed various monitoring and network analyzing programs to try and understand what was going on (I'm only familiar with Linux, not so much Windows, so this was rough). Everything pointed to ZeroTier, but that didn't make any sense to me. It's a VPN, so why would traffic be flowing through it unless configured to do so? We don't do anything special during deployment. We install the default Windows MSI package, login, and restart the machine. That's it.

Can you tell from your side how much network bandwidth goes through? Is it just pinging servers? Opening Task Manager shows ZeroTier using 2MB/s connecting to various Vultr servers. Should I be using a different product for our use case where data is scarce? Or did I miss a configuration step somewhere?

[adamierymenko]

We may be able to reproduce this, and are investigating. It seems to happen on one of our engineers' Windows machines, but no others.

[sjansen1]

Maybe Windows 10 Update? i got several machines that does peer to peer Windows Update over Zerotier because it thinks that all the Systems are in the same (virtual) Ethernet.

[laduke]

Wow. Could this be it, @heydonovan ?

[janjaapbos]

If that is the case, perhaps you are able to create zt network rules that deny the windows update traffic.

[laduke]

Some info on turning it off in windows:
https://www.pcworld.com/article/2955491/windows/how-to-stop-windows-10-from-using-your-pcs-bandwidth-to-update-strangers-systems.html

It'd be great to block it with the rules. It looks like port 7680 and possibly 3544

https://docs.microsoft.com/en-us/windows/deployment/update/waas-delivery-optimization#clients-arent-able-to-connect-to-peers-offered-by-the-cloud-service

https://social.technet.microsoft.com/Forums/ie/en-US/eeeec6f7-f180-492b-aaed-9d0a49a7f067/firewall-and-windows-10-delivery-optimization?forum=win10itpronetworking

[adamierymenko]

Other than Windows P2P updates, I think this issue is not Windows-specific. It may be manifesting more on Windows because Windows subscribes to more multicast groups.

I have some ideas about reducing this overhead. After giving it some thought I think a simple rework of scheduling for these packets could cut this traffic a lot. Will implement and test soon.

[ts-sz]

Hello,
I have the same high trafic in Linux machins.
Anybody have any idea how we can reduce this trafic?
It's a lot of trafic for only stdalone machine without any application. Only ntp, etc..

For information I use a archlinux os.

Thanks

[ariselseng]

Same here on raspberry pies.

[WimOnFire]

I'm actually experiencing the exact same problem on 2 raspberry pi's (running latest raspbian stretch). Since these 2 pi's are running behing a 4G/LTE modem with a very limited data plan (and ridiculous overdraft prices, but that's besides the question), I configured the firewall on these pi's to block all ingoing AND outgoing traffic. Leaving it like that leaves the bandwith usage to 0 as expected, but of course zerotier doesn't work. Only opening port 9993 both incoming and outgoing then generates around 2 MB/hour of data (without doing anything on the zerotier network, this is just the 'keep alive'). On a monthly basis for 2 pi's this is about 1.5 GB of "keep alive"... Really looking forward to an update on this, as zerotier is really fantastic technology!

[ariselseng]

@WimOnFire This amount is the exact data usage as I have.

[jasonfharris]

I have to say the very same issue is biting us. Right now we have a dedicated server where each Pi / IOT device (connecting over 3G) sshs to and sets up a reverse tunnel (The pi's use /usr/sbin/nologin as their login shell when setting up the reverse tunnel). Then we can connect to the Pi's via this central tunnel. But the whole process is kind of awkward.

ZeroTier (which we use in fully connected servers) would solve a lot of our problems if the traffic was minimal. With an experiment we found it was taking around 100K an hour, which is still far too much. It would be far better if it was using something like 100K per week...

Are there plans on the horizon to support this? Are there experimental branches we could build to support this?

Thanks in advance.

[rjsc]

I'm getting the same with ZeroTier in OpenWrt routers.
1.5GB per month just doing nothing. Those routers are connected 3G connections with 3GB data cap.
Half of the allowed traffic is being wasted.
Please fix this.
The only reason i'm using zerotier in the routers is because the ISP is preforming "Carrier Grade NAT" / "Large Scale NAT": they attribute private range IPs to the customers (10.X.X.X) and perform NAT in their core network. The customer has no chance os configuring port forwarding in the ISP NAT box, so there is no way to access services from the public internet.

[adamierymenko]

We did some of this... more will require more comprehensive changes, but it's definitely on the radar. 1.4.0 should somewhat improve.

[fragmuffin]

With the release of v1.4.0, the data rate is improved, but still significant.

Test Method

sudo apt install iftop
# observe port 9993 traffic for 1hr
sudo iftop -i wlan0 -f "dst port 9993" -t -s 3600 | tail

Observations
v1.2.12 : total sent/received = 1.33MB
v1.4.0 : total sent/received = 852kB

v1.2.12's 1.33MB an hour equates to ~974MB per month of data... not quite the 1.5GB measured by @WimOnFire, but close enough for comparison.
v1.4.0, in this case, would consume ~623MB per month; 64% of the data consumed by v1.2.12.