Zerotierone: Replace calls to /sbin/ip on Linux with direct programmatic configuration of network devices (was: "unable to add ip address" with unprivileged user on Manjaro)
_(Additions to the original are in italics)_
Summary
_When there is an unprivileged user "zerotier-one" present on a Manjaro system,_ ZeroTier fails to add an IP address to a zt0 device with the message "ERROR: unable to add ip address 10.0.0.1/24" (where 10.0.0.1 is the IPv4 address assigned via the web dashboard). It is repeated for the assigned IPv6 address.
Manually adding the ZT IP and route via ip succeeds.
Details
I run ZT on a variety of devices (Linux, Windows, Mac, Android) and this issue occurs on only one of my Linux machines. Configuration is pretty much identical to another machine which works perfectly. I can't see any reason for it, and I can't seem to find or generate any more logging output.
Manually running
sudo ip addr add 10.0.0.1/24 dev zt0
sudo ip route add 0.0.0.0/1 via 10.0.0.254 dev zt0
succeeds; the ZT network becomes accessible and traffic is routed over the global/managed route. Restarting the ZT service after this "loses" the IP and route.
Log output
The output from journalctl -u zerotier-one shows:
Apr 16 09:33:28 box systemd[1] Started ZeroTier One.
Apr 16 09:33:28 box zerotier-one[5691]: ERROR: unable to add ip address 10.0.0.1/24
Apr 16 09:33:28 box zerotier-one[5691]: ERROR: unable to add ip address fde5:::1/88
Output from zerotier-one directly is:
$ sudo zerotier-one
ERROR: unable to add ip address 10.0.0.1/24
ERROR: unable to add ip address fde5:::1/88
_This also occurs when specifying not to drop privileges:_
$ sudo zerotier-one -U
ERROR: unable to add ip address 10.0.0.1/24
ERROR: unable to add ip address fde5:::1/88
System details
OS: Manjaro Linux 64-bit
Tested kernels: 4.14.34, 4.15.17, 4.16.2
ZeroTier version: 1.24 (from https://www.archlinux.org/packages/community/x86_64/zerotier-one/)
systemd version: 238.76
(The same setup works perfectly on another machine)
jonathonf
All 27 comments
Are you using selinux?
adamierymenko
on 18 Apr 2018
No selinux or apparmor or any kernel hardening. I'm quite happy to admit it's _something_ I've done but I haven't yet found a way to work out what...
(I've also removed and reinstalled, as well as updating to 1.2.6, same end result ("unable to add ip address")).
Given that it worked previously something must have changed on this particular system that didn't change on my other systems. I'll have another dig through package update logs...
jonathonf
on 18 Apr 2018
It could be some issue with dropping privileges. There is a -U flag to not do this, or you can remove the zerotier-one user and it won't do it.
adamierymenko
on 18 Apr 2018
It works on CentOS, Fedora, Debian, etc. but maybe something is different about your distribution.
adamierymenko
on 18 Apr 2018
Dropping privileges is the reason. The Arch package doesn't add a zerotier-one user, and after adding one manually [1] the error manifests. Interestingly, however, running with sudo zerotier-one -U _also_ fails.
I'll do some more digging and ask around to see whether this is an isolated case. The fact it used to work but stopped working is weird (to me, but then lots of things are).
[1] Based on RH .spec file, useradd -r -d /var/lib/zerotier-one -s /sbin/nologin zerotier-one
jonathonf
on 18 Apr 2018
We did not change that code at all, so I'm curious to hear what you discover.
adamierymenko
on 20 Apr 2018
We made a Manjaro VM and can duplicate this issue, but we're totally stuck. I tried giving all the requisite capabilities directly to /sbin/ip and that still doesn't work. Disabling the redirect-output-to-null code in LinuxEthernetTap I get:
RTNETLINK answers: Operation not permitted
... when we attempt to invoke /sbin/ip to add IP addresses.
This works on pretty much every other distribution AFIAK.
adamierymenko
on 24 Apr 2018
My guess is that Manjaro is doing something really weird with permissions or other security settings. No idea what it could be.
adamierymenko
on 24 Apr 2018
I set up a plain Arch VM and have replicated the issue there, so it's likely something with Arch-like distros as opposed to specifically Manjaro.
An Arch user said they have had no issues so I'm not sure what the difference is (I suspect they haven't added the non-privileged user).
jonathonf
on 24 Apr 2018
Does Arch maybe set permissions differently? I know the nosuid bit in filesystems can do this but root does not seem to be mounted nosuid.
adamierymenko
on 24 Apr 2018
Maybe the solution is not to create the special zerotier-one user on Arch and not drop privileges. We're definitely going to leave privilege drop in since the vast majority of Linux users want it, but if it's broken in Arch there's not much we can do. We can add a hack if it's something simple.
adamierymenko
on 24 Apr 2018
I had an system running an older kernel, and it still worked with the zerotier-one user. Then it broke when I upgraded.
Somewhere between 4.14.23-1-lts and 4.14.35-1-lts something happened. Made a thread on the arch bbs.
https://bbs.archlinux.org/viewtopic.php?pid=1781638#p1781638
laduke
on 25 Apr 2018
Narrowed it down to iproute2-4.15 vs iproute2-4.16. With 16 you get the "unable to add ip address" messages.
laduke
on 25 Apr 2018
Apparently iproute2 v4.16 drops all capabilities and only allows root/sudo access. It looks like this was done to allow setting capabilities for "ip vrf exec" that won't allow access to the rest of the multicall binary. Probably an example why multicall binaries can be limiting, since that broke running ip with reduced capabilities.
progandy
on 25 Apr 2018
OK, so a little progress. If I:
sudo setcap cap_net_admin,cap_net_raw+eip /usr/bin/zerotier-one
then this:
sudo zerotier-one -U
works as expected. However,
sudo zerotier
is still unable to add ip address.
I'm wondering whether setting capabilities and allowing running as the zerotier-one user (instead of root) might be the way forward, e.g.
$ sudo setcap cap_net_admin,cap_net_raw,cap_net_bind_service+ep /usr/bin/zerotier-one
$ sudo -u zerotier-one /usr/bin/zerotier-one -U
ERROR: unable to add ip address...
as is already the case for ping:
$ getcap /usr/bin/ping
/usr/bin/ping = cap_net_raw+ep
Or is this a regression in iproute2?
jonathonf
on 25 Apr 2018
Okay, I'm re-titling this issue. It will not be blocking for 1.2.8 (a minor tag after 1.2.6 before we do binaries) but will get addressed eventually.
adamierymenko
on 25 Apr 2018
That sounds fine to me. If it's only hitting Arch (etc.) because of iproute2=4.16 it may still be a little time before affecting other distros (Fedora Rawhide is on 4.15, Sid is on 4.15 (Experimental is on 4.16), Tumbleweed is on 4.16).
At least now we know the root cause (and it's not just me being silly), and in the meantime we have a workaround. :slightly_smiling_face:
jonathonf
on 25 Apr 2018
Or is this a regression in iproute2?
@jonathonf You'd have to ask the maintainer Stephen Hemminger if this was intended. This use case might be too rare and deemed unimportant or even bad practice (with sudo you have finer control over allowed ip commands)
progandy
on 25 Apr 2018
The edge branch now uses RTNETLINK (the API iproute2 uses) instead of calling out to execute iproute2. It's a bit noisy in the logging department and sometimes says there's an error when everything worked fine, but it appears to be working thus far. Any testing & bug reports are much appreciated!
glimberg
on 5 Jun 2018
I think this might be affecting nixos users trying to join networks with managed routes too. One thing to note:
⇒ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 600 0 0 wlp2s0
10.147.17.0 0.0.0.0 255.255.255.0 U 0 0 0 zt1
172.25.0.0 0.0.0.0 255.255.0.0 U 0 0 0 zt0
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp2s0
The routes to the networks themselves are added, but the route that I need to route to the managed route subnet isnt: 10.0.0.0 172.25.X.X 255.255.0.0 UG 0 0 0 zt0
I had to run this command manually to get it to work: route add -net 10.0.0.0/16 gw 172.25.X.X dev zt0
commandodev
on 14 Aug 2018
Possible same issue? https://github.com/zerotier/ZeroTierOne/issues/726
commandodev
on 14 Aug 2018
@jonathonf what's the work around you suggest? I can try adding that to the the nixos module in the linked issue to see if it works? Did you mean
sudo setcap cap_net_admin,cap_net_raw,cap_net_bind_service+ep /usr/bin/zerotier-one
I've fixed this on the nixos end here NixOS/nixpkgs#45022
commandodev
on 21 Aug 2018
Same happened to us with recent ContainerLinux update stable (2023.5.0) (kernel 4.19.25-coreos).
liquid-sky
on 13 Mar 2019
Unfortunately, the workarounds provided here do not work on CoreOS, since Zerotier has to run in Docker container 😔
liquid-sky
on 13 Mar 2019
Turned out to be related _not_ to recent CoreOS update, but to update of Ubuntu version (to Xenial) in our Zerotier Docker image. After downgrading it to Trusty it had no issues adding the IP and the route. Meanwhile, we'll pick an alternative OS for our Zerotier image.
liquid-sky
on 13 Mar 2019
Done for a while.
adamierymenko
on 11 Jun 2019
Related issues
lburtts
·
4Comments
unquietwiki
·
3Comments
kblackcn
·
3Comments
nolanl
·
4Comments
tim77
·
4Comments
Most helpful comment
Narrowed it down to iproute2-4.15 vs iproute2-4.16. With 16 you get the "unable to add ip address" messages.