Tailscale: Tailscale 0.100 does not work on arm 32-bit (Synology) anymore

[palesz]

Describe the bug
Daemon starts, tailscale admin console shows it's up and running, but cannot access services (SMB, etc) anymore through tailscale on my Synology NAS.

To Reproduce
sudo ./tailscaled --state=tailscaled.state

• I can see in the log that Tailscale that it accepts

Expected behavior
Pre-0.100 versions worked without an issue.

Screenshots
Nope, but see logs.

Version information:

• Device: Synology NAS D215j
• OS: Synology
• OS version: DSM 6.2.3-25426
• Tailscale version: v0.100.0-153-gc6c838785

Additional context

palesz@palesz-synology:~/syno-tailscale$ sudo ./tailscaled --state=tailscaled.state
logtail started
Program starting: v0.100.0-153-gc6c838785, Go 1.14.4-ts56db765: []string{"./tailscaled", "--state=tailscaled.state"}
LogID: 81a0febb6cd9b39a9a5d2702deb9f092768d4054296736ef325210b87e9b15f9
2.5M/13.1M Starting userspace wireguard engine with tun device "tailscale0"
2.6M/13.9M CreateTUN ok.
2.9M/14.5M Routine: event worker - started
2.9M/14.5M Interface set up
3.3M/14.6M UDP bind has been updated
logtail: dialed "log.tailscale.io:443" in 87ms
3.4M/14.8M router: failed to delete legacy rule, continuing anyway: checking for [-m comment --comment tailscale -i tailscale0 -j ACCEPT] in filter/FORWARD: running [/sbin/iptables -t filter -C FORWARD -m comment --comment tailscale -i tailscale0 -j ACCEPT --wait]: exit status 2: iptables v1.6.0: Couldn't load match
`comment':No such file or directory                                                                                                                            
Try `iptables -h' or 'iptables --help' for more information.
3.7M/15.1M router: failed to delete legacy rule, continuing anyway: checking for [-m comment --comment tailscale -o eth0 -j MASQUERADE] in nat/POSTROUTING: running [/sbin/iptables -t nat -C POSTROUTING -m comment --comment tailscale -o eth0 -j MASQUERADE --wait]: exit status 2: iptables v1.6.0: Couldn't load match `comment':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
4.1M/15.2M external route MTU: 1280 (<nil>)
...
5.6M/16.3M SetPrefs: Prefs{ra=false mesh=true dns=true want=true notepad=false derp=true shields=false routes=[] snat=true nf=on Persist{m=[aoYHd], o=[AAAAA],
n=[zpP9u] u="[email protected]"}}
5.6M/16.3M netmap packet filter: [*=>*:*]
...
here comes my ping, it got accepted, but no response
...
5.9M/16.3M Accept: ICMP{100.66.16.79:0 > 100.88.155.128:0} 84 icmp ok
6.1M/16.3M Accept: ICMP{100.66.16.79:0 > 100.88.155.128:0} 84 icmp ok
5.5M/16.3M magicsock: starting endpoint update (periodic)
5.6M/16.3M netcheck: probePortMapServices: me 10.8.0.1 -> gw 10.8.0.2
5.6M/16.3M netcheck: udp=true v6=false mapvarydest=false hair=false portmap= v4a=...:34552 derp=1 derpdist=1v4:28ms,2v4:61ms,4v4:100ms

[palesz]

Previous tested and working version:

palesz@palesz-synology:~/syno-tailscale$ logtail started
Program starting: v0.98-0, Go 1.14.2-tsad6f208: []string{"./tailscaled", "--state=tailscaled.state"}
LogID: 81a0febb6cd9b39a9a5d2702deb9f092768d4054296736ef325210b87e9b15f9
7.4M/13.8M Starting userspace wireguard engine with tun device "tailscale0"
16.0M/13.9M CreateTUN ok.
12.4M/13.9M Routine: event worker - started
12.4M/13.9M Interface set up
magicsock: starting endpoint update (initial)
12.9M/14.1M UDP bind has been updated
12.9M/14.3M external route MTU: 1280 (<nil>)
logtail: dialed "log.tailscale.io:443" in 148ms
13.0M/14.5M iptables forward failed: running [/sbin/iptables -t filter -C FORWARD -m comment --comment tailscale -i tailscale0 -j ACCEPT --wait]: exit status 2: iptables v1.6.0: Couldn't load match `comment':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
13.3M/15.2M iptables nat failed: running [/sbin/iptables -t nat -C POSTROUTING -m comment --comment tailscale -o eth0 -j MASQUERADE --wait]: exit status 2: iptables v1.6.0: Couldn't load match `comment':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

[bradfitz]

I have a 32-bit ARM Synology I can try to reproduce this with later, but I think @danderson knows what the issue is here.

How are you running it? In a container or just ssh'ing to the synology itself?

[palesz]

Not in a container. I am running through Task Scheduler (to start after reboot automatically), but as you can see, I get the same results by running manually using sudo via ssh.

[bradfitz]

Thanks.

@danderson, @dshynkev, I guess we can just probe whether the "comment" kernel module exists and just not use it if it's not there. And then finding our own rules back is a little trickier, but not terrible: if it mentions "tailscale0" it's a pretty obvious sign it's ours?

[palesz]

Actually after reverting back to 0.98 I am not sure it's the "comment". The same warning is present in the 0.98 logs too (that version works):

palesz@palesz-synology:~/syno-tailscale$ logtail started
Program starting: v0.98-0, Go 1.14.2-tsad6f208: []string{"./tailscaled", "--state=tailscaled.state"}
LogID: 81a0febb6cd9b39a9a5d2702deb9f092768d4054296736ef325210b87e9b15f9
7.4M/13.8M Starting userspace wireguard engine with tun device "tailscale0"
16.0M/13.9M CreateTUN ok.
12.4M/13.9M Routine: event worker - started
12.4M/13.9M Interface set up
magicsock: starting endpoint update (initial)
12.9M/14.1M UDP bind has been updated
12.9M/14.3M external route MTU: 1280 (<nil>)
logtail: dialed "log.tailscale.io:443" in 148ms
13.0M/14.5M iptables forward failed: running [/sbin/iptables -t filter -C FORWARD -m comment --comment tailscale -i tailscale0 -j ACCEPT --wait]: exit status 2: iptables v1.6.0: Couldn't load match `comment':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
13.3M/15.2M iptables nat failed: running [/sbin/iptables -t nat -C POSTROUTING -m comment --comment tailscale -o eth0 -j MASQUERADE --wait]: exit status 2: iptables v1.6.0: Couldn't load match `comment':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

[bradfitz]

It's the same issue. It's just that we stopped ignoring errors from the routing layer.

[danderson]

That's not it. The --comment error is always optional, it's our best-effort attempt to remove the firewall rules from older versions of Tailscale. As the logs say: 3.4M/14.8M router: failed to delete legacy rule, continuing anyway

I don't think the router is the problem here - at least not in isolation. There's something else going on.

Are there other iptables errors in the log? As I recall, synology's kernel may not ship with the "mark" netfilter module either, which would definitely break router setup - but in that case, we'd see those errors in the log as well.

If that's the case, you can try getting things to work with tailscale up --netfilter-mode=off. This will prevent use of overlapping subnet routes, and disable some anti-spoofing protection, but if it gets the network engine to come up, at least we'll have confirmed that this is the issue.

[bradfitz]

That's not it. The --comment error is always optional

We should clean up the log spam a bit at least.

Are there other iptables errors in the log?

There's also:

2020-07-22 09:39:22.447248457 -0400 -0400: 6.0M/15.2M authReconfig: ra=false dns=true 0x01: adding [-i tailscale0 -j MARK --set-mark 0x10000] in filter/ts-forward: running [/sbin/iptables -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x10000 --wait]: exit status 1: iptables: No chain/target/match by that name.

[danderson]

Okay, so that's indeed the problem that the Synology kernel doesn't ship with many useful netfilter modules, so we can't implement our security and routing policy correctly there. --netfilter-mode=off will work around that, at the expense of subnet routing (I think routing loop avoidance will actually work, as long as disabling the mark module doesn't prevent sockets from setting a mark, and policy routing from matching on marks).

[apenwarr]

Even if fwmark stuff is missing, we fall back to SO_BINDTODEVICE which is
generally fine on devices (presumably including Synology) that aren't
routers.

ᐧ

On Wed, Jul 22, 2020 at 12:25 PM Dave Anderson [e-mail]
wrote:

Okay, so that's indeed the problem that the Synology kernel doesn't ship
with many useful netfilter modules, so we can't implement our security and
routing policy correctly there. --netfilter-mode=off will work around
that, at the expense of subnet routing (I think routing loop avoidance will
actually work, as long as disabling the mark module doesn't prevent sockets
from setting a mark, and policy routing from matching on marks).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tailscale/tailscale/issues/582#issuecomment-662552439,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAFA4G7EPUHS476YC55YBDR44HIXANCNFSM4PEX37QQ
.

--
Avery Pennarun // CEO @ Tailscale

[bradfitz]

Update: https://github.com/tailscale/tailscale/issues/707 will let us do NAT-style subnet routers on Synology too, even without netfilter.

/cc @nirev

[bradfitz]

This has since been fixed. #451 is still open for subnet routing.