Tailscale: wgengine/router: enable subnet routing on Synology (work around missing iptables modules)

[nirev]

Describe the bug
I'm running tailscale in a Synology nas using https://github.com/nirev/synology-tailscale (tailscale static binaries)
Versions 0.97-45 and 0.98-0 work perfectly, however with v0.99.1-0 tailscale0 doesn't get an IP address and no routes are added.

I've tried removing the state file and starting from scratch and get the same behavior.

To Reproduce
Consistently able to reproduce on v0.99.1
Went back and forth with versions.

Expected behavior
Expected to have an IP address assigned, routes added, and be able to ping hello.ipn.dev

Version information:

• Device: Synology DS218+ (amd64)
• OS: Linux
• OS version: Synology DSM 6.2.3 -- Linux 4.4.59+ #25426 SMP PREEMPT Tue May 12 04:54:55 CST 2020 x86_64 GNU/Linux synology_apollolake_218+
• Tailscale version: 0.99.1-0

Additional context
Logs attached

• tailscale-0.97-45.log
• tailscale-0.98-0.log
• tailscale-0.99.1-0.log

[danderson]

0.99 became more strict about requiring network configuration to succeed. Previous versions just kept going despite failures and tolerated partial configurations.

The problem here seems to be that the synology kernel hasn't included the netfilter mark module, which we require. So, network bringup fails.

First, let's verify that there isn't just some permissions problem: if you ssh into your NAS as root and run iptables -A FORWARD -m mark --mark 0x10000 -j ACCEPT, does it also fail?

If the mark module is unavailable, then it won't be possible to use tailscale with the fully managed netfilter rules. However, with 0.99 you can pass --netfilter-mode=off to tailscale up, and manage firewall rules by hand. For uses other than subnet routing, the firewall rules can be pretty static.

[nirev]

Indeed when running iptables -A FORWARD -m mark --mark 0x10000 -j ACCEPT as root, it fails with:
iptables v1.6.0: Couldn't load match 'mark':No such file or directory

I'm really not familiar with iptables, but it seems like it should be loaded:

cat /proc/net/ip_tables_matches
ipvs
mark
conntrack
conntrack
conntrack
addrtype
mac
limit
state
addrtype
geoip
recent
recent
iprange
udplite
udp
tcp
multiport
icmp

[bradfitz]

@nirev, what about:

$ lsmod | grep mark
xt_mark                16384  30
x_tables               36864  17 xt_comment,xt_multiport,ipt_REJECT,xt_nat,iptable_mangle,xt_statistic,ip_tables,nft_compat,iptable_filter,xt_set,xt_mark,xt_tcpudp,iptable_raw,ipt_MASQUERADE,xt_recent,xt_addrtype,xt_conntrack

@danderson, I imagine we can detect whether mark/xt_mark/etc is available (at least after a failure to use it). But then is there a reasonable configuration we could fall back to when it's not available, at least if subnet routing or some other features are not in use?

[apenwarr]

If I recall correctly, if we're not using subnet routing we shouldn't need
the mark feature at all anymore?

ᐧ

On Tue, Jun 9, 2020 at 3:39 PM Brad Fitzpatrick [e-mail]
wrote:

@nirev https://github.com/nirev, what about:

$ lsmod | grep mark
xt_mark 16384 30
x_tables 36864 17 xt_comment,xt_multiport,ipt_REJECT,xt_nat,iptable_mangle,xt_statistic,ip_tables,nft_compat,iptable_filter,xt_set,xt_mark,xt_tcpudp,iptable_raw,ipt_MASQUERADE,xt_recent,xt_addrtype,xt_conntrack

@danderson https://github.com/danderson, I imagine we can detect
whether mark/xt_mark/etc is available (at least after a failure to use
it). But then is there a reasonable configuration we could fall back to
when it's not available, at least if subnet routing or some other features
are not in use?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tailscale/tailscale/issues/451#issuecomment-641528537,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAFA4BXHW22SC7CDZEVKJDRV2FWDANCNFSM4NZUTW7Q
.

--
Avery Pennarun // CEO @ Tailscale

[nirev]

$ lsmod | grep mark
xt_mark                 1317  0
x_tables               16302  21 ip6table_filter,xt_ipvs,xt_iprange,xt_mark,xt_recent,ip_tables,xt_tcpudp,ipt_MASQUERADE,xt_geoip,xt_limit,xt_state,xt_conntrack,xt_LOG,xt_mac,xt_nat,xt_multiport,iptable_filter,xt_REDIRECT,iptable_mangle,ip6_tables,xt_addrtype

[nickstenning]

As @nirev has said, it looks like the kernel module is available. (I had to manually insmod /lib/modules/xt_mark.ko).

I did a bit more digging and it seems the problem is that the iptables binary can't load the relevant shared libraries:

...
6983  13:29:32.578265 ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 <0.000024>
6983  13:29:32.578795 stat64("/lib/iptables/libipt_mark.so", 0x7e88a280) = -1 ENOENT (No such file or directory) <0.000050>
6983  13:29:32.578964 stat64("/lib/iptables/libxt_mark.so", 0x7e88a280) = -1 ENOENT (No such file or directory) <0.000035>
6983  13:29:32.579203 write(2</dev/pts/16>, "iptables v1.6.0: ", 17) = 17 <0.000045>
6983  13:29:32.579562 write(2</dev/pts/16>, "Couldn't load match `mark':No su"..., 53) = 53 <0.000039>
6983  13:29:32.579797 write(2</dev/pts/16>, "\n", 1) = 1 <0.000036>
6983  13:29:32.580010 write(2</dev/pts/16>, "Try `iptables -h' or 'iptables -"..., 61) = 61 <0.000038>

Neither of these exist, although there is a libxt_MARK.so. Symlinking to that doesn't work, so I assume libxt_MARK.so is for the target, rather than the match...

$ ls -l /lib/iptables/ | grep -i mark
-rwxr-xr-x 1 root root  9876 May  9  2019 libxt_MARK.so

Seems like it might be a Synology packaging problem?

[nirev]

I added a feature request for Synology to include netfilter modules in DSM.
The form is here: https://www.synology.com/en-global/form/inquiry/feature
I think it's worth doing this, specially since DSM 7 is supposedly coming later this year.

[nirev]

I was trying to understand what worked, and what didn't, so I wrapped iptables and logged what tailscaled was doing.

wrapper:

#!/bin/bash

/sbin/iptables $@

exit_status=$?
if [ $exit_status -ne 0 ]; then
    echo -ne "[FAIL]\t" >> /tmp/iptables.log
else
    echo -ne "[OK]\t" >> /tmp/iptables.log
fi
echo "$@" >> /tmp/iptables.log
exit 0

when starting (tailscale 1.0.5):

[FAIL]  -t filter -C FORWARD -m comment --comment tailscale -i tailscale0 -j ACCEPT --wait
[FAIL]  -t filter -D FORWARD -m comment --comment tailscale -i tailscale0 -j ACCEPT --wait
[FAIL]  -t nat -C POSTROUTING -m comment --comment tailscale -o eth0 -j MASQUERADE --wait
[FAIL]  -t nat -D POSTROUTING -m comment --comment tailscale -o eth0 -j MASQUERADE --wait
[OK]    -t filter -N ts-input --wait
[OK]    -t filter -N ts-forward --wait
[OK]    -t nat -N ts-postrouting --wait
[OK]    -t filter -A ts-input ! -i tailscale0 -s 100.115.92.0/23 -j RETURN --wait
[OK]    -t filter -A ts-input ! -i tailscale0 -s 100.64.0.0/10 -j DROP --wait
[OK]    -t filter -A ts-forward -i tailscale0 -j MARK --set-mark 0x40000 --wait
[FAIL]  -t filter -A ts-forward -m mark --mark 0x40000 -j ACCEPT --wait
[OK]    -t filter -A ts-forward -o tailscale0 -s 100.64.0.0/10 -j DROP --wait
[OK]    -t filter -A ts-forward -o tailscale0 -j ACCEPT --wait
[OK]    -t filter -I ts-input 1 -i lo -s 100.113.49.15 -j ACCEPT --wait
[FAIL]  -t nat -A ts-postrouting -m mark --mark 0x40000 -j MASQUERADE --wait

when stopping:

[FAIL]  -t filter -N ts-input --wait
[FAIL]  -t filter -N ts-forward --wait
[FAIL]  -t nat -N ts-postrouting --wait
[FAIL]  -t filter -N ts-input --wait
[FAIL]  -t filter -X ts-input --wait
[FAIL]  -t filter -N ts-forward --wait
[FAIL]  -t filter -X ts-forward --wait
[FAIL]  -t nat -N ts-postrouting --wait
[OK]    -t nat -X ts-postrouting --wait

as expected the comment and mark rules don't work

(NOTE: closed by mistake)

[nirev]

@bradfitz I want to update the Synology package to tailscale 1.0.5
and would like to understand what would be the best way to do that.

As I understand it, right now it's already not providing the full feature set (eg subnet routing/acls) but it's just failing silently.

I want to update the package, add a note about this, but don't want to break anything that is already working.
The options I can see are:

1) package an iptables wrapper like the above

• ugly hack
• tricks tailscale, so harder to debug
• anything that can be applied, will be applied. don't know if that's better or worse

2) migrate tailscaled.state to disable netfilter when upgrading
can be done with something like:

jq --arg updated "$(
    jq -r '._daemon' tailscaled.state | base64 -d | \
    jq -r 'del(.UsePacketFilter) | .ShieldsUp = false | .oSNAT = false | .NetfilterMode = 0 | @base64' \
  )" \
  '._daemon = $updated' \
  tailscaled.state

I did this to migrate my own .state file and it worked perfectly.
I also noticed that if I deleted the state and did a new tailscale up from scratch, there are some other keys there, but all empty:

"AdvertiseTags": null,
"Hostname": "",
"OSVersion": "",
"DeviceModel": "",

3) delete the state file entirely :(

• and add a note to run tailscale up again with netfilter-mode=off

what do you think would be the best option, @bradfitz?

(note: I also have on my todo list to try compiling and loading netfilter modules, but I don't know when I'll have the time to look into this. I also want to figure out how to create a Synology UI for the package, which would be great for adding this sort of thing, but, well..)

Thank you again for all the help!

[jucor]

I have also filed the request form on Synology side. Tailscale is so much simpler to use that Synology's current OpenVPN setup... Would love to be able to use it!

[bradfitz]

I've sent https://github.com/tailscale/tailscale/pull/750 to get Synology working again, even if not completely yet. Baby steps. With that change, subnet routing is just disabled.

Future changes can bring it up to parity with stock Linux.

/cc @danderson @nirev

[bradfitz]

I'm now running 1.1.335 on my Synology DS916+. It defaults to --netfilter-mode=off now when running on Synology. I didn't need to do anything besides:

bradfitz@tsdev:~/hack/synology-tailscale$ git di
diff --git a/Makefile b/Makefile
index 7d65c5c..6f32b8c 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,4 @@
-TAILSCALE_VERSION="0.98-0"
+TAILSCALE_VERSION="1.1.335"
 SPK_BUILD="1"

 .PHONY: tailscale-% clean purge
diff --git a/build-package.sh b/build-package.sh
index 07e506c..59362b3 100755
--- a/build-package.sh
+++ b/build-package.sh
@@ -7,7 +7,7 @@ ARCH=$2
 SPK_BUILD=$3

 download_tailscale() {
-  local base_url="https://pkgs.tailscale.com/stable"
+  local base_url="https://pkgs.tailscale.com/unstable"
   local pkg_name="tailscale_${TAILSCALE_VERSION}_${ARCH}.tgz"
   local src_pkg="${base_url}/${pkg_name}"
   local dest_pkg="_tailscale/${pkg_name}"

We'll get subnet routing working later. I'll repurpose this bug to be about that.

/cc @nirev

[nirev]

That's great! I'll give it a try and update the package!

thank you, @bradfitz

Update:
Working perfectly in my DS218+ too.
For reference, this was my state file:

# jq -r '._daemon' tailscaled.state | base64 -d | jq .
{
  "ControlURL": "https://login.tailscale.com",
  "RouteAll": false,
  "AllowSingleHosts": true,
  "CorpDNS": true,
  "WantRunning": true,
  "UsePacketFilter": true,
  "AdvertiseRoutes": null,
  "NotepadURLs": false,
  "DisableDERP": false,
  "Config": {
    "PrivateMachineKey": "privkey:...",
    "PrivateNodeKey": "privkey:...",
    "OldPrivateNodeKey": "privkey:..",
    "Provider": "google",
    "LoginName": "..."
  }
}