Streisand: WireGuard on GCE not forwarding

Created on 19 Apr 2017 · 25Comments · Source: StreisandEffect/streisand

Hello, I'm not sure if I'm missing something or there is a bug, after successful install, I can ping and connect to the server thru the tunnel but any other IP wont answer, I checked forwarding and it looks like enabled but it's a no go. Any idea welcome :-)

arewireguard kinnetwork providegce

Source

x8x

Most helpful comment

Looks like upstream fixed this problem, kernel.org's stable kernels fixed this problem, but Ubuntu forgot to backport the fixes.

Please "vote" on this issue if you'd like it to be fixed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685416

zx2c4 on 22 Apr 2017

👍2

All 25 comments

What distribution are you on?

jlund on 19 Apr 2017

Tried with LEDE nightly and Ubuntu 16.04.2 LTS, both can ping server, read https, query dns, but no connection to other IPs.
If it's supposed to forward then there must be an issue, if not I would welcome infos on how to set up forwarding. Thanks!

x8x on 19 Apr 2017

john@ubuntu:~$ ping -c 1 10.192.122.1
PING 10.192.122.1 (10.192.122.1) 56(84) bytes of data.
64 bytes from 10.192.122.1: icmp_seq=1 ttl=64 time=63.5 ms

--- 10.192.122.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 63.581/63.581/63.581/0.000 ms
john@ubuntu:~$ tracepath 8.8.8.8
 1?: [LOCALHOST]                                         pmtu 1420
 1:  10.192.122.1                                         52.591ms 
 1:  10.192.122.1                                         63.825ms 
 2:  no reply
 3:  no reply

Firewall logs:

Apr 19 01:28:45 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29517 DF PROTO=ICMP TYPE=8 CODE=0 ID=31016 SEQ=1 
Apr 19 01:28:46 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29622 DF PROTO=ICMP TYPE=8 CODE=0 ID=31016 SEQ=2 
Apr 19 01:28:47 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29824 DF PROTO=ICMP TYPE=8 CODE=0 ID=31016 SEQ=3 
Apr 19 01:28:48 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29894 DF PROTO=ICMP TYPE=8 CODE=0 ID=31016 SEQ=4

x8x on 19 Apr 2017

Is masquerading turned on on the server? Please send the output of

# iptables -t nat -L -v -n
# iptables -L -v -n
# cat /proc/sys/net/ipv4/ip_forward
# cat /sys/module/wireguard/version

zx2c4 on 19 Apr 2017

...
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  165 10509 MASQUERADE  all  --  any    any     anywhere             anywhere            
    0     0 MASQUERADE  all  --  any    ens4    10.8.0.0/24          anywhere            
    0     0 MASQUERADE  all  --  any    ens4    10.9.0.0/24          anywhere

x8x on 19 Apr 2017

john@ubuntu:~$ /etc/rc.local 
Warning: ignored obsolete keyword 'nat_traversal'
Warning: ignored obsolete keyword 'oe'
Warning: ignored obsolete keyword 'nat_traversal'
Warning: ignored obsolete keyword 'oe'
Warning: ignored obsolete keyword 'nat_traversal'
Warning: ignored obsolete keyword 'oe'
Redirecting to: systemctl start ipsec.service
[ ok ] Restarting xl2tpd (via systemctl): xl2tpd.service.
0

I switched order of rules in /etc/rc.local moving WireGuard rules before those of L2TP/IPsec like here:

# WireGuard
iptables -A FORWARD -s 10.192.122.0/24 -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.192.122.0/24 -o ens4 -j MASQUERADE

# L2TP/IPsec
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT
iptables -A INPUT -p udp -m multiport --dports 500,4500 -j ACCEPT
iptables -A INPUT -p udp --dport 1701 -m policy --dir in --pol ipsec -j ACCEPT
iptables -A INPUT -p udp --dport 1701 -j DROP
iptables -A FORWARD -i eth+ -o ppp+ -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ppp+ -o eth+ -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.42.0/24 -o eth+ -j SNAT --to-source 10.148.0.2

Now after reboot iptables -t nat -L -v shows:

...
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  139  8692 MASQUERADE  all  --  any    any     anywhere             anywhere            
    0     0 MASQUERADE  all  --  any    ens4    10.8.0.0/24          anywhere            
    0     0 MASQUERADE  all  --  any    ens4    10.9.0.0/24          anywhere            
    0     0 MASQUERADE  all  --  any    ens4    10.192.122.0/24      anywhere

Masquerade rule is now there, but something still missing, can't ping other IPs thru tunnel.

x8x on 19 Apr 2017

@zx2c4 That is after I switched order in rc.local (see above):
https://pastebin.com/raw/9UBGXeHw

x8x on 19 Apr 2017

Just made a fresh install.

#

After fresh install:

Output of:

iptables -t nat -L -v -n
iptables -L -v -n
cat /proc/sys/net/ipv4/ip_forward
cat /sys/module/wireguard/version

https://pastebin.com/raw/YdQ8Gy5p

Output of systemctl:
https://pastebin.com/raw/ckhLZ3T5

Output of systemctl status:
https://pastebin.com/raw/CWbYJ4n2

#

After fresh install and reboot:

Output of:

iptables -t nat -L -v -n
iptables -L -v -n
cat /proc/sys/net/ipv4/ip_forward
cat /sys/module/wireguard/version

https://pastebin.com/raw/4yqtMtei

Output of systemctl:
https://pastebin.com/raw/3Wta71ZS

Output of systemctl status:
https://pastebin.com/raw/r58H5f2F

x8x on 19 Apr 2017

Thanks. Sorry one more command: ip addr

zx2c4 on 19 Apr 2017

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 42:01:0a:94:00:02 brd ff:ff:ff:ff:ff:ff
    inet 10.148.0.2/32 brd 10.148.0.2 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::4001:aff:fe94:2/64 scope link 
       valid_lft forever preferred_lft forever
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::9d35:7bcf:2405:ae1/64 scope link flags 800 
       valid_lft forever preferred_lft forever
4: wg0-server: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1
    link/none 
    inet 10.192.122.1/24 scope global wg0-server
       valid_lft forever preferred_lft forever
5: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.9.0.1 peer 10.9.0.2/32 scope global tun1
       valid_lft forever preferred_lft forever
    inet6 fe80::1a08:3269:b995:ae7a/64 scope link flags 800 
       valid_lft forever preferred_lft forever
6: ip_vti0@NONE: <NOARP> mtu 1332 qdisc noop state DOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0

x8x on 19 Apr 2017

@zx2c4 I have a fairly fresh GCE instance you can use to troubleshoot if you would like. Just send me your public key and I can add it.

CorbanR on 21 Apr 2017

A nice Github trick allows for this: https://github.com/zx2c4.keys

I'll be back in ~10 hours and will take a look.

zx2c4 on 21 Apr 2017

TIL, thanks!
Keys has been added to [email protected] feel free to do whatever you want with this sever.

This may or may not matter, but after reading the backlog again, I started looking at the GCE module we use. The module has an ip_forward options which is disabled by default. https://docs.ansible.com/ansible/gce_module.html. I can enable that and spin up a new server if needed.

CorbanR on 21 Apr 2017

Tried that without success, maybe worth a retry as by the time I had already messed up the firewall rules. Thought, it should run NATed like L2TP and Openvpn which work fine with the GCE instance forwarding option turned off.
With tcpdump, I couldn't see traffic coming from wg0-server go thru the outgoing interface, also worth a retest.

x8x on 21 Apr 2017

Just updated client and server to WG 0.0.20170421, NAT seams to be working now, but the firewall seams to be blocking:

# journalctl -xf
Apr 21 19:22:23 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=34008 DF PROTO=ICMP TYPE=8 CODE=0 ID=4787 SEQ=12 
Apr 21 19:22:24 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=34192 DF PROTO=ICMP TYPE=8 CODE=0 ID=4787 SEQ=13 
Apr 21 19:22:25 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=34234 DF PROTO=ICMP TYPE=8 CODE=0 ID=4787 SEQ=14 
Apr 21 19:22:26 streisand kernel: [UFW AUDIT] IN=wg0-server OUT=ens4 MAC= SRC=10.192.122.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=34291 DF PROTO=ICMP TYPE=8 CODE=0 ID=4787 SEQ=15 

# tcpdump -i ens4 not port 22
19:22:23.253165 IP 10.148.0.2 > google-public-dns-a.google.com: ICMP echo request, id 4787, seq 12, length 64
19:22:24.245219 IP 10.148.0.2 > google-public-dns-a.google.com: ICMP echo request, id 4787, seq 13, length 64
19:22:25.241188 IP 10.148.0.2 > google-public-dns-a.google.com: ICMP echo request, id 4787, seq 14, length 64
19:22:26.241714 IP 10.148.0.2 > google-public-dns-a.google.com: ICMP echo request, id 4787, seq 15, length 64

x8x on 21 Apr 2017

After 24 hours of debugging, I figured it out, and it's extremely low level. I'm following up with the relevant parties and will report here once it's fixed.

zx2c4 on 22 Apr 2017

👍1

Thank you @zx2c4 for taking care of this! Yours is an amazing project! Streisand as well! :-)

x8x on 22 Apr 2017

Looks like upstream fixed this problem, kernel.org's stable kernels fixed this problem, but Ubuntu forgot to backport the fixes.

Please "vote" on this issue if you'd like it to be fixed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685416

zx2c4 on 22 Apr 2017

👍2

@zx2c4 Do you think its worth conditionally skipping Streisand's Wireguard role on GCE until the required patches are backported?

cpu on 22 Apr 2017

Give them til Tuesday to respond, and then we'll see what to do. I suspect there will be a fairly quick cycle for fixing this, since Google Cloud platform is a big customer, and breaking stuff for them isn't good.

zx2c4 on 22 Apr 2017

In the meantime you can get WG working on GCE by installing:
linux-generic-hwe-16.04-edge

x8x on 22 Apr 2017

The Ubuntu kernel guy wrote:

Hi Jason, actually we found out that someone else already had been requesting those changes last week. It was decided that it is better to get these changes out quickly so it should be in the next few days. I will mark your report as a duplicate to the older age report.

Prior bug report is here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1683947

zx2c4 on 24 Apr 2017

I'm pretty sure this should be fixed by now. Can somebody test with a fresh instance and report back?

zx2c4 on 20 May 2017

I have a fresh GCE instance coming up now. Will report back in a short while.

cpu on 20 May 2017

I was able to confirm that Wireguard appears to be working fine with a fresh GCE instance. I was able to route test traffic through the Wireguard interface without issue.

Thanks everyone! I'm going to close this issue - please open a new issue if you run into any further problems.

cpu on 20 May 2017

❤1

Was this page helpful?

0 / 5 - 0 ratings