Streisand: Wireguard stopped working one day after installation

[dmos62]

Expected behavior:

To be able to use Wireguard.

Actual Behavior:

Clients (tried Windows, iOS, Android) fail to ping or load any webpages when Wireguard is activated. However, Wireguard worked the day when it was setup (yesterday). It's hosted on DigitalOcean, and I hadn't changed any configurations or even SSH'ed in. At the same time, Shadowsocks works.

Windows client's log prints this repeatedly after activation: Handshake did not complete after 5 seconds. The log has no indication that the handshake ever succeeds.

Additional Details:

Target Cloud Provider:

DigitalOcean

Operating System of target host:

Ubuntu 16.04.6 (LTS) x64

Operating System of client:

Windows 10, iOS, Android

Version of Ansible, using ansible --version :

As provisioned by install scripts yesterday.

Output from git rev-parse HEAD in your Streisand directory :

[dmos62]

Tips to troubleshoot would be appreciated. Meanwhile I'll try and spin up another Streisand droplet.

[bensteinberg]

I have had this exact experience with two droplets. The mechanism, I believe, is that unattended-upgrades installs a new kernel and headers (4.4.0-161), and then wireguard-dkms can't build the new module. This is basically https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934763 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935134 -- for which Debian has a fix, https://salsa.debian.org/debian/wireguard/commit/dca2647fa39e037448b450cb82d5d62b13685b25 -- but the WireGuard Ubuntu Packaging team does not yet. I've emailed them.
I was just experimenting with manually adding the fix on my droplet, and got the module to build, but have not gotten wireguard to work.

[bensteinberg]

I got it working -- I added

#define hsiphash_2u32 siphash_2u32
#define hsiphash_3u32 siphash_3u32
#define hsiphash_key_t siphash_key_t

to /var/lib/dkms/wireguard/0.0.20190702/source/compat/compat.h, ran

dpkg-reconfigure wireguard-dkms
modprobe dkms                       ## probably not necessary
reboot                              ## but this seems to be

and when the droplet came back up, I could use wireguard.

[nopdotcom]

Thanks much, @bensteinberg.

[dmos62]

You have my thanks as well, @bensteinberg; it works! Is this a
permanent fix, or should we expect the next unattended kernel upgrade
to again disable Wireguard?

On 9/5/19, Jay Carlson [e-mail] wrote:

Thanks much, @bensteinberg.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/StreisandEffect/streisand/issues/1637#issuecomment-528400063

[bensteinberg]

Good question, @dmos62. This should not be considered a permanent fix.
With any luck, the the real fix to wireguard-dkms will precede or coincide with the next kernel upgrade (which should itself contain a fix obviating this workaround or the one in wireguard-dkms). However, I don't know whether unattended-upgrades will fail to update wireguard-dkms if this local fix is present.

[bensteinberg]

I have now had the same problem on an AWS EC2 instance, and fixed it the same way. The kernel package in that case is linux-image-4.4.0-1092-aws.

[martelrkm]

I just figured this out out as well. Same problem, same solution. Once building a kernel module from source everything works as expected.

[bensteinberg]

There are new packages for wireguard-dkms and wireguard-tools in the PPA -- thanks, @zx2c4.

apt update
apt upgrade
reboot

did the trick for me, though I think

apt update
apt upgrade
rmmod wireguard
modprobe wireguard

possibly followed by

systemctl daemon-reload

should also work.
(apt upgrade produced a warning, dpkg: warning: unable to delete old directory '/usr/src/wireguard-0.0.20190702': Directory not empty which I assume is an artifact of the fix above, and can be ignored.)
This was on a Digital Ocean droplet -- I haven't upgraded an EC2 instance yet.

[zx2c4]

Thanks for the report. Ubuntu's kernel team made an error in backporting a patch from mainline. They're aware of this error, but they're unable to rectify it for another 7 weeks, which clearly isn't an acceptable amount of downtime for WireGuard servers. So yesterday we made a new WireGuard release that works around this issue. Then, this morning, @EggieCode, the maintainer of our Ubuntu package, published a new Ubuntu package for our new release. So you should be all set by running apt update && apt upgrade. Sorry for the hassle. I'm not too happy about this situation myself, and I'm going to try to figure out how the Ubuntu kernel people can make sure they don't break WireGuard in the future with some sort of CI.

[bensteinberg]

My EC2 Streisand instance upgraded the wireguard-dkms and wireguard-tools packages automatically, and everything's working fine.

[bensteinberg]

I think this has just happened again, though slightly differently. An update on a Digital Ocean droplet got the 4.4.0-164 kernel, but didn't automatically install the linux-headers-4.4.0-164-generic package, which is necessary for the wireguard kernel module to build. I installed that package and rebooted, and now wireguard is working again. I guess I need to tip the hat to #1640.