Algo: Connection Fails on Windows 10 Error 809

[whitelje]

OS / Environment

Windows 10

Ansible version

2.2.1.0

Version of components from requirements.txt

Summary of the problem

Windows 10 fails to connect to the VPN. Message given is "The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices between you and the remote server is not configured to allow VPN connections. This is Error 809.

Steps to reproduce the behavior

Install using normal steps. Make sure to say yes to windows 10 support.

The way of deployment (cloud or local)

Local

Expected behavior

Windows 10 connects.

Actual behavior

Windows 10 throws error 809.

Full log

No installation errors. I'm happy to provide the log if necessary, but it doesn't feel this way.

Debugging so far...

• I've added the fix for double NAT to my registry, but I don't think this is a double NAT situation.
• I've enabled logging for charon by disabling the apparmor rule.
• Logging on charon reveals that the IKE_SA_INIT packets are exchanged correctly. Windows makes a request and charon responds. This is confirmed by tcpdump and the Microsoft ripoff of Wireshark.
• Windows sends three response packets of type IKE_AUTH. These are the cert. They are fragmented at some point. I'm pretty sure this is at the Windows level. tcpdump receives these packets and warns that they have invalid lengths. Netstat shows that these packets fail to be reassembled. Port escalation also takes place during this step. The IKE_SA_INIT packets were sent at port 500. These are at port 4500. Charon is listening to this port.
  ~
  root@algo:~# tcpdump -vv -i eth0 'port 4500'
  tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
  07:50:02.212130 IP (tos 0x0, ttl 116, id 31090, offset 0, flags [+], proto UDP (17), length 1500)
  cpe-174-97-249-29.ma.res.rr.com.ipsec-nat-t > algo.local.ipsec-nat-t: NONESP-encap: isakmp 2.0 msgid 00000001 cookie ff19042cd25c8f5d->e383f9da91d60f23: child_sa ikev2_auth[I]: [|v2e] (len mismatch: isakmp 2528/ip 1468)
  07:50:03.209949 IP (tos 0x0, ttl 116, id 31091, offset 0, flags [+], proto UDP (17), length 1500)
  cpe-174-97-249-29.ma.res.rr.com.ipsec-nat-t > algo.local.ipsec-nat-t: NONESP-encap: isakmp 2.0 msgid 00000001 cookie ff19042cd25c8f5d->e383f9da91d60f23: child_sa ikev2_auth[I]: [|v2e] (len mismatch: isakmp 2528/ip 1468)
  07:50:04.209639 IP (tos 0x0, ttl 116, id 31092, offset 0, flags [+], proto UDP (17), length 1500)
  cpe-174-97-249-29.ma.res.rr.com.ipsec-nat-t > algo.local.ipsec-nat-t: NONESP-encap: isakmp 2.0 msgid 00000001 cookie ff19042cd25c8f5d->e383f9da91d60f23: child_sa ikev2_auth[I]: [|v2e] (len mismatch: isakmp 2528/ip 1468)
  ~
  These three packets never make it to charon.
  ~
  root@algo:~# netstat -s | grep reass
  172 reassemblies required
  172 packet reassembles failed
  ~
• I've played around with smaller MTU's. I've verified that my router is set for MTUs equal to those of Windows.
• Android sends packets of a maximum length of 1400. The VPN does work on Android.
  ~
  root@algo:~# tcpdump -vv -i eth0 'port 4500'
  tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
  07:53:56.039006 IP (tos 0x0, ttl 52, id 54785, offset 0, flags [DF], proto UDP (17), length 1400)
  cpe-174-97-249-29.ma.res.rr.com.49376 > algo.local.ipsec-nat-t: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9467c5635053f986->5520d8bef3f2d259: child_sa ikev2_auth[I]:
  (#53) [|v2IDi]
  07:53:56.048031 IP (tos 0x0, ttl 52, id 54787, offset 0, flags [DF], proto UDP (17), length 1400)
  cpe-174-97-249-29.ma.res.rr.com.49376 > algo.local.ipsec-nat-t: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9467c5635053f986->5520d8bef3f2d259: child_sa ikev2_auth[I]:
  (#53)
  07:53:56.048103 IP (tos 0x0, ttl 52, id 54788, offset 0, flags [DF], proto UDP (17), length 913)
  cpe-174-97-249-29.ma.res.rr.com.49376 > algo.local.ipsec-nat-t: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 9467c5635053f986->5520d8bef3f2d259: child_sa ikev2_auth[I]:
  (#53)
  ~

Ideas

This feels like an issue with how Windows is fragmenting the packets. But most of the time when I'm totally sure I've narrowed down where the bug is coming from I'm wrong.
Is it possible there's an issue with reassembling UDP packets alltogether? Either they're not making it to the host properly and can't be reassembled, or maybe there's like an iptables rule killing the packets?

[whitelje]

Works on a second Windows 10 machine from a different internet connection. Packet reassemblies succeed.

I'm guessing something installed on the first machine has made some sort of changes at the network layer which is mangling packets. I'll have to look into it further after work.

[joshmeisels]

@whitelje I hit this yesterday, but I think Windows was giving the wrong error message. The real issue was that the cert hadn't properly installed. Check that both necessary certs are in the right folders (run certmgr.msc).

[jcll]

Getting this even with both certs copied to the right locations. Did some troubleshooting and seems to have the same issue even on a fresh windows 10 install. Version 1703 (15063.138). Wondering if its something in the creators update breaking things.

[MiWCryptAnalytics]

I also have seen the issues noted by whitelje, where I could not connect from a Windows 10 VM.
We also share the same ISP, which is a cable provider.
I did some tcpdumps/wireshare to determine why this was failing.

I believe this issue is caused by IP packet fragmentation in your environment. Playing by making MTU lower for interfaces wont solve this, as the issue is that the ISAKMP payload wont fit in a single IP datagram, and this fragmented datagram is dropped upstream from you.
I am proposing that many consumer cable modems are shipping with IP Fragmentation blocks, which is breaking IPSec.

This is config, strongswan requires a windows client send its cert in the IKE_SA_INIT exchange but the IKE_AUTH request was not reaching the VPN server. I also tried to modify strongswan to not request rightsidecert as suggested here: https://wiki.strongswan.org/issues/1485 but it had no effect on the window client behavior.

in my setup the IKE_AUTH request was being fragmented over 2 IP datagrams at 2100ish bytes; I believe the encrypted payload is so big because of RSA certificates. One fix here may be to switch to ECDSA certs, which would likely fit a single IP datagram without fragmentation. OpenSSL CA should handle generation of these and should be supported on modern things.

My solution:
I solved this by noticing that block IP fragmentation was enabled on my cable model (Tp-link Archer CR700).
Advanced, Firewall, Basic, Block Fragmented IP Packets set to Disable.
As soon as fragmentation was permitted, VPN worked as expected.

[jcll]

@MiWCryptAnalytics You are 100% right. Issue was caused my UDP fragmentation being borked by my Edgerouter lite. Thanks for the help!

[dguido]

This is really strange. Can someone contribute a Troubleshooting entry for this? I think it deserves one.

[whitelje]

@jcll Can you post how you fixed the fragmentation issue with the ERL? I've got the same router.

[jcll]

Wasn't able to fix the issue. Ended up just pulling it from my network. Going to try to troubleshoot it. But I am guessing it needs a FM fix from ubnt.

[jcll]

Updated the ERL to v1.9.7alpha1 from 1.9.1 and can confirm the issue has been fixed.

[darland6]

I was having fragmentation issues with the IKE_AUTH requests using strongSwan in ubuntu 16.04, but luckily I stumbled across an easy fix in my ipsec.conf file. By setting fragmentation=no in my config file the packet was not fragmented and the connection worked. Hopes this helps save someone else some time!

[chi11ax]

How do I do the equivalent of the above (darland6 method) in Windows 10? Some router from my ISP is causing the problem if it indeed is the router fragmentation configuration listed above. My apartment ISP and a public one I can log into both don't allow my WIn10 machine to connect but my Androids work fine.

That kinda negates the VPN advantage if a router can set a configuration that defeats it. :) Both my Android devices work fine on both VPNs (I set up two different ones to test ...) but my Win10 can't connect to either.

I tried this registry fix and it didn't work:

https://github.com/hwdsl2/setup-ipsec-vpn/blob/master/docs/clients.md#windows-error-809

[lance0]

@chi11ax this isn't a support forum.

[Ralph-Lee]

@darland6 set "_fragmentation=no_" in _ipsec.conf_ ,worked for me, thanks!