Crouton: VPN traffic issues after ChromeOS 77 update?

[adamallgood]

name: bionic
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/bionic...
crouton: version 1-20190902190258~master:9a594a66
release: bionic
architecture: amd64
xmethod: xiwi
targets: xfce,xiwi
host: version 12499.66.0 (Official Build) stable-channel rammus 
kernel: Linux localhost 4.4.190-16770-gb3c36ab5af98 #1 SMP PREEMPT Thu Nov 14 23:21:48 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /mnt/stateful_partition/crouton/chroots/bionic...
name: test
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/test...
crouton: version 1-20190902190258~master:9a594a66
release: bionic
architecture: amd64
xmethod: xiwi
targets: xfce,xiwi
host: version 12499.66.0 (Official Build) stable-channel rammus 
kernel: Linux localhost 4.4.190-16770-gb3c36ab5af98 #1 SMP PREEMPT Thu Nov 14 23:21:48 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /mnt/stateful_partition/crouton/chroots/test...

Please describe your issue:

I use openconnect in a crouton chroot to connect to my office VPN because a smartcard (CAC) is required for authentication, and the Chrome version of Cisco AnyConnect does not have smartcard support (booo). The setup worked just fine for over a year, but starting in late September, after establishing the SSH connection to the host, no further responses are received, and dead peer detection closes the connection.

This is not an issue of shill destroying the tun0 device, and the PKCS#11 middleware successfully provides a token for my smartcard certs, and I get prompted for my pin and seemingly successfully connect. But something is preventing any traffic going through the tunnel.

I have ruled out a problem with my credentials by successfully connecting via AnyConnect on a Windows device. What's hard is determining if the problem is with how the work VPN is set up, or whether something changed when ChromeOS updated to v77 (stable).

Has anyone else experienced this type of problem, or know of anything in recent ChromeOS updates that may be breaking VPN connectivity?

If known, describe the steps to reproduce the issue:

1. Install smartcard reading software (libpcsclite1 pcscd pcsc-tools), gnutls-bin, and use CACkey as the PKCS#11 middleware.
2. Use openconnect to attempt to connect to a Cisco ASA head end
3. Sit back and grimace, or desperately try to ping anything while waiting for the dreaded dead peer detected, attempting to reconnect error. Or use the -vvv option to see your system scream into the abyss with no response.

PS - This problem also occurs in a xenial chroot, so it's not some bionic beaver weirdness. I've tried the older openconnect available in the apt repository, and I've tried building the latest version (v8.05) with no luck. I've tried with https and with dtls. I have been working with the openconnect developers, and they suggested using mitm-proxy to capture all of the http traffic and compare it to a successful connection, but I do not have a Windows system at home and my office IT department wasn't thrilled with helping me with that idea.

[adamallgood]

Additional information: I ran xubuntu from a flash drive on an old PC with a dead hard disk I had lying around. I installed the same packages and used the same procedures as in the crouton chroot, and lo and behold, a stable VPN connection was made! This seems to be further evidence that it is something in the recent CrOS updates that is preventing the VPN connection from working right.

[adamallgood]

Two questions:

1. Is there a place where I could see a more detailed list of what exactly was updated when ChromeOS went from v76 to v77? The Chrome releases blog is too general to be helpful for me.
2. If a ChromeOS update causes a problem with my crouton chroot, where is the best place to report the problem? Is this it, or should I open an issue somewhere else?

Thanks!

[DennisLfromGA]

@adamallgood,

I can give you a bit of help with 1. I think.

You can run a diff between two versions, say 76.0.3809.136 and 77.0.3865.105 by specifying them in a url, like:

• https://chromium.googlesource.com/chromium/src/+log/76.0.3809.136..77.0.3865.105?n=10000

or even

• https://chromium.googlesource.com/chromium/src/+log/76.0.3809.136..77.0.3865.105?pretty=fuller&n=10000

That will be the 'Chrome' browser diffs though, not just CrOS.

You might be able to find something about networking changes that pertain to your issue.

You can file a report using the key-combo Alt+Shift+i when you next run into the issue so the devs can take a look at it and try and figure it out.

You could also search for a crbug about it and if it's not found file a new one.

And you can bring it up in the chromium-os-discuss forum that the devs frequent to see if someone will take a shot at it.

Hope this helps,
-DennisLfromGA

[jamgregory]

@adamallgood - I'm having the same problem as you. If I don't exclude the VPN network interface (tap0) from shill, I can connect to the VPN, but shill will intercept it and stop the IPv4 address from being assigned. When I exclude it, it looks like the VPN traffic is being blocked by something in the OS as it does make the initial connection but nothing more.

Did you get any further with investigating what's going on?

[jamgregory]

Ah-ha! It's taken a while, but it looks like ChromeOS has changed where the routing information is stored in the latest versions of Chrome. By default, your VPN client likely tries to put the routing information into the default routing table, but ChromeOS actually uses table 1002 (not sure if that's standard across all installs, but that's where my system is putting them).

I'm using vpnc, so the following post connect actions work:

ip route del default dev ${TUNDEV}
ip route add **NETWORK/MASK** dev ${TUNDEV} table 1002

[adamallgood]

@jamgregory - thanks for taking a look into this! No, I had not been able to make further headway into this problem. As a workaround, I connect my smartcard to the Xubuntu box I have at home, and remote into it from the Chromebook using Chrome remote desktop. This allows me to telework on my laptop at home, but does not help me if I am out of the house!

Unfortunately, I do not have a lot of experience with networking issues, so I do not understand your workaround, and when I tried running those commands after connecting using OpenConnect, I got errors that **NETWORK/MASK** was an invalid argument.

1. What steps did you take to discover that ChromeOS was using a different table than the default to direct the routing?
2. Are you running vpnc and those post connection actions in a crouton chroot shell, or directly in the ChromeOS shell?
3. Did you use any online resources to help diagnose this issue? If so, would you mind linking them here? Google searches I've tried have yielded little information.

Thanks so much! I'm feeling more hopeful about this issue for the first time in a while.

[adamallgood]

Ok, so I wasn't kidding when I said I do not have a lot of experience with networking issues. I realize that **NETWORK/MASK** is a stand-in for the IP address and mask, so I need to figure out what these are for my case. As far as ${TUNDEV}, that for me is the tun0 device, but in my experiments thus far, I get tun0 not found errors, even though ip route shows the device. Strange. Still hopeful, but not there yet.

[jamgregory]

No worries @adamallgood - just glad to have been able to sort it out 👍

Unfortunately, I do not have a lot of experience with networking issues, so I do not understand your workaround, and when I tried running those commands after connecting using OpenConnect, I got errors that **NETWORK/MASK** was an invalid argument.

Ah yes, sorry - NETWORK/MASK was a placeholder for the specific routing I needed (rather than sending all traffic over the VPN.

1. What steps did you take to discover that ChromeOS was using a different table than the default to direct the routing?

I did some searching around, and some folks suggested that ChromeOS security improvements involved changes to how IP traffic was routed around the system. Lots of searching later, when I checked ip rule, it looked like all traffic was going into table 1002.

1. Are you running vpnc and those post connection actions in a crouton chroot shell, or directly in the ChromeOS shell?

Yeah, I'm having to run the VPN inside crouton as vpnc isn't (or wasn't) supported in ChromeOS directly.

Pretty sure I used this script with vpnc - http://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/HEAD:/vpnc-script - and then I added a vpnc-script-post-connect-action script to handle the VPN routing.

As with most VPN clients (I expected), by default they send all the traffic down the VPN when connected, which is why I guessed it was a network routing issue. It was only when I played around with the post connect script and messed around with some combinations of rules that I found out adding the new routes was causing a problem.

1. Did you use any online resources to help diagnose this issue? If so, would you mind linking them here? Google searches I've tried have yielded little information.

This article helped flag up the ip rule issue: https://www.reddit.com/r/chromeos/comments/9lgpr8/linux_and_vpns_on_my_samsung_chromebook_plus/e773co4/

Other than that, it was mostly changing things in ip rule and ip route to see what stuck! I've some experience with configuring networks in Linux, so that's kind of how I managed to figure it out. Sorry I can't provide any more useful links though.

As far as ${TUNDEV}, that for me is the tun0 device, but in my experiments thus far, I get tun0 not found errors, even though ip route shows the device.

Ah, TUNDEV is an environment variable that vpnc passes through to the post-connect script, which thankfully takes care of some configuration in my post-connection script.

I don't know whether OpenConnect offers something similar? I assume you're also restarting shill to stop it from intercepting the tun0 set-up, otherwise it drops out the IPv4 address so you can't use the VPN.

[adamallgood]

I finally got it to work! At this point I'm ok with letting everything go through the VPN tunnel, because that's the functionality I had before. I used the changes that OpenConnect/vpnc-script made to my routing tables to guide what needed to go into table 1002. Ultimately, this is what worked for me:

ip route del default dev ${TUNDEV}
ip route add default dev ${TUNDEV} table 1002
ip route add 192.168.56.0/24 dev ${TUNDEV} table 1002

Thank you @jamgregory for pointing me in the right direction! I bought a new Chromebook for teleworking back in September, and very soon after the purchase this issue cropped up. Now finally I can use the device directly, rather than plugging my card into my Linux desktop to get on the VPN (and using remote desktop if I wanted to work from the Chromebook). That workaround was fine while I was physically home, but until now I couldn't telework in a different location. Thanks also @DennisLfromGA for the helpful resources.

[jamgregory]

Glad to hear it is working @adamallgood and happy I could help :+1:

[adamallgood]

@jamgregory and anyone else stumbling onto this issue, if you are interested, the devs at the Chromium project have provided a lot of helpful information about routing on ChromeOS in this crbug: https://bugs.chromium.org/p/chromium/issues/detail?id=1033979

One point in particular to help future-proof this workaround: ChromeOS sets up the separate table in the RPDB based on the network interface ID at connection time. For WiFi networks (wlan0) this is 1002, but that isn't set in stone. Instead of hardcoding table 1002 in the routing commands, they suggested using something like: CRTABLE=$(ip route show table 0 0/0 | grep wlan0 | grep -o 'table [0-9]*'). This should work for WiFi networks, but not a directly plugged in ethernet connection, which uses a different device (e.g., eth0). I guess a separate post-connection script could be used for those cases.

From the discussion, the devs may create a way to do this that is safe from shill, but for now the workaround is protected since we tell shill to blacklist tun0 devices in order to use openconnect in crouton.