Shadowsocks-android: App do not behave normally using vpn via wifi

[evopen]

Please read FAQ then answer these questions before submitting your issue. Thanks!

Environment

• Android version: 8.1.0 OPM1.17.10.19.013
• Device: OnePlus 3T
• Shadowsocks version: 4.5.1
• Last version that did not exhibit the issue (if applicable): Android 8.0 and before

Configuration

_Put an x inside the [ ] that applies._

• [x] IPv4 server address
• [ ] IPv6 server address
• [x] Client IPv4 availability
• [ ] Client IPv6 availability
• Encrypt method: rc4
• Route
  
  
  
  • [x] All
  
  
  • [ ] Bypass LAN
  
  
  • [ ] Bypass China
  
  
  • [ ] Bypass LAN & China
  
  
  • [ ] GFW List
  
  
  • [ ] China List
  
  
  • [ ] Custom rules
  
  
• [ ] IPv6 route
• [ ] Apps VPN mode
  
  
  
  • [ ] Bypass mode
  
  
• Remote DNS: 8.8.8.8
• [ ] DNS Forwarding
• Plugin configuration (if applicable):
• [x] Auto Connect
• [x] TCP Fast Open
• If you're not using VPN mode, please supply more details here:

What did you do?

Connecting the VPN using wifi network and use Google Photos

What did you expect to see?

Google Photos back up automatically.

What did you see instead?

Google Photos does not backup at all

Android Issue Tracker: https://issuetracker.google.com/issues/68657525
And latest reply from Googler:

Status: Won't Fix (Intended Behavior)
When a VPN is up the OS has no way to guarantee that network traffic will go on wifi. The VPN app could decide to send and receive traffic on any available interface, including cell data. Recent changes to the VPN code have made it so that if the VPN calls setUnderlyingNetworks, the networking APIs will reflect some of the properties of the underlying networks (e.g., whether they are metered or not) into the VPN's networkcapabilities. But it's up to the podcast app to use the required APIs, such as isActiveNetworkMetered.

Not sure this is a Shadowsocks client issue, but since Google Play can not download with vpn via wifi. I think it might be a shadowsocks issue.

Thanks

[Mygod]

Looking at the source code, I think this can be fixed. Google is so lazy they don't want to bother checking the underlying Network so we have to do it for them. Great.

[Mygod]

I've tried both setUnderlyingNetworks and none of them even triggers onCapabilitiesChanged. NetworkCapabilities seems to be stuck at [ Transports: VPN Capabilities: INTERNET&NOT_RESTRICTED&TRUSTED&VALIDATED&FOREGROUND].

[Mygod]

It seems like there's no way to change network capabilities in Android 8.1 at least natively and we can only fix this for Android P.

[Mygod]

The latest beta branch should fix this issue on Android P. Please refer to https://github.com/shadowsocks/shadowsocks-android/commit/3a8f863bd1f0deed081de082a674020368f30319 message to see why I don't think it's possible to fix this issue on Android 8.1.

@madeye You might want to test whether this works if you have a Pixel.

[madeye]

@Mygod I'm reading this from the doc

a non-empty array: an array of one or more Networks, in decreasing preference order. For example, if this VPN uses both wifi and mobile (cellular) networks to carry app traffic, but prefers or uses wifi more than mobile, wifi should appear first in the array.

an empty array: a zero-element array, meaning that the VPN has no underlying network connection, and thus, app traffic will not be sent or received.

null: (default) signifies that the VPN uses whatever is the system's default network. I.e., it doesn't use the bindSocket or bindDatagramSocket APIs mentioned above to send traffic over specific channels.

Since we set nothing, Android should think our VPN upstream is using default channel. It makes no sense that Google claims that they don't know which upstream we are using.

Another thing I want to try, if I setUnderlyingNetworks to an empty array, would it solve this issue as well as the battery stats one? As Android system would think our VPN server won't use any traffic?

[Mygod]

I think that doc corresponds to this part of the source code: https://android.googlesource.com/platform/frameworks/base/+/2df4c7d/services/core/java/com/android/server/ConnectivityService.java#3645
However, I don't find this underlyingNetworks to be used anywhere else besides collecting stats. So you might be right for the second question. But one thing I noticed while developing VPN hotspot is that whenever the default network changes, Android actually creates another tun interface (say tun1) and then destroy the old one (say tun0). You can monitor this change using this class. That being said, I haven't found where in the Android source code that actually does this change.

As for why can't we set it to empty array to solve the metered issue, this line should answer your question: https://android.googlesource.com/platform/frameworks/base/+/adbf1d0/services/core/java/com/android/server/connectivity/Vpn.java#312

Feel free to investigate further.

[kcschan]

@Mygod @madeye
How could Google possibly assumes it to be metered with an empty input here?
It looks quite unresponsible:

Vpn#updateCapabilities(ConnectivityManager, Network[], NetworkCapabilities)

[Mygod]

It's actually pretty responsible so that your backup app won't eat up all your data plan while you left your VPN on.

They're just being lazy and delegate the responsibility to developers of VPN and network apps.

[iKirby]

EDIT: installed apk built from beta branch using adb install -t. Behavior is the same as described below.

I manually added changes from 3a8f863 to the code in master branch and built an apk. It can be installed on my device.

With this build (on Wi-Fi network):
Google Photos can detect Wi-Fi network and backup photos automatically.
Google Drive can detect Wi-Fi network and start pending upload tasks automatically.
Play Store and other apps using system DownloadManager are still unable to start download.
Telegram X enables data saver automatically if I choose "When switching to mobile data" in "Turn on automatically" settings.
ConnectivityManager.isActiveNetworkMetered() returns false

Network capability info:

Transports: WIFI|VPN
Capabilities: NOT_METERED&INTERNET&NOT_RESTRICTED&TRUSTED&VALIDATED&NOT_ROAMING&FOREGROUND&NOT_CONGESTED

[wrlssqi]

@iKirby Test OS version is Android 8.1 or Android P Preview?

[Mygod]

That looks perfectly normal to me. Unfortunately I don't have time to look into every app and figure out why they don't work. If you find out any other system API that's reporting this Network as metered, post it here and I might be able to fix it.

[iKirby]

@wrlssqi Android P Preview. Currently I don't have devices running 8.1, and commit 3a8f863 only makes it report network capabilities on Android P.

[ghost]

@Mygod I have a pixel device. OS is Android P . what can i do?

[Mygod]

Send it to me.

[iKirby]

I decompiled Telegram X and found that it gets network type using NetworkInfo.getType(), which returns the value of TYPE_VPN when VPN is connected. But its getConnectionType() method returns 1 when network type is TYPE_WIFI or TYPE_ETHERNET , 2 when network is roaming, and 0 for all other types. So VPN is always treated as mobile network in Telegram X.

As for system download manager, the source of Android P won't be available until final release, so I decompiled it. The connectivity check code seems to be OK. Maybe we should submit a bug report to Google.