Docker-transmission-openvpn: Implement official PIA "Next-gen" scripts

[taylorcox75]

How it currently works:

• User specifies server
• We download zip files containing openvpn configs and implement the configuration for that server
• Contact pia/api/v2 for a token
• Contact vpnserver:19999 for payload (port/signature)

Before releasing official headless Linux scripts to select their next gen servers, they sunset their legacy servers. So although we do the same thing, the new official version is something to consider

New Official Method (released a few months ago):
Rather than specify a server to use, the user specifies requirements such as

• openvpn_udp_standard, openvpn_udp_strong, openvpn_tcp_standard, openvpn_tcp_strong
• Port forwarding (yes or no)
• max server ping (default 0.02)

Then it picks the best server that fits those requirements.

Process Oversimplified

Code wise the official method finds the best server from the below list, and creates the openvpn config file from JSON parameters
https://serverlist.piaservers.net/vpninfo/servers/v4

They also utilize a V3 API running on the vpn server for tokens and the payload(port/sig), while we use a V2 API on www.privateinter....com/api/v2 to get the token, then contact the VPN server for the payload. Potential issues could arise if they decide to sunset the v2 running on there main domain.

In other words, instead of configuration files, a few lines of code takes the JSON parameters and builds the configuration file. So the config files that we're using aren't necessarily the wrong way to do it, But there's a better more dynamic way to do it that doesn't rely on setting static URLs and depending on formatting of names

Recent Issues (from not using the official method)

Our method relies on the zipped configuration files from a URL on their website. Thats more geared towards the "I need to setup PIA config files for my openvpn client" opposed to "I'm often going to be downloading from that URL and rely on the names and capitalizations being the same"

Recently they renamed all the names to lowercase, so Sweden being specified as a configuration caused the container not to start, lowercase sweden fixed it.

A week ago they renamed the zip files we pull, #1579 fixed it by updating URLs

Solution

I have a spaghetti code version of this I've been using, has worked pretty well.

The only issue is you no longer will be able to specify a specific server to use. The new method takes your requirements and finds the best server with the lowest ping to give you.

I know many people are superstitious about the servers they use with theirs being the "best for peer connection", etc. That's really the only aspect that's stopping me from actually implementing this enhancement.

Path
So I guess we can use this for a discussion, Do you want to be able to pick a server, or have it decide for you?

I'll give it a few weeks and whatever the consensus is will determine if I do a pull request for it for it

[clement-z]

The scripts from pia-foss/manual-connections can definitely be used to connect to a specific server though. The scripts are pretty straight forward and can be adapted.

Even if the names are not frozen on PIA's side, I think it makes sense to let the user specify the server. I tested the scripts a month or so ago, and the delay from picking a server was quite noticeable if I remember well. So keeping the server option has that going for it. :)

However, using the scripts to automatically choose the server with the lowest ping is good, and could be used when OPENVPN_CONFIG is empty.

[superkrups20056]

Definitely want to be able to pick a server. Not even a question.

[taylorcox75]

@superkrups20056

Played around with the official scripts, and found I can set bestRegion to = OPENVPN_CONFIG (so sweden in my case) to force a specific location. Openvpn config wise, identical. Only difference is The official way uses ip's from the json file that change with each request.
Inside .opvpn:
remote 195.246.120.86 1198 udp
Our Zip way
remote sweden.privacy.network 1198

Both ways use an api attached to the region's 'meta' domain, one just does before starting openvpn, the other on first contact.

More Specifically:

Json of selecting sweden manually
https://pastebin.com/NpccHQzX

Server inside my .openvpn after running:
remote 195.246.120.44 1198 udp

When I use the traditional method (zip file with config), starting my container on the latest master release my .opvpn is identical, but instead of remote xxx.xxx.xxx.xx udp (or tcp) I have

proto udp
remote sweden.privacy.network 1198

Docker logs

[stockholm402] Peer Connection Initiated 195.246.120.42

So you can see it's exactly the same IP (192.246.120.4[1-9]

So from here just 2 new docker variables, defaulted to udp and normal (as it is now)
The existing:
OPENVPN_CONFIG=server

New:

PIA_PROTOCAL=tcp or udp
PIA_ENCRYPTION=strong, normal

If $server is empty, the best one based in ping will be chosen. Or if we can crowdsource a "Top 10 servers" we can set it to the closest of those.

[haugene]

I think this sounds really good :raised_hands: For me picking a server wouldn't have been a deal breaker but it's nice to have that feature as well. And as it's important for some it's great that you found the bestRegion option.

What happens if you set bestRegion to some of the US servers that don't have port forwarding? You mentioned that the api had requirement specifications and I guess our default would be a "yes" to port-forwarding. Do we get a "not found" error from PIA? If a user then really wanted to be connected to a US server, should we then allow an override there and at the same time set the DISABLE_PORT_UPDATER=true flag automatically?

Anyways. Looking forward to this. I think it'll be a great enhancement. I'm seeing several issues where users connect to US servers and get a bunch of errors from the port-forwarding scripts (albeit prior to your change there, but that probably doesn't change). This might be a good and automated way to check that the location supports it before we try to reserve ports.

[taylorcox75]

Good news for you! Every server but US has port forwarding enabled.

At the moment, all of our non US Servers offer port-forwarding functionality and you can find a list of all of the servers we offer

The api is also less "api' more dynamic list.
https://serverlist.piaservers.net/vpninfo/servers/v4
It changes a few times a minute based on load of ip nodes

Every server in the json has tcp, udp, strong, and normal encryption. So the only way you would get "no results" is if port forwarding = true and server = US. Easy enough to prevent.

The below json is how they all look. The one entry does
Tcp normal encryption
Tcp strong encryption
Udp normal encryption
Udp strong encryption
All of the above port forwarding of requested

Ex:
```
"id": "aus",
"name": "AU Sydney",
"country": "AU",
"auto_region": true,
"dns": "au-sydney.privacy.network",
"port_forward": true,
"geo": false,
"servers": {
"ovpnudp": [
{
"ip": "117.120.10.180",
"cn": "sydney409"
}
],
"ovpntcp": [
{
"ip": "117.120.10.179",
"cn": "sydney409"
}

    ],
    "meta": [
      {
        "ip": "117.120.10.130",
        "cn": "sydney409"
      }

[haugene]

Allright, looks great! :+1: There are some ways to go about this, but I definitely see an end game here that is better than what we have today. Given that PIA keeps the dynamic list format relatively stable :pray:

[taylorcox75]

It's a json so really hard to screw that up! Plus I already checked if capitalizations matter (it doesn't!)

[clement-z]

I really like it 👍

Since the PIA scripts got out, I thought to myself that we should eventually migrate to it, once they can be considered stable.

How do you plan on integrating this though? I think there are several ways to go about it, the most obvious being:

1. git submodule the pia repo + patch
2. PR the pia repo with changes that we need (as I think they are quite "general purpose") and submodule
3. (Dirty) copy and adapt

I think option 2 makes most sense, as it's IMO an improvement we give them, but then it's on them to keep it up to date (or us with new PRs ^^). If they are not reactive enough, we can always do 1, which is quite OK too.

I am definitely against 3 as it's then an unnecessary maintenance burden on us. It would really only be an option if the PIA repo lags a lot behind their upstream (VPN servers config and API) changes.

You mentioned 🍝 code in your first comment, and it kinda scared me 😅 I'd really like for us to stay aligned with the PIA config (with commit log reviewed as we update the submodule) rather than do things on our side.

So, opinions ?

[taylorcox75]

I don't really have any specific plan, so no need to be scared! In reality as far as code goes, I think what it will end up with only 2 new pieces of logic That's ours it will fit somewhere into what they have

1. "If the user specifies a server, you don't need to do the dance in get_region_and_token, just parse the JSON for the arg"

2. Stop executing connect_to_openvpn.sh the second you generate the .ovpn. (Since we deal with physically connecting to the server in the parent scripts all the VPNs use)

I agree doing #3 puts us back in this same situation in 6 months. And generally I'm pretty lazy, so I'm not going to rewrite what they already wrote. My motto and life is 'any code I ever need to write, somebody's probably already written'

As for if we should go #2 or #3, It really depends on how We decide to eventually structure it. Though they have a git that's updated maybe once every few weeks, on their there "get PIA for headless Linux" guide they tell you to download the v1.0.0 release from the GitHub page. I think this might be the best source since It's basically their production. That way we wouldn't be running The equivalent of nightlys from them

Or we could potentially go the route of basing it off of the master branch opposed to release.zip If it would make implementation on our easier.

[taylorcox75]

2 actually seems like the winner thinking about it again.

Then whomever can PR new releases from their source.

[clement-z]

As for if we should go #2 or #3

I assume you meant options #1 or #2.

Now some answers/comments to your points:

Their repo does not have many recent commits, but their are some PR in the pipelines, such as https://github.com/pia-foss/manual-connections/pull/52 which implements (interactive) server specification. It's not perfect for us but still it's going in the right direction (and the repo is not stale).

As I see it we only want features which are "compatible" with the goal of their repo, right? So I think they will take in our changes if we submit them. As to how to go forward now, I think the classical OSS workflow (option #2) is the best candidate:

1. Fork it,
2. Patch it to work as we want it (i.e. fully non-interactive use + server name parameter + generate config without connecting),
3. Test and work with that fork until they accept our PR,
4. Switch to mainstream if/once it's merged

On the haugene/docker-transmission-openvpn side, it just means:

• Adding a reference to our script repo, either via submodule (best) or in pia/configure_openvpn.sh (acceptable, I guess).
• Adapting the pia/configure_openvpn.sh and pia/update_port.sh scripts to do what we want, given the environment variables given (and maybe split PIA_OPENVPN_CONFIG_BUNDLE into ...PROTO and ...STRENGTH).

I don't think it's very hard overall. I will be able to help with coding/reviewing/testing during the holidays, as most of our usual Christmas reunions are cancelled. 😢

[taylorcox75]

Yes, I agree.

I definitely like that PR you mentioned. So going off of that it seems like the best path forward to fork the repo.

I have an open weekend (depending on work lol) where I can get something like you described built and put on a docker hub repo to test (or build from source)