Streisand: Support pick-and-choose modular services

[cpu]

This issue is an "omnibus" issue to track first party support for pick-and-choose ("modular") Streisand services. This issue will track the proposed design & the overall progress made to convert individual services/roles.

Motivation

Historically Streisand's anti-censorship goal has been the driving force behind providing many services pre-configured out-of-box. For users on censored & hostile networks it can be difficult to know what will work ahead of time and often the "best" or "most secure" choices are flat out unusable. This userbase continues to need a service like Streisand provides.

On the other hand, the recent Internet climate w.r.t Internet Service Provider surveillance has driven increased demand for a simple do-it-yourself VPN service usable on traditional uncensored networks, leading many to evaluate Streisand. From the perspective of this use-case Streisand's attack surface & complexity are a disadvantage and a risk.

How can a balance be struck? By allowing services to be enabled/disabled, both use-cases can be addressed. For the user's already served by Streisand who need to balance what works with the "academically pure" solution, all services can be enabled/included. For users with a more narrow & straightforward use-case just a bare minimum of the services they require can be included.

Challenge

The difficulty of providing this modularity lies in the interconnections between roles that exists today. Simply disabling the Shadowsocks role (for example) cascades out into other roles that have implicit dependencies on variables & files that the Shadowsocks role populates & creates. The documentation makes reference to Shadowsocks variables. The mirror role assumes that Shadowsocks clients are required. The firewall role requires knowledge of Shadowsocks to enable/disable port mappings.

The most immediate & straight forward way to address this challenge is to pepper the various roles with if statements evaluating whether or not a role was included. This approach (or variations of it) have been proposed in community contributed PRs in the past, but the complexity and overall brittleness of this approach make it hard to manage.

For this Issue we intend to exploit the relative similarity of the needs of a Streisand service to encourage isolated role design. Put simply, each service's role will be required to do its own firewall updates, its own documentation generation, and its own client mirroring. Base firewall/documentation roles will fill in the gaps by providing required system-wide dependencies (the ufw package being installed for instance) and stitch together index documentation based on which services were enabled/disabled.

Base System

I consider the following to be the "base system" (e.g. these are not things you can pick-and-choose). These pieces are either integral to the goal of Streisand (things like a documentation web portal) or required for generic system access/administration (e.g. ufw, ssh).

• ufw
• nginx
• monit (likely to be replaced with better systemd units)
• ssh (the base sshd & config)
• the gateway docs
• the client mirror (this might be something that would make sense to allow toggling for)

Modular Services

Services/roles to modularize/update:

• [x] - shadowsocks - merged https://github.com/jlund/streisand/pull/794
• [x] - tor - merged https://github.com/jlund/streisand/pull/808
• [x] - openvpn - merged https://github.com/jlund/streisand/pull/816
• [x] - l2tp-ipsec - merged https://github.com/jlund/streisand/pull/896
• [x] - openconnect - merged #834
• [x] - ssh fowarding user - merged https://github.com/jlund/streisand/pull/871
• [x] - tinyproxy - merged https://github.com/jlund/streisand/pull/845
• [x] - sslh? - going to punt on this one for now and revisit later.
• [x] - wireguard - merged https://github.com/jlund/streisand/pull/802

Additional tasks:

• [x] Document the zen/approach we're adopting for writing Streisand modules
• [x] Integrate module/service selection into the streisand wrapper script
• [x] Output some kind of "manifest" of user choices that can be referenced in support issues - covered by https://github.com/jlund/streisand/pull/926
• [x] Implement a testing strategy. Ideally we'd test every possible enabled/disabled configuration but that's ~n^2 - merged https://github.com/StreisandEffect/streisand/pull/988
• [x] Update English documentation to describe modularity/service choices (& maybe suggest useful "profiles" of choices?) - punted to https://github.com/StreisandEffect/streisand/issues/1041

[cpu]

I've started some proof-of-concept work modularizing Shadowsocks in a branch: https://github.com/cpu/streisand/commit/039ccc11fa265863acb43c525c811858a857ccfc

This branch also has an initial attempt at providing documentation for the modularization design: https://github.com/cpu/streisand/blob/cpu-optional-shadowsocks/documentation/modular_roles.md

There are outstanding questions to answer about things like dnsmasq and other shared infrastructure that bridges multiple roles. I'm hoping to get into the weeds with these things in the coming week(s).

[cpu]

Finished up with making WireGuard optional in a branch cut off of the cpu-optional-shadowsocks branch mentioned above: https://github.com/cpu/streisand/tree/cpu-optional-wireguard

Doing this cleanly required a fairly major retuning of the dnsmasq role: https://github.com/cpu/streisand/commit/bab644b9e745d43cfd631b433c93a2055c39c7ed The approach used is documented here.

Similarly, the rc-local role was removed entirely and replaced by individual services managing their own /etc/init.d/ scripts describing their own runlevels & inter-dependency needs: https://github.com/cpu/streisand/commit/56b53ab9697580fdf3732c25a181f34ea1d87950

[cpu]

The first modular role (shadowsocks) was merged to master in https://github.com/jlund/streisand/pull/794.
I'm going to move on to cleaning up the wireguard branch for master next.

[cpu]

Second modular role (wireguard) was merged to master in https://github.com/jlund/streisand/issues/802.

[cpu]

Submitted a PR for the Tor role: https://github.com/jlund/streisand/pull/808

[alimakki]

Submitted a PR for the OpenVPN role: #816

[cpu]

Modular Tor role merged to master :tada:

[cpu]

Modular OpenVPN role merged to master thanks to @alimakki :taco: :1st_place_medal:

[alimakki]

Submitted a PR for the OpenConnect role: #834

[cpu]

Modular OpenConnect merged to master. Thanks @alimakki :trophy: :tropical_drink:

[alimakki]

Modular tinproxy merged!

[cpu]

Opened a PR for a modular SSH forward user https://github.com/jlund/streisand/pull/871

[cpu]

Modular SSH forward user merged! Another one down :white_check_mark: :checkered_flag:

[cpu]

Opened a PR for l2tp: https://github.com/jlund/streisand/pull/896

[cpu]

Modular l2tp/ipsec merged. I think that's all of the service roles completed!

[cpu]

Opened a PR to track a manifest of which roles are selected as well as other useful diagnostic information; https://github.com/jlund/streisand/pull/926

[cpu]

Almost there folks! I just merged #936 and we hit an important milestone. You can now interactively enable/disable Streisand services when you run ./streisand. :tada:

The only things that remain are better testing of different enabled/disabled combinations & a README/docs update.

[cpu]

I've implemented a testing strategy for different configurations of Streisand services in https://github.com/StreisandEffect/streisand/pull/988

[cpu]

I've been dragging my feet on a README update because of the translation impact. Since the rest of the work is completed I cut a separate issue for the README update: https://github.com/StreisandEffect/streisand/issues/1041

We can officially call this closed! You can now pick and choose which services will be configured for your Streisand host. Thanks to @alimakki for all the help! This is a real milestone :-)