Lxd: Proxy device: port range

[killua-eu]

Required information

• Distribution: ubuntu
• Distribution version: 18.04
• The output of "lxc info" or if that fails:

config:
  core.https_address: 192.168.88.100:8443
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 192.168.88.100:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
...
    -----END CERTIFICATE-----
  certificate_fingerprint: [masked]
  driver: lxc
  driver_version: 3.0.0
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.15.0-20-generic
  server: lxd
  server_pid: 1785
  server_version: 3.0.0
  storage: btrfs
  storage_version: "4.4"
  server_clustered: true
  server_name: lxdhost

Issue description

Proxy device port ranges possibly not implemented?

Steps to reproduce

lxc launch ubuntu:18.04 c1
lxc exec c1 -- apt install nginx
lxc config device add c1 http proxy listen=tcp:192.168.88.100:80 connect=tcp:127.0.0.1:80 bind=host

using the browser to get 192.168.88.100:80 results in the nginx welcome page. continuing with

lxc config device remove c1 http 
lxc config device add c1 http proxy listen=tcp:192.168.88.100:70-90 connect=tcp:127.0.0.1:70-90 bind=host

returns Device http added to c1 but the port range doesnt work. So either the syntax is different (and I couldn't find it), or this isn't implemented. Implementing this is IMO quite important for the usability of the proxy device, some services, i.e. ftp, require a dedicated range of ports (in case of ftp for passive transfers). Other uses may be i.e. in p2p networking, etc.

[brauner]

Seems like something that might be worth implementing.

[stgraber]

Should be fine so long as we can have a single forkproxy instance handle the entire port range, if we need one proxy process per port that'd be quite wasteful and difficult to track.

[brauner]

That shouldn't be too hard. We basically have two good options:

• threaded-listeners aka separate go routines for each port
• single-threaded epoll-listener

[brauner]

Also, I'm just going to support this for TCP now, not UDP.

[stgraber]

Ah, what's making udp harder?

[brauner]

If we do UDP support we also need to support cross-protocol which takes a little more thinking since we need to poll on multiple instances then.

[brauner]

Also I'm almost certain that {e}poll() might be the way to go here. If users give us something like:

tcp:ip:1,2,3,4,5,6,10-200,800-1200

we're going to be using a lot of go routines.

[brauner]

Oh right, I plan on supporting the following syntax tcp:ip:1,2,3,4,5,6,10-200,800-1200 aka:

<protocol>:<addr>:<port_0>[-]<port_n>[,<port>]...

[stgraber]

Well, forwarding between udp and tcp is something we can certainly skip (even in general if that lets us cleanup some code) as I can't really think of a legitimate usecase for this.

However forwarding a range of UDP port is very common (needed for FTP for example) so we should really support that part :)

Does banning forwarding between udp and tcp in general (and returning a suitable error) make things easier for us? If so, please do that :)

[brauner]

I'm currently thinking about brute-force way using go routines for a first try and if we run into performance issues going down the epoll route. It would be nice but it would also involve a lot of C boilerplate and careful handling of cgo memory handling. The way I thought about it was essentially to register an fd in an epoll instance and then as data add the go listener but I suspect that the go listener contains go pointers which is illegal in cgo. So go routines it is for now.

[killua-eu]

I concur with @stgraber. Cross-protocol support on whole port ranges is something I can't imagine to actually use, except for some really unorthodox setups. Otoh having the UDP range forward to an UDP range makes total sense. The FTP usecase + some mailstack scenarios are actually one the important reasons why I opened this issue.

[brauner]

If it's abstractable easily I'll support everything. I have the infrastructure in place already.

[brauner]

@killua-eu, I just sent a branch marked as RFC. Please take a look if this covers your use case. I specifically need to know whether the following is what we want or not (//cc @stgraber):

The old implementation used to call connect() every time a new client got
accepted. Iiuc, this is not what we want. Ideally, we'd want all clients to
dump their traffic to the same connect()ion. This is especially true when we
are forwarding multiple ports. Unfortunately, this makes the actual
implementation more complex.
In any case, I might be mistaken and what we want is that each new accepted
client on the forwarded port also causes a new connect() call.

The gist is that each new client (for UDP and TCP) causes accept() but all share the same connect() to forward to.

[killua-eu]

@brauner , the usual usecase for me is to map a port range to a port range, keeping the port range sizes the same on listen and connect, I commented https://github.com/lxc/lxd/pull/4663/files#diff-fc8681e9ed281b244aced7594682e21f accordingly. Not being familiar with the lxd codebase and go, I only skimmed through the code.

[killua-eu]

@stgraber , @brauner: Guys, I just want to send a big THANK YOU to both of you. This (along with the other pf issues you closed really fast) is a really big deal greatly simplifying all my LXD deployments. If you're ever in the Czech rep., let me know, I owe you some beer/coffee.