Lxd: Network Setup Is Too Hard

[srkunze]

Cf. #1292 as a blueprint of what's necessary to make setting up containers as easy as possible.

1) People seem not to care about setting up a bridge. To them it's a low-level solution to their problem: "make the container publicly available (ping, ssh, web, etc.) outside of the container host". I would appreciate an option where one can (with a simple switch) turn on/off this kind of behavior.

2) Bridging wlan0 and lxcbr0 needs to feel the same as bridging eth0 and lxcbr0. To us end-users, it makes absolutely no sense to handle them differently.

You might argue all of that is not LXD's problem because it's "basic" networking. However, I argue it is LXD's problem as one public objective of LXD is to provide a simple tool for setting up containers. Regarding networking that objective is not met IMO.

[jsimonetti]

Personally I feel this is not the responsibility of lxd. The linux mantra is to do one thing and do it well. In that regard, lxd managers your containers extremely well. There are a plethora of tools that do networking very well. NetworManager for the networking, or ufw and shorewall for easy setup of firewalling and nat.
I feel it is not up to lxd to manage your network stack.
On a sidenote, when installing lxd on an Ubuntu system, everything is setup for you to start with containers quickly and easily. The setup shouldn't take advanced users into account.
On another Note, once lxd supports macvlans, it should be very easy for people to do public facing containers with the touch of a button.

[srkunze]

@jsimonetti Thanks for your reply and I agree with the Unix philosophy of doing one thing well. macvlans seems like an interesting approach to the issue at hand. I cannot evaluate whether that's the best solution, though, so I leave that to the ones able to do so.

You said, LXD should not manage your network stack, however, LXC already sets up lxcbr0 for me. It sets up an HWAddr for me. It sets up an IP address for me. So, why stop there?

Do macvlans work with wlan interfaces?

EDIT: https://blog.flameeyes.eu/2010/09/linux-containers-and-networking actually shows two variants of the macvlan approach and another solution using veths.

[jsimonetti]

I see no reason why macvlan would not work on a wlan interface except for maybe wlan driver restrictions or AP restriction. Some APs will forbid multiple mac addresses per association.

Good article, though a bit old, still acurate. Vepa mode for macvlans is pretty much unusable for 99% of the people sinds it requires hardware support from the connected switch and few switches support it. Its mostly only seen in very expensive datacenter switches and is pretty much not-done these days except for very specific use cases involving vm isolation. Bridge mode is what i was refering to.
The veth methode is actually what lxd uses by default.
Flockport has a good guide about advanced container networking here https://www.flockport.com/lxc-advanced-networking-guide/.

[srkunze]

@jsimonetti

The veth methode is actually what lxd uses by default.

Why can I not access the container from another host then? Given that diagram, it looks like as if it should.

Interesting read on Flockport which might really come in handy when the basic use cases are all set.

[jsimonetti]

You cannot get to the containers externally because the default setup doesn't add a physical interface to the lxcbr0. This is intentional, since the default setup is hosting dhcp on lxcbr0 which could cause issues to other systems in the same network.

[srkunze]

@jsimonetti You are right. I have been talking to somebody with a lot more experience in that field and we came to the conclusion that bridging is not the right way here. Routing would be.

So, I configured the DHCP server (Fritz Box) to include an additional route.

I worked magically. :)

No hacky scripts, no 20 lines of difficult configuration; it just works! Thanks for your time and effort here. :)

[srkunze]

Maybe, one could put this into some small section on https://linuxcontainers.org/lxd/getting-started-cli/ ?

[srkunze]

Furthermore, I would still appreciate some switch on LXD's side to support at least the two or three most used setups in order to increase acceptance.

[techtonik]

The veth methode is actually what lxd uses by default.

That's something new to me. Could you expand more on that? Current proposals to configure access from the LAN to containers are way too intrusive for my system as I see it.

[techtonik]

It looks to me that routing is a way to go. At least it looks like common practice among some of LXD developers too, see https://github.com/lxc/lxd/issues/591#issuecomment-99900927, but I don't see a way how LXD could clean up after itself https://unix.stackexchange.com/questions/245151/firewall-that-allows-to-tag-or-comment-rules-for-easy-filtering. And if LXD can not clean up, then I would definitely bring my routes to a mess.

[srkunze]

Current proposals to configure access from the LAN to containers are way too intrusive for my system as I see it.

Intrusive describes it best. I would need to reconfigure almost everything; and even worse by hand.

[srkunze]

@techtonik Not sure if you read my comment the other day, but I use my DHCP server to tell the network that the 10.0.3.0/24 network is reachable via the LXD host as a gateway. It's not port-specific but if not necessary that would fix easily it for now.

[techtonik]

I thought that 10.0.3.0/24 traffic never comes out of container. Only in NATed form from host eth0 interface.

We really need to find some way to share pictures, because maybe we are discussing different things. In my scenario the stuff you've described is only possible is LXD itself somehow runs this DHCP server.

[srkunze]

I thought that 10.0.3.0/24 traffic never comes out of container.

It does. When lxc list shows an IP address (say 10.0.3.3), you can ping it using your local machine. Let's call your local machine H.

H can reach 10.0.3.3 because it is part of the network 10.0.3.0/24 (well, in fact H creates this network, so it is automatically part of it).

There is a router which spans another network (mine is 192.168.178.0/24). H is part of this network, too (via eth0). Assume further, you have a Raspberry Pi (call it P) connected to the 192.168.178.x network. P needs access to 10.0.3.0/24.

General idea: tell P that H is the gateway to 10.0.3.0/24. Then P can access 10.0.3.0/24 and uses H as the entry point into this network.

From what I can tell is that you have two ways to accomplish that (surely among 10 others. ;) ):

1) configure a route on P
2) configure the DHCP server in your router

Does this makes sense?

[srkunze]

This would only enable connectivity from P to Container. If you need it vice verse, you need to configure the DHCP server for the 10.0.3.0/24 network, so that all containers know where to send the packages in order to reach 192.168.178.x addresses.

[techtonik]

Why H should forward packets from P from external network to C in internal network? It looks like security hole for me.

[P]  --- 192 --- [H] --- 10 --- [C]

[jsimonetti]

It is only a security hole for some situations. It is wanted behaviour in other situations.
This is exactly why I feel LXD should not tamper with networking and I do all the networking setup and firewalling myself, as should most of the people who want to use LXD in production environments.

First thing I do after installing LXD i disable the default lxcbr0 and dnsmasq DHCP server and setup my own. Be advised: all these defaults are done by the ubuntu packages and not by LXD itself, so LXD as a piece of software does none of the above. The package maintainers decided to add some sane defaults to get newcomers up and running fast.

So in short, I believe this is a discussion for generic networking and should not be handled as a LXD issue, but that's just me :)

[srkunze]

@jsimonetti You are completely right; it's generic networking what we are discussing here.

This said, I still got the feeling it's LXD's problem (even though it's not its responsibility).

The solution provided by the Ubuntu packages appears to be a first (and dirty) step in that direction. Maybe, there are other tools (LXD-plugin/extension) that can be installed alongside with LXD to make that step more explicit and reversible (e.g. by not installing the extension package).

I think what is clear from this discussion is that networking

• is a central piece of LXD containers (management)
• needs an out-of-the-box experience

I am not an expert on this, so I leave the solution to @stgraber, @jsimonetti and @hallyn.

[srkunze]

Making setup easy and configurable, one could imagine the following package structure:

 - lxd-core        # that is what the experts would use
 - lxd-networking  # that would give easy tools to newbies
 - lxd             # lxd-core + lxd-networking

The implementation of #1363 might be a good candidate for lxd-networking.

[techtonik]

Wow. Adding route from P to H made my C accessible. This is what I had to do on Vista (still need admin rights):

route ADD 10.0.3.0 MASK 255.255.255.0 192.168.1.15

[srkunze]

@stgraber Can this be closed? Not sure if you really consider the idea outlined in https://github.com/lxc/lxd/issues/1294#issuecomment-160156938 . If not, I close this as I am satisfied with the routing solution.

[stgraber]

Yeah, let's close that for now.

[michacassola]

I spent all of yesterday to try to find an easy and up to date setup instruction for networking. I could not find one that worked for me.
I have one IPv4 that I want all my containers to be reachable under.

(I use lxd containers only to separate my users resources, then I want to install VestaCP on them and container1 should be reachable under c1.domain.com, container 2 under c2.domain.com and so on but all under one IPv4; the simplest solution would be preferred)

I can not figure out how to do that!
@stgraber Maybe a post about containers for hosting would be nice please?
Also a general official up to date documentation would be nice! :)

[stgraber]

@michacassola what you're describing can only be done with a reverse proxy service and protocols that include the hostname as part of the request, so pretty much just HTTP.

For HTTP you'd probably want to setup the proxy on the host. It can be done with haproxy, nginx, apache2, squid, ... That proxy would bind the HTTP (and probably HTTPs) ports for your public IP and be configured to forward traffic for http://c1.domain.com to http://[IP of container c1] and https://c1.domain.com to https://[IP of container c1].

As I said, this will only really work for http and https. If your users need SSH or FTP access, you'll need to get a public IP per container or setup some NAT rules to redirect unique ports on your public IP to specific containers.

[michacassola]

@stgraber Thank you for that info!
Do you have a link of where I can find how to set up a reverse proxy for lxd containers?
Do you have a link of where I can find set up instructions for NAT rules for lxd containers for unique redirecting?
Will there be an up to date comprehensible official documentation or did I just not see it?

[stgraber]

Neither of those are LXD specific, you'd get the exact same problem if you were using virtual machines or physical system behind a single public IP, so it's not something we're likely to include in LXD's own documentation.

For haproxy, I'd expect it to be something like this:
https://unix.stackexchange.com/questions/166169/how-to-configure-haproxy-to-redirect-multiple-domain

For iptables, it'd just be something like:

iptables -t nat -A PREROUTING -d PUBLIC-IP -p tcp --dport SOME-PORT -j DNAT --to CONTAINER-IP:22