Lxd: Improve support for OVS based nic devices

[jfgibbins]

Currently using lxd with openvswitch, and for the most part is a piece of cake. Nice job! However, as far as I can tell, I only have two options. If parent is an access port, ie ovs-vsctl add-br mainbridge subbridge 400, the nic gets an untagged vlan, as expected, as conversely if parent is a trunk, ie ovs-vsctl add-br bridge, I get an trunk with all vlans. I would like to be able to get only certain vlans to the container. I created trunk with ovs-vsctl add-br bridge and set trunks=1000,2000, but when the container uses it as a parent, it doesn't inherit it's parents paremeters, and I instead get a full trunk with all vlans, 0-4095. Possibly one of the following, or maybe something I haven't thought of:
allow lxd to set additional paremeters, ie trunks, vlan, etc similiar to would be done with ovs-vsctl
inherit parents parameters
or very cool, create for lxd a connector to allow use of something like OVN, openvswitch's openflow controller

And again, great job guys, It's an absolute blast playing with LXD. For virtualizing linux, it's beats the hell out of other things like vmware, etc
Current environment is all on lxd, with exception of a couple freebsd boxes for FWs running on qemu/kvm, and blah, windows 10 on vmware workstation to run a couple jump boxes for windows only apps, damn users and their need for a windows gui

[stgraber]

Sounds like we've been hitting a number of limitations related to liblxc's somewhat limited OVS support.
It may be time for me to look at doing the OVS integration directly in LXD itself and just tell LXC what device to use in such case.

[jfgibbins]

Sounds great. I've been using LXD and OVS together quite a bit, and exploring ways to expand it. Needless to say I've bounced around the limitations a few different ways. Hit me up if you'd like any ideas, and especially if you want some testing around ovs.

[brauner]

Could also be nice to have liblxc itself support this in a little more depth. The code is already there.

[llitz]

I hope this is relevant but this is what I have been doing:

• Create new patch interface on "central/target" bridge
• Put specific tag information on Cental/Target port
• Create a new internal only bridge
• Create new patch interface on new bridge
• Connect both patch ports using the peer option

While this works quite nicely, it could be easier if we were able to specify extra information to the ovs-vsctl add-port (as it is right now, I don't run systemd and my ports are never automatically created as tap ports, quite annoying).

I don't know how far we can go with it (not that familiar with the design of LXC/LXD) but it would be nice if we were able to use extra commands from ovs, like

• "set Interface PORT type=tap"
• "set port PORT tag=5"
• "set port PORT trunks="1 ,2, 5"

If all this is passed at the time we create the interface "ovs-vsctl add-port BRIDGE PORT -- set Interface PORT type=tap -- set port PORT tag=VLANID" we don't need to use a custom script (like I've seen done with lxc a couple times) and openvswitch would handle it cleanly.

[jfgibbins]

I do similar in regards to creating bridges to attach containers on to, and then placing the tags on the patch port. But inherently limited to functions by bridge. Would love to be able to utilize the OVN and take advantage of the entire SDN/micro-segmentation concept. <-new buzzword for marketing to resell what we've been doing in networking already for 20+ years.

[jfgibbins]

@stgraber @brauner Not expecting you have any code written, but was curious any thoughts or ideas you might have brainstormed for this.

[stgraber]

No clear plans yet. I need to experiment a bit more with openvswitch, then decide what we want to do in LXD and what we should do in lower level liblxc. OVN is also pretty interesting but I need to figure out how to have it fit in our existing networking model and our future cross-host story.

[jfgibbins]

Wish I could help more in terms of how it fits with your current networking, but I have to admit, I haven't used any of it. Everything I do networking wise, is simply to attach the container to an openvswitch bridge, either as a switchport or trunk. Static is set through cloud-init, and dns/dhcp is all done through traditional bind/dhcpd servers on the network.

[jfgibbins]

@stgraber While written for OVS and OpenStack, it makes an interesting read. Might be useful for picking some ideas for enabling LXC/LXD to work with OVS/OVN.
https://github.com/yeasy/easyOVS

[gstanden]

Orabuntu-LXC may also be useful for picking some ideas for enabling LXC/LXD to work with OVS/OVN.
https://github.com/gstanden/orabuntu-lxc

[Jean-Daniel]

Just to weigh-in. One feature I really miss is the support of the vlan property when attaching a OVS backed bridge.

That property is already used by macvlan and physical interface. While supporting it for all kind of bridge may not be possible, at least, it should be pretty straightforward to support it when this is a OVS bridge, as the only requirement would be to tag the port when it is created.

This will allow to use lxd with OVS without requiring to create a sub-bridge for each VLAN we want to use in a VM. (see issue #3414)

[stgraber]

It's unfortunately harder than it seems. Right now we rely on liblxc to detect OVS and connect the veth to the bridge. liblxc config doesn't know about vlans on OVS bridges so we can't currently pass that through.

We do have our own matching logic for this, but it's only used when injecting a network interface after the container has already started. In that case, it would indeed be easy to use our existing vlan property and pass it through, but it doesn't make sense to do so if we can't also have it work during normal container startup.

I suspect the answer will be to special case OVS and have LXD pre-create the veth and then pass it through liblxc as a pre-existing device, but that'll need quite a bit more code shuffling than just adding a property...

[jokogr]

Is it possible to address this in liblxc and then for lxd to handle it properly?

[stgraber]

@jokogr Yes, if someone has interest in adding the needed properties to liblxc, that'd be a great start for this. We can then add support for it in LXD when we detect a suitable liblxc version. This won't be a particularly quick fix though as we don't release liblxc quite as often as lxd and most distributions will choose to stick to the LTS release of liblxc even once we do so.

[stgraber]

I believe we now have this. LXD has improved OVS device support (properly releases the ports) and supports setting the VLAN ID for the untagged VLAN and additional tagged VLANs.