lxd networking documentation

[pgassmann]

I would greatly appreciate some writeup/documentation on the networking topics around lxd. I'd like to use lxd as a faster alternative to Virtualbox for testing and development and also for production, but I stumbled already on various issues concerning networking.

I know that it is not directly part of the lxd project, it used some default linux networking mechanisms. But from a user's point of view it is part of the experience.

Some topics I'd like to know more about in regard to lxd:

• default lxdbridge, custom bridge
• dhcp, hostname & ip assignment
• how templates work for /etc/hosts /etc/hostname /network-scripts/ etc.
• veth interfaces
• firewall rules that are managed by lxd?
• iptables (Host kernel requirements & limits, kernel fs permissions, features, what it means for different distros in container)
• NAT/Bridging basics
• Making use of ipv6 to access containers directly

[jeromebenois]

+1

[niccolox]

+1

[CalebEverett]

Here is one approach from a novice perspective:

Networking

• Use the default lxdbridge, but also specify an extra dnsmasq.conf file at LXD_CONFILE in /etc/default/lxd-bridge
• Set the lxc config to listen on port 8443 with lxc config set core.https_address [::]:8443
• You can then assign specific ip addresses to containers by their name in dnsmasq.conf like
  dhcp-host=mycontainer,10.19.248.50
• Restart the lxd-bridge service with sudo systemctl restart lxd-bridge to have the changes picked up
• One thing that made it easier as I was getting this going was to make sure to stop containers before you delete them. If you force delete them without stopping first, the ip address isn't released and therefore isn't available to be assigned to a new container
• Then you can enter whatever host name you'd like for a container's ip address in /etc/hosts like 10.19.248.50 mycontainer.dev to have a user friendly way of getting to the container
• I used ufw to open up a port inside the container with:

ufw enable
ufw allow 8080/tcp
ufw reload
ufw status

To                         Action      From
--                         ------      ----
8080/tcp                   ALLOW       Anywhere                  
8080/tcp (v6)              ALLOW       Anywhere (v6)   

• To avoid having to add :8080 to the end of the url, you can forward port 80 to the port that you opened by adding the following to /etc/ufw/before.rules inside the container:

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
COMMIT

• If you want to allow access to a container from only another container, this is the ufw command:

ufw allow from <ip-address-of-container-to-allow-traffic-from> to any port <port-number>

Shared folder

• To be able to edit code inside the container from the host, you add a device with:

lxc config device add [remote:]container1 <device-name> disk source=/share/c1 path=opt

• Make sure your files have the right permissions on the host and inside the container (more info here). I ended up making a user inside the container the owner of the files (so no issues accessing the files inside the container) and adding an entry to the file access control list to allow my user access to the files on the host.

sudo setfacl -R -m d:u:<user-name>:xwr,u:<user-name>:xwr,d:g:<user-name>:xwr,g:<user-name>:xwr <path-to-shared-folder>

Environment variables

• Profiles seemed like a great place to put environment variables for development:

lxc profile create <profile-name>
lxc profile edit <proflile-name>

name: studentfiles-dev
config:
  environment.NODE_ENV: development
  environment.PORT: "8080"
description: ""
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic

lxc profile apply <container-name> <profile-name>

• If you are starting your process from outside the container, make sure to include the HOME environment variable for the working directory of your app:

lxc exec <container-name> --env HOME=<path-to-working-directory> -- <command-to-start-process>

• For production, I created a systemd service unit, which starts your app when the container starts and restarts it if it crashes. If you do that, the service doesn't pick up the environment variables in the profile and you have to set them in the service file.

Templates

• There is probably a more efficient way to do this using lxd/lxc functionality and/or general pre-seeding, but one approach to creating containers is just to create a script to do your configuration inside the container using:

lxc exec <container> -- <command>

• This is probably pretty mickey-mouse, but here is a copy of a script I've been using to launch containers for developing node apps. I put this in a new folder and run it and it creates a container, installs node and git, clones code from a repository into a shared folder and makes the container accessible at the container ip address from the host. (It doesn't modify your dnsmasq.conf or /etc/hosts files, but will open up chrome pointing to the app if you're on linux and have chrome installed.)

Deployment

• Deployment is super simple with the ability to connect to an instance of lxd on another machine with:

lxc config set core.trust_password blah
lxc remote add <name> <url>

on both machines and then copy containers from your development machine to your public server with:

lxc copy [remote:]<source container> [remote:]<destination container>

• If you make sure that the entries in /etc/subgid and /etc/subuid are the same on both machines you won't have to worry about shifting subordinate user and group ids.
• I also copied in the /etc/default/lxd-bridge and extra dnsmasq.conf files from my development machine so that any ufw rules created in the containers on the development machine work. Apparently ufw doesn't work with host names.
• I created a container running an nginx reverse proxy on the server that traffic to the containers runs through and used ufw on the server to open up ports and forward traffic to the nginx container.
• Deploying new containers is as easy as copying the container onto the server and then using lxc exec to get into the nginx container to add the new container as a server.

[niccolox]

excellent, looks like its the community contrib that will save the day !!

[niccolox]

btw, you should make a branch with these docs added alongside the README and get credit for such a nice writeup

I think I solved my own lxd networking and file mounting issues, with some help from @antitoine and @stgraber and others but it took way too long with lots of trial and error and search

this kind of docs could form the basis of some really decent acceptance of a cool technology

big ups

[CalebEverett]

I think these guys are frying bigger fish than individual developer environments ;). It took some time to get going (mainly due to lack of experience) - but what these guys have created is awesome and seems great for smaller scale projects too. Happy to help out if I can.

[niccolox]

submit a PR

[niccolox]

maybe DEVELOPER.md or something

[mikesteele81]

I was looking around for information about how the network bridging automation works, and found this blog post: http://tycho.ws/blog/2016/04/lxdbr0.html

[naisanza]

lxc profile device set default eth0 parent containerbr from @mikesteele81 's link was what I was looking for. LXD's documentation for it really needs more work

Update: Link to documentation https://github.com/lxc/lxd/blob/master/doc/configuration.md
Update2: proper syntax is lxc profile device add <profile-name> eth0 nic name=eth0 nictype=bridged parent=br0

[jradxl]

+1

[emprovements]

+1

[jfgibbins]

For networking, I've found the easiest and most flexible way is to utilize openvswitch on the host. The bridges show up as networks in 'lxc network list'. Utilizing the cloud enabled versions from images.linuxcontainers.org, I use the following example in my lxc configs. No need for lxdbr0, dnsmasq or anything else setup specifically for lxc. Just handle everything networking, dns, dhcp, static, dhcp, firewall, etc, as you would any other bare-metal, virtualized, cloud device. Cloud-init configs are documented under https://github.com/lxc/lxd/blob/master/doc/cloud-init.md

eth0:
host_name: docker
hwaddr: 0015:0A08:000B
nictype: bridged
parent: patch
type: nic
user.network-config: |
version: 1
config:
- type: physical
name: eth0
- type: nameserver
address: ['10.255.0.29']
- type: vlan
name: eth0.1000
vlan_link: eth0
vlan_id: 1000
subnets:
- address: 10.16.0.11/12
control: auto
type: static
- type: vlan
name: eth0.1200
vlan_link: eth0
vlan_id: 1200
subnets:
- address: 10.8.0.11/24
gateway: 10.8.0.254
control: auto
type: static

[jfgibbins]

@stgraber @brauner hint for OVN wink, you're integration with OVS is awesome, simple and easy, and makes containers indistinguishable from a physical server, OVN provider and you've got SDN for linux containers as good as any other solution out there, including ACI and NSX

[sequoiar]

+1

[phamti]

I've been implementing LXC and LXD and have been doing a writeup http://www.bonsaiframework.com/wiki/display/bonsai/5.1+LXC+Advanced+Networking+-+Exposing+Containers+to+the+Network

[bmullan]

@phamti - fyi but that link you posted for bonsaiframwork.com... fails

[phamti]

Thanks, the server was down for an upgrade. It should be back up now.

Sent from my iPhone

On Nov 7, 2017, at 9:18 PM, bmullan [e-mail] wrote:

@phamti - fyi but that link you posted for bonsaiframwork.com... fails

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[stgraber]

Closing as there is nothing clearly actionable on this and workflow discussions are better handled by our forum (https://discuss.linuxcontainers.org).