Lxd: Document macvlan scenario

[techtonik]

It would be extremely handy to document macvlan scenario for accessing containers from the LAN. Without knowing about macvlan it is hard to discover it, and without knowing about MAC and NIC concepts it is hard to configure.

This is default LXD setup.

$ lxc profile show default
config: {}
description: ""
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/containers/rpm
- /1.0/containers/lxdtox
- /1.0/containers/lxdtox/init

• [ ] switch profile to macvlan mode (and document implications)
  
  
  
  • [ ] change default profile
  
  
  • [ ] create separate profile for LAN containers
  
  

Explain why this works (https://stgraber.org/2016/10/27/network-management-with-lxd-2-3/#comment-241550):

lxc network attach-profile eth0 default eth0

SVG diagrams would greatly help.

[stgraber]

Yeah, we probably need a bridge vs macvlan section in network.md to explain the two different modes and their limitations (well, mostly macvlan's limitations).

[techtonik]

Creating a separate webdev profile with macvlan mode:

$ lxc profile copy default webdev
$ lxc profile device set webdev eth0 nictype macvlan
$ lxc profile device set webdev eth0 parent eth0
$ lxc profile assign rpm webdev
$ lxc profile show webdev
config: {}
description: ""
devices:
  eth0:
    nictype: macvlan
    parent: eth0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: webdev
used_by:
- /1.0/containers/rpm

[techtonik]

Wow. It works. =)

[techtonik]

[eaojnr]

Macvlan solution seems easy but implementation is not consistent across ubuntu variants i think. I've tried below and seem not to be working, is there something i'm doing wrong? Anyone able to share info? I am trying to use macvlan on a new profile so i can assign public IPs myself.

root@hello:/var/lib# uname -a
Linux hello 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@hello:/var/lib# lxc --version
4.1
root@hello:/var/lib# lxc profile list
+---------------+---------+
| NAME | USED BY |
+---------------+---------+
| default | 1 |
+---------------+---------+
| mcvlanprofile | 1 |
+---------------+---------+
root@hello:/var/lib# lxc profile device set mcvlanprofile eth0 nictype macvlan
Error: Device validation failed "eth0": Cannot use "nictype" property in conjunction with "network" property
root@hello:/var/lib# lxc profile show mcvlandefault
Error: Fetch profile: No such object
root@hello:/var/lib# lxc profile show mcvlanprofile
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: mcvlanprofile
type: disk
name: mcvlanprofile
used_by:

• /1.0/instances/lanVM
  root@hello:/var/lib#

[derekmahar]

root@hello:/var/lib# lxc profile device set mcvlanprofile eth0 nictype macvlan
Error: Device validation failed "eth0": Cannot use "nictype" property in conjunction with "network" property
root@hello:/var/lib# lxc profile show mcvlanprofile
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: mcvlanprofile
type: disk
name: mcvlanprofile
used_by:
* /1.0/instances/lanVM

@eaojnr, I think you have a syntax error in your macvlan profile specification, macvlanprofile. My understanding is that LXD associates a macvlan interface to a physical parent interface, not a bridge, so you need to replace network: lxdbr0 with parent: your_host_nic_name, and add nictype: macvlan. For example:

$ lxc profile show macvlan
config:
  environment.TZ: America/Montreal
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: enp5s0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: macvlan
used_by:
- /1.0/instances/ubuntu-focal-ssh
- /1.0/instances/development1

[eaojnr]

Hi derekmahar,

thanks so much for taking the time to help clarify. I have fixed the typo and tried. Below is the current status:

root@hello:~# lxc info --show-log lanVM
Name: lanVM
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/05/23 09:04 UTC
Status: Stopped
Type: container
Profiles: mcvlanprofile

Log:

lxc lanVM 20200523090451.308 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.lanVM"
lxc lanVM 20200523090451.308 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.lanVM"
lxc lanVM 20200523090451.309 ERROR utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc lanVM 20200523135744.382 ERROR commands - commands.c:lxc_cmd_get_init_pidfd_callback:441 - Failed to send init pidfd
lxc lanVM 20200523135746.830 WARN network - network.c:lxc_restore_phys_nics_to_netns:3347 - No interface corresponding to ifindex 9
lxc lanVM 20200523135746.842 WARN network - network.c:lxc_delete_network_priv:3213 - Failed to rename interface with index 0 from "eth0" to its initial name "vethb3a5d856"

root@hello:~# lxc info --show-log lanV
Error: not found
root@hello:~# lxc profile show mcvlanprofile
config: {}
description: Default LXD profile
devices:
eth0:
nictype: macvlan
parent: eno0
type: nic
root:
path: /
pool: mcvlanprofile
type: disk
name: mcvlanprofile
used_by:

• /1.0/instances/lanVM
  root@hello:~# lxc profile edit mcvlanprofile
  root@hello:~# lxc list
  +-------+---------+---------------------+-----------------------------------------------+-----------+-----------+
  | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
  +-------+---------+---------------------+-----------------------------------------------+-----------+-----------+
  | lanVM | STOPPED | | | CONTAINER | 0 |
  +-------+---------+---------------------+-----------------------------------------------+-----------+-----------+
  | omeet | RUNNING | 10.190.21.42 (eth0) | fd22:1188:3245:b9db:216:3eff:fea5:692d (eth0) | CONTAINER | 0 |
  +-------+---------+---------------------+-----------------------------------------------+-----------+-----------+
  root@hello:~# lxc start lanVM
  Error: Common start logic: Failed to start device "eth0": Parent device 'eno0' doesn't exist
  Try lxc info --show-log lanVM for more info
  root@hello:~# ip addr
  1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
  inet6 ::1/128 scope host
  valid_lft forever preferred_lft forever
  2: eno1: mtu 1500 qdisc mq state UP group default qlen 1000
  link/ether XXXXXX brd ff:ff:ff:ff:ff:ff
  inet XXXX/26 brd YYYYY scope global eno1
  valid_lft forever preferred_lft forever
  inet BBBBBB/26 brd YYYYY scope global secondary eno1
  valid_lft forever preferred_lft forever
  inet6 AYGHSERERE/64 scope global
  valid_lft forever preferred_lft forever
  inet6 ESDFSRTRHD/64 scope link
  valid_lft forever preferred_lft forever
  3: eno2: mtu 9000 qdisc mq state UP group default qlen 1000
  link/ether 6c:2b:59:b8:4d:14 brd ff:ff:ff:ff:ff:ff
  inet6 fe80::6e2b:59ff:feb8:4d14/64 scope link
  valid_lft forever preferred_lft forever
  4: lxdbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000
  link/ether b2:5e:85:e8:6a:63 brd ff:ff:ff:ff:ff:ff
  inet 10.190.21.1/24 scope global lxdbr0
  valid_lft forever preferred_lft forever
  inet6 fd42:15ab:6d28:b9db::1/64 scope global
  valid_lft forever preferred_lft forever
  inet6 fe80::7c17:efff:fe14:6be7/64 scope link
  valid_lft forever preferred_lft forever
  8: vethc2448063@if7: mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
  link/ether b2:5e:85:e8:6a:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0

[derekmahar]

root@hello:~# lxc profile show mcvlanprofile
config: {}
description: Default LXD profile
devices:
eth0:
nictype: macvlan
parent: eno0
type: nic
root:
path: /
pool: mcvlanprofile
type: disk
name: mcvlanprofile
used_by:
* /1.0/instances/lanVM
root@hello:~# lxc start lanVM
Error: Common start logic: Failed to start device "eth0": Parent device 'eno0' doesn't exist
Try lxc info --show-log lanVM for more info
root@hello:~# ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether XXXXXX brd ff:ff:ff:ff:ff:ff
inet XXXX/26 brd YYYYY scope global eno1
valid_lft forever preferred_lft forever
inet BBBBBB/26 brd YYYYY scope global secondary eno1
valid_lft forever preferred_lft forever
inet6 AYGHSERERE/64 scope global
valid_lft forever preferred_lft forever
inet6 ESDFSRTRHD/64 scope link
valid_lft forever preferred_lft forever
3: eno2: mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 6c:2b:59:b8:4d:14 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6e2b:59ff:feb8:4d14/64 scope link
valid_lft forever preferred_lft forever
4: lxdbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:5e:85:e8:6a:63 brd ff:ff:ff:ff:ff:ff
inet 10.190.21.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:15ab:6d28:b9db::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::7c17:efff:fe14:6be7/64 scope link
valid_lft forever preferred_lft forever
8: vethc2448063@if7: mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether b2:5e:85:e8:6a:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0

@eaojnr , in profile mcvlanprofile, rename parent interface from eno0 to eno1 or eno2. I don't see eno0 in your list of host network interfaces.

[eaojnr]

Hi Derek, Thanks for that catch. I'll try to edit and revert if i'm having any more challenges.

[eaojnr]

HI @derekmahar any chance to help guide me on assigning static IP's to an LXC with a macvlan profile?

I tried within the container, by going to the /etc/netplan/50-cloud-init.yaml but when i applied it, ip addr doesn't reflect the assignment.

Guidance appreciated.

[derekmahar]

@eaojnr, did you apply the static IP address example in _Using DHCP and static addressing_ to the container? I may be mistaken, but I believe you need _not_ specify the macvlan interface type in the configuration of the container network interface. If I'm mistaken and you must specify the macvlan interface type, then you must do so using ifupdown because Netplan still does not yet support the macvlan interface.

[tomponline]

The error Error: Device validation failed "eth0": Cannot use "nictype" property in conjunction with "network" property is caused when trying to modify an existing NIC device (either defined in a profile applied to a an instance or directly on an instance). A few releases back a new property called network was added that is used in the default profile setup by LXD. It indicates to the NIC where to get several of its settings from (including nictype and parent) and so those cannot also be defined at the same time as specifying the network property.

So the network property needs to be removed for macvlan type NICs and replaced with nictype=macvlan and parent=<parent name> properties.

[eaojnr]

@tomponline thanks for the additional hint.

@derekmahar thanks a bunch. That was more clearer than existing docs. Perhaps the way you put it makes it easy to understand and apply.
Now I've got the interface up with an IP and a gateway set but I cannot ping the gateway which i find weird.
❯ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 15: eth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:3f:c3:8b brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.1.15/24 brd 10.0.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe3f:c38b/64 scope link valid_lft forever preferred_lft forever

❯ ip r default via 10.0.1.1 dev eth0 onlink 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.15

❯ ping 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. From 10.0.1.15 icmp_seq=1 Destination Host Unreachable From 10.0.1.15 icmp_seq=2 Destination Host Unreachable

`manny@mannyelite:~$ lxc profile show macvlan
config: {}
description: Default LXD profile
devices:
eth0:
nictype: macvlan
parent: wlo1
type: nic
root:
path: /
pool: default
type: disk
name: macvlan
used_by:

• /1.0/instances/rnd1
  `

Any advice on where I've missed it? Do I need to enable IP forwarding or routing categorically?

thanks for the time and effort.

[EmanNo1]

I am getting the same error "Error: Device validation failed "eth0": Cannot use "nictype" property in conjunction with "network" property". From my understanding, I need to remove the 'network' key and add 'nictype' and parent keys to the profile under eth0. I am probably issuing the commands in the incorrect order or with errors.

$ lxc profile show lanprofile
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: lanprofile
used_by: []

$ lxc profile device unset lanprofile eth0 network
Error: Device validation failed "eth0": Unsupported device type

$ lxc profile device set lanprofile eth0 parent enp2s0
Error: Device validation failed "eth0": Cannot use "parent" property in conjunction with "network" property

$ lxc profile device set lanprofile eth0 nictype macvlan
Error: Device validation failed "eth0": Cannot use "nictype" property in conjunction with "network" property

[tomponline]

@EmanNo1 the way I would suggest the way to do it is to remove the old bridged eth0 device and add a new macvlan device in its place.

Here are the steps, I've copied the default profile in my case to a new profile called macvlan.

lxc init ubuntu:20.04 c1

lxc profile copy default macvlan
lxc profile device remove macvlan eth0
lxc profile device add macvlan eth0 nic nictype=macvlan parent=enp3s0

lxc profile add c1 default_macvlan
lxc config show c1 --expanded
...
devices:
  eth0:
    nictype: macvlan
    parent: enp3s0
    type: nic
...
lxc ls c1
+------+---------+----------------------+-------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                   IPV6                    |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-------------------------------------------+-----------+-----------+
| c1   | RUNNING | 192.168.1.199 (eth0) |                                           | CONTAINER | 0         |
+------+---------+----------------------+-------------------------------------------+-----------+-----------+