Lxd: After 1000 container - the new opened containers don't receive local IP

[miqayel-manvelyan]

Required information

• The output of "lxc info" or if that fails:

config:
  core.https_address: '[::]'
  core.trust_password: true
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 148.251.31.171:8443
  - 148.251.43.154:8443
  - 172.16.0.1:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    .....
    -----END CERTIFICATE-----
  certificate_fingerprint: ebcd1e102c659da53317a7a0744ceec7c688fd2fa05855cf2057b847466b4b63
  driver: lxc
  driver_version: 3.0.3
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.15.0-72-generic
  server: lxd
  server_pid: 20409
  server_version: 3.0.3
  storage: btrfs
  storage_version: 4.15.1
  server_clustered: false
  server_name: server
  project: ""

• Kernel version: Linux server 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• LXC version: 3.0.3
• LXD version: 3.0.3
• Storage backend in use: btrfs

Issue description

After reaching 1000+ containers - the new ones don't receive local IP

lxc network show lxdbr0

config:
  ipv4.address: 172.16.0.1/20
  ipv4.nat: "true"
description: ""
name: lxdbr0
type: bridge

Information to attach

• [x] Any relevant kernel output (dmesg)
• [x] Container log (lxc info NAME --show-log)
• [x] Container configuration (lxc config show NAME --expanded)
• [x] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

dmesg

[396576.850745] audit: type=1400 audit(1576676157.987:3883): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=28691 comm="apparmor_parser"
[396576.850766] audit: type=1400 audit(1576676157.987:3884): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=28691 comm="apparmor_parser"
[396576.850786] audit: type=1400 audit(1576676157.987:3885): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=28691 comm="apparmor_parser"
[396576.851680] audit: type=1400 audit(1576676157.991:3886): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/sbin/tcpdump" pid=28702 comm="apparmor_parser"
[396576.851734] audit: type=1400 audit(1576676157.991:3887): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine" pid=28696 comm="apparmor_parser"
[396576.851753] audit: type=1400 audit(1576676157.991:3888): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=28696 comm="apparmor_parser"
[396576.852221] audit: type=1400 audit(1576676157.991:3889): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=28687 comm="apparmor_parser"
[396664.981371] kauditd_printk_skb: 8 callbacks suppressed
[396664.981372] audit: type=1400 audit(1576676246.120:3898): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-yogurtcon--yogurtcon--2118_</var/lib/lxd>" pid=30890 comm="apparmor_parser"
[396665.014259] lxdbr0: port 965(veth3IB69K) entered disabled state
[396665.018838] device veth3IB69K left promiscuous mode
[396665.018841] lxdbr0: port 965(veth3IB69K) entered disabled state
[396670.940401] audit: type=1400 audit(1576676252.080:3899): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-murisceman--test12341212--2040_</var/lib/lxd>" pid=31583 comm="apparmor_parser"
[396670.986186] lxdbr0: port 320(vethNW7QHB) entered disabled state
[396670.991193] device vethNW7QHB left promiscuous mode
[396670.991199] lxdbr0: port 320(vethNW7QHB) entered disabled state
[396671.786402] audit: type=1400 audit(1576676252.924:3900): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=31677 comm="apparmor_parser"
[396671.854348] lxdbr0: port 703(veth4BN0LC) entered disabled state
[396671.859519] device veth4BN0LC left promiscuous mode
[396671.859524] lxdbr0: port 703(veth4BN0LC) entered disabled state
[396673.082258] audit: type=1400 audit(1576676254.220:3901): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=31722 comm="apparmor_parser"
[396673.124181] lxdbr0: port 320(vethU6L9BL) entered blocking state
[396673.124183] lxdbr0: port 320(vethU6L9BL) entered disabled state
[396673.124924] device vethU6L9BL entered promiscuous mode
[396673.125982] IPv6: ADDRCONF(NETDEV_UP): vethU6L9BL: link is not ready
[396673.273768] eth0: renamed from veth0VJ5SW
[396673.311742] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396673.313292] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396673.313426] IPv6: ADDRCONF(NETDEV_CHANGE): vethU6L9BL: link becomes ready
[396673.313869] lxdbr0: port 320(vethU6L9BL) entered blocking state
[396673.313870] lxdbr0: port 320(vethU6L9BL) entered forwarding state
[396673.636929] audit: type=1400 audit(1576676254.776:3902): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=32112 comm="apparmor_parser"
[396673.636948] audit: type=1400 audit(1576676254.776:3903): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=32112 comm="apparmor_parser"
[396673.636981] audit: type=1400 audit(1576676254.776:3904): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=32112 comm="apparmor_parser"
[396673.637588] audit: type=1400 audit(1576676254.776:3905): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=32110 comm="apparmor_parser"
[396673.637607] audit: type=1400 audit(1576676254.776:3906): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=32110 comm="apparmor_parser"
[396673.637626] audit: type=1400 audit(1576676254.776:3907): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=32110 comm="apparmor_parser"
[396673.637645] audit: type=1400 audit(1576676254.776:3908): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=32110 comm="apparmor_parser"
[396678.732886] kauditd_printk_skb: 9 callbacks suppressed
[396678.732887] audit: type=1400 audit(1576676259.872:3918): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=32398 comm="apparmor_parser"
[396678.746047] lxdbr0: port 320(vethU6L9BL) entered disabled state
[396678.750619] device vethU6L9BL left promiscuous mode
[396678.750622] lxdbr0: port 320(vethU6L9BL) entered disabled state
[396679.840248] audit: type=1400 audit(1576676260.980:3919): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=32444 comm="apparmor_parser"
[396679.878074] lxdbr0: port 320(veth0VBM67) entered blocking state
[396679.878075] lxdbr0: port 320(veth0VBM67) entered disabled state
[396679.878803] device veth0VBM67 entered promiscuous mode
[396679.880169] IPv6: ADDRCONF(NETDEV_UP): veth0VBM67: link is not ready
[396680.058781] eth0: renamed from vethXD9QVQ
[396680.080104] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396680.082426] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396680.082601] IPv6: ADDRCONF(NETDEV_CHANGE): veth0VBM67: link becomes ready
[396680.083330] lxdbr0: port 320(veth0VBM67) entered blocking state
[396680.083332] lxdbr0: port 320(veth0VBM67) entered forwarding state
[396680.399464] audit: type=1400 audit(1576676261.540:3920): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=395 comm="apparmor_parser"
[396680.399481] audit: type=1400 audit(1576676261.540:3921): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=395 comm="apparmor_parser"
[396680.399498] audit: type=1400 audit(1576676261.540:3922): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=395 comm="apparmor_parser"
[396680.400133] audit: type=1400 audit(1576676261.540:3923): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine" pid=400 comm="apparmor_parser"
[396680.400149] audit: type=1400 audit(1576676261.540:3924): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=400 comm="apparmor_parser"
[396680.400164] audit: type=1400 audit(1576676261.540:3925): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/lxc-start" pid=394 comm="apparmor_parser"
[396680.400363] audit: type=1400 audit(1576676261.540:3926): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=393 comm="apparmor_parser"
[396680.400378] audit: type=1400 audit(1576676261.540:3927): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=393 comm="apparmor_parser"
[396703.132410] kauditd_printk_skb: 8 callbacks suppressed
[396703.132411] audit: type=1400 audit(1576676284.272:3936): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=1563 comm="apparmor_parser"
[396703.145465] lxdbr0: port 320(veth0VBM67) entered disabled state
[396703.150355] device veth0VBM67 left promiscuous mode
[396703.150359] lxdbr0: port 320(veth0VBM67) entered disabled state
[396704.301695] audit: type=1400 audit(1576676285.440:3937): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=1595 comm="apparmor_parser"
[396704.343508] lxdbr0: port 320(veth7YSSGA) entered blocking state
[396704.343509] lxdbr0: port 320(veth7YSSGA) entered disabled state
[396704.344208] device veth7YSSGA entered promiscuous mode
[396704.345347] IPv6: ADDRCONF(NETDEV_UP): veth7YSSGA: link is not ready
[396704.514126] eth0: renamed from veth0G5HJX
[396704.555899] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396704.557946] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396704.558118] IPv6: ADDRCONF(NETDEV_CHANGE): veth7YSSGA: link becomes ready
[396704.558599] lxdbr0: port 320(veth7YSSGA) entered blocking state
[396704.558601] lxdbr0: port 320(veth7YSSGA) entered forwarding state
[396704.837001] audit: type=1400 audit(1576676285.976:3938): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=1933 comm="apparmor_parser"
[396704.837021] audit: type=1400 audit(1576676285.976:3939): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=1933 comm="apparmor_parser"
[396704.837039] audit: type=1400 audit(1576676285.976:3940): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=1933 comm="apparmor_parser"
[396704.837148] audit: type=1400 audit(1576676285.976:3941): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=1931 comm="apparmor_parser"
[396704.837167] audit: type=1400 audit(1576676285.976:3942): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=1931 comm="apparmor_parser"
[396704.837184] audit: type=1400 audit(1576676285.976:3943): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=1931 comm="apparmor_parser"
[396704.837201] audit: type=1400 audit(1576676285.976:3944): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=1931 comm="apparmor_parser"
[396704.837334] audit: type=1400 audit(1576676285.976:3945): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine" pid=1934 comm="apparmor_parser"
[396714.163235] kauditd_printk_skb: 8 callbacks suppressed
[396714.163237] audit: type=1400 audit(1576676295.304:3954): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-mariogomesbarbosa--delicious--2762_</var/lib/lxd>" pid=2137 comm="apparmor_parser"
[396714.209345] lxdbr0: port 549(vethI8SLEU) entered disabled state
[396714.213568] device vethI8SLEU left promiscuous mode
[396714.213572] lxdbr0: port 549(vethI8SLEU) entered disabled state
[396721.025619] audit: type=1400 audit(1576676302.164:3955): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=3077 comm="apparmor_parser"
[396721.045712] lxdbr0: port 320(veth7YSSGA) entered disabled state
[396721.050322] device veth7YSSGA left promiscuous mode
[396721.050325] lxdbr0: port 320(veth7YSSGA) entered disabled state
[396722.123514] audit: type=1400 audit(1576676303.264:3956): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=3122 comm="apparmor_parser"
[396722.161733] lxdbr0: port 320(veth2IBOFT) entered blocking state
[396722.161735] lxdbr0: port 320(veth2IBOFT) entered disabled state
[396722.162463] device veth2IBOFT entered promiscuous mode
[396722.163942] IPv6: ADDRCONF(NETDEV_UP): veth2IBOFT: link is not ready
[396722.317790] eth0: renamed from vethG3RP4M
[396722.355823] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396722.357395] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396722.357524] IPv6: ADDRCONF(NETDEV_CHANGE): veth2IBOFT: link becomes ready
[396722.357892] lxdbr0: port 320(veth2IBOFT) entered blocking state
[396722.357892] lxdbr0: port 320(veth2IBOFT) entered forwarding state
[396722.695640] audit: type=1400 audit(1576676303.836:3957): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/lxc-start" pid=3619 comm="apparmor_parser"
[396722.696217] audit: type=1400 audit(1576676303.836:3958): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=3620 comm="apparmor_parser"
[396722.696236] audit: type=1400 audit(1576676303.836:3959): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=3620 comm="apparmor_parser"
[396722.696254] audit: type=1400 audit(1576676303.836:3960): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=3620 comm="apparmor_parser"
[396722.696509] audit: type=1400 audit(1576676303.836:3961): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/sbin/tcpdump" pid=3624 comm="apparmor_parser"
[396722.696694] audit: type=1400 audit(1576676303.836:3962): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine" pid=3621 comm="apparmor_parser"
[396722.696713] audit: type=1400 audit(1576676303.836:3963): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=3621 comm="apparmor_parser"
[396722.697136] audit: type=1400 audit(1576676303.836:3964): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=3618 comm="apparmor_parser"
[396739.160148] kauditd_printk_skb: 8 callbacks suppressed
[396739.160150] audit: type=1400 audit(1576676320.300:3973): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-mostafa--yesthisistest--2221_</var/lib/lxd>" pid=4645 comm="apparmor_parser"
[396739.170331] lxdbr0: port 515(vethPWG2QO) entered disabled state
[396739.174603] device vethPWG2QO left promiscuous mode
[396739.174606] lxdbr0: port 515(vethPWG2QO) entered disabled state
[396745.467375] audit: type=1400 audit(1576676326.608:3974): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=5236 comm="apparmor_parser"
[396745.498603] lxdbr0: port 320(veth2IBOFT) entered disabled state
[396745.503841] device veth2IBOFT left promiscuous mode
[396745.503845] lxdbr0: port 320(veth2IBOFT) entered disabled state
[396746.625333] audit: type=1400 audit(1576676327.764:3975): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=5280 comm="apparmor_parser"
[396746.669528] lxdbr0: port 320(vethWE7HWP) entered blocking state
[396746.669530] lxdbr0: port 320(vethWE7HWP) entered disabled state
[396746.670330] device vethWE7HWP entered promiscuous mode
[396746.671813] IPv6: ADDRCONF(NETDEV_UP): vethWE7HWP: link is not ready
[396746.821697] eth0: renamed from vethFFU66X
[396746.855682] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396746.857167] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396746.857304] IPv6: ADDRCONF(NETDEV_CHANGE): vethWE7HWP: link becomes ready
[396746.857728] lxdbr0: port 320(vethWE7HWP) entered blocking state
[396746.857729] lxdbr0: port 320(vethWE7HWP) entered forwarding state
[396747.138077] audit: type=1400 audit(1576676328.276:3976): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=5731 comm="apparmor_parser"
[396747.138098] audit: type=1400 audit(1576676328.276:3977): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=5731 comm="apparmor_parser"
[396747.138117] audit: type=1400 audit(1576676328.276:3978): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=5731 comm="apparmor_parser"
[396747.139456] audit: type=1400 audit(1576676328.280:3979): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine" pid=5732 comm="apparmor_parser"
[396747.139476] audit: type=1400 audit(1576676328.280:3980): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=5732 comm="apparmor_parser"
[396747.139766] audit: type=1400 audit(1576676328.280:3981): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/sbin/tcpdump" pid=5734 comm="apparmor_parser"
[396747.139878] audit: type=1400 audit(1576676328.280:3982): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/lxc-start" pid=5727 comm="apparmor_parser"
[396747.139925] audit: type=1400 audit(1576676328.280:3983): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="lxc-container-default" pid=5725 comm="apparmor_parser"
[396754.378358] kauditd_printk_skb: 8 callbacks suppressed
[396754.378360] audit: type=1400 audit(1576676335.516:3992): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-tehiwo5186--ccdayone--2141_</var/lib/lxd>" pid=6923 comm="apparmor_parser"
[396754.397778] lxdbr0: port 862(veth4JIQ2P) entered disabled state
[396754.402105] device veth4JIQ2P left promiscuous mode
[396754.402109] lxdbr0: port 862(veth4JIQ2P) entered disabled state
[396761.185008] audit: type=1400 audit(1576676342.324:3993): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=7359 comm="apparmor_parser"
[396762.259075] audit: type=1400 audit(1576676343.396:3994): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-haproxy_</var/lib/lxd>" pid=7399 comm="apparmor_parser"
[396762.297622] lxdbr0: port 515(vethD6MK2B) entered blocking state
[396762.297624] lxdbr0: port 515(vethD6MK2B) entered disabled state
[396762.298422] device vethD6MK2B entered promiscuous mode
[396762.300165] IPv6: ADDRCONF(NETDEV_UP): vethD6MK2B: link is not ready
[396762.474217] lxdbr0: port 320(vethWE7HWP) entered disabled state
[396762.482251] device vethWE7HWP left promiscuous mode
[396762.482258] lxdbr0: port 320(vethWE7HWP) entered disabled state
[396762.570570] eth0: renamed from vethOHWEQ5
[396762.596277] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[396762.598022] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[396762.598159] IPv6: ADDRCONF(NETDEV_CHANGE): vethD6MK2B: link becomes ready
[396762.598654] lxdbr0: port 515(vethD6MK2B) entered blocking state
[396762.598656] lxdbr0: port 515(vethD6MK2B) entered forwarding state
[396762.899711] audit: type=1400 audit(1576676344.040:3995): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/man" pid=7779 comm="apparmor_parser"
[396762.899731] audit: type=1400 audit(1576676344.040:3996): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_filter" pid=7779 comm="apparmor_parser"
[396762.899749] audit: type=1400 audit(1576676344.040:3997): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="man_groff" pid=7779 comm="apparmor_parser"
[396762.900115] audit: type=1400 audit(1576676344.040:3998): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/bin/lxc-start" pid=7778 comm="apparmor_parser"
[396762.901023] audit: type=1400 audit(1576676344.040:3999): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/sbin/dhclient" pid=7777 comm="apparmor_parser"
[396762.901038] audit: type=1400 audit(1576676344.040:4000): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=7777 comm="apparmor_parser"
[396762.901052] audit: type=1400 audit(1576676344.040:4001): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=7777 comm="apparmor_parser"
[396762.901066] audit: type=1400 audit(1576676344.040:4002): apparmor="STATUS" operation="profile_load" label="lxd-haproxy_</var/lib/lxd>//&:lxd-haproxy_<var-lib-lxd>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=7777 comm="apparmor_parser"
[396766.664117] kauditd_printk_skb: 8 callbacks suppressed
[396766.664118] audit: type=1400 audit(1576676347.804:4011): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-haproxy_</var/lib/lxd>" name="/home/" pid=8200 comm="(ionclean)" flags="ro, nosuid, nodev, remount, bind"
[397941.333644] audit: type=1400 audit(1576677522.472:4012): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-vladimir--testmock--2968_</var/lib/lxd>" pid=17889 comm="apparmor_parser"
[397941.341441] lxdbr0: port 1006(veth5TV1RL) entered disabled state
[397941.348008] device veth5TV1RL left promiscuous mode
[397941.348011] lxdbr0: port 1006(veth5TV1RL) entered disabled state
[397942.430045] audit: type=1400 audit(1576677523.568:4013): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-vladimir--testmock--2968_</var/lib/lxd>" pid=17918 comm="apparmor_parser"
[397942.472101] lxdbr0: port 320(vethTFDD9M) entered blocking state
[397942.472103] lxdbr0: port 320(vethTFDD9M) entered disabled state
[397942.472873] device vethTFDD9M entered promiscuous mode
[397942.474190] IPv6: ADDRCONF(NETDEV_UP): vethTFDD9M: link is not ready
[397942.629483] eth0: renamed from vethMO4D7R
[397942.667210] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[397942.668774] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[397942.668911] IPv6: ADDRCONF(NETDEV_CHANGE): vethTFDD9M: link becomes ready
[397942.669384] lxdbr0: port 320(vethTFDD9M) entered blocking state
[397942.669385] lxdbr0: port 320(vethTFDD9M) entered forwarding state
[398563.477420] audit: type=1400 audit(1576678144.616:4014): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-haproxy_</var/lib/lxd>" name="/home/" pid=4598 comm="(ionclean)" flags="ro, nosuid, nodev, remount, bind"

lxc info test --show-log

Remote: unix://
Architecture: x86_64
Created: 2019/12/18 12:54 UTC
Status: Running
Type: persistent
Profiles: hosting-free
Pid: 17928
Ips:
  eth0: inet6   fe80::216:3eff:feca:33c1    vethTFDD9M
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
Resources:
  Processes: 8
  CPU usage:
    CPU usage (in seconds): 4
  Memory usage:
    Memory (current): 4.46MB
    Memory (peak): 9.48MB
  Network usage:
    eth0:
      Bytes received: 1.43MB
      Bytes sent: 359.00kB
      Packets received: 12219
      Packets sent: 1025
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0

Log:

lxc test 20191218135843.858 WARN     conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance

lxc config show test --expanded (test = N1001 container)

architecture: x86_64
config:
  boot.autostart: "true"
  image.architecture: x86_64
  image.description: Alpinelinux 3.10 x86_64 (20190918_13:00)
  image.name: alpinelinux-3.10-x86_64-default-20190918_13:00
  image.os: alpinelinux
  image.release: "3.10"
  image.serial: "20190918_13:00"
  image.variant: default
  limits.cpu: "1"
  limits.cpu.allowance: 10%
  limits.memory: 800MB
  limits.memory.enforce: soft
  user.network_mode: dhcp
  volatile.base_image: 98c4926d765df91f1e3d9ec17601c39b85c71cc69890cbe8ebbd9b000827e77a
  volatile.eth0.hwaddr: 00:16:3e:ca:33:c1
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size: 800MB
    type: disk
ephemeral: false
profiles:
- hosting-free
stateful: false
description: ""

/var/log/lxd/lxd.log

Just info logs

[stgraber]

So can be one of a few things:

• Linux bridges used to (and still might) hit a limit at 1024 bridged interfaces
• You hit some network, processes or similar limits
• dnsmasq is misbehaving somehow

What image are you using for your containers?
What's the full dnsmasq command line being run (ps aux | grep dnsmasq)?
Did you apply all the suggested sysctls from production-setup.md?

[vtigranv]

Hi @stgraber
Thanks for reply,
Production-setup is applied.

We use dnsmasq which is inside LXD.
dnsstub of local machine is disabled.

Here is the reply from
ps aux | grep dnsmasq

root     17333  0.0  0.0  18960  1052 pts/887  S+   20:50   0:00 grep --color=auto dnsmasq
lxd      19998  0.7  0.0  56608  3348 ?        S    16:06   2:14 dnsmasq --strict-order --bind-interfaces --pid-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=172.16.0.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/lib/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/lib/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 172.16.0.2,172.16.15.254,1h -s lxd -S /lxd/ --conf-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.raw -u lxd

[stgraber]

Ok, dnsmasq looks fine. Did you try manually running the DHCP client in an affected container and dumping DHCP traffic on the host side veth (name can be found in lxc info NAME) and on the bridge itself?

Effectively trying to see where the traffic is dropped if anywhere, if it's not dropped, then that would suggest an issue with dnsmasq.

[stgraber]

-X, --dhcp-lease-max=<number>
    Limits dnsmasq to the specified maximum number of DHCP leases. The default is 1000. This limit is to prevent DoS attacks from hosts which create thousands of leases and use lots of memory in the dnsmasq process. 

This sounds suspect :)

[stgraber]

Can you try setting raw.dnsmasq to dhcp-lease-max=4000 on your bridge, see if that takes care of the issue?

I'm not sure that we'd want to bump this out of the box. We could add another config option to control it though or just let those few that need to go past it, directly configure dnsmasq through raw.dnsmasq.

[stgraber]

Note that you're quite likely to immediately hit another limit though, while looking into this issue, I've confirmed that the expected limit for Linux bridges is 1024 interfaces, so to get past that, you'd need to move to openvswitch or use multiple bridges.

[stgraber]

Closing this as it's the first time we have someone really hit this and not immediately also hit the bridge limit. We could add a knob but that knob would only be useful for an additional 24 containers in the most case.

If someone else hits this, please comment and we'll consider adding more logic, likely expose a config key and have it refuse to be set past 1023 unless the driver is also changed to openvswitch.