Lxd: Docker - could not open moddep file

[BenoitGEORGELIN]

The template below is mostly useful for bug reports and support questions.
Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Required information

Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

• The output of "lxc info" or if that fails:
  
  
  
  • Kernel version:
    
    4.4.0-28-generic
  
  
  • LXC version:
    
    lxc --version
    
    2.0.3
  
  
  • LXD version:
    
    lxd --version
    
    2.0.2
  
  
  • Storage backend in use:
    
    ZFS
  
  
• Docker Version : 1.11 and 1.12 from ubuntu

# Issue description

Cannot load specific kernel modules for Docker ( br_netfilter , nf_nat)lxc

# Steps to reproduce

1. Step one
Install docker : apt-get install docker.io

2. Step two
Start docker daemon

Information to attach

....WARN[0001] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module bridge not found in directory /lib/modules/4.4.0-28-generic
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.4.0-28-generic
, error: exit status 1

[stgraber]

Hmm, the docker profile loads nf_nat for you, are you using it?

[BenoitGEORGELIN]

Yes I do, but I looks like It's now loaded inside de container

lxc profile show docker

name: docker
config:
  linux.kernel_modules: overlay, nf_nat, br_netfilter
  raw.lxc: lxc.aa_allow_incomplete = 1
  security.nesting: "true"
description: Profile supporting docker in containers
devices:
  aadisable:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  fuse:
    path: /dev/fuse
    type: unix-char

lxc config show dock-m1

name: dock-m1
profiles:
- default
- docker
config:
  security.privileged: "true"
  volatile.base_image: 3fb03bf23058dacff32a8054fd71c7218648073017ad3c1a457fa46e008592e9
  volatile.eth0.name: eth0
  volatile.eth1.name: eth1
  volatile.last_state.idmap: '[]'

[stgraber]

stgraber@dakara:~$ lxc exec docker -- docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
952132ac251a: Pull complete
82659f8f1b76: Pull complete
c19118ca682d: Pull complete
8296858250fe: Pull complete
24e0251a0e2c: Pull complete
Digest: sha256:f4691c96e6bbaa99d99ebafd9af1b68ace2aa2128ae95a60369c506dd6e6f6ab
Status: Downloaded newer image for ubuntu:latest
root@7b8dc4b35039:/# exit

stgraber@dakara:~$ lxc exec docker -- dpkg -l docker.io
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version               Architecture          Description
+++-=================================-=====================-=====================-========================================================================
ii  docker.io                         1.11.2-0ubuntu5~16.04 amd64                 Linux container runtime
stgraber@dakara:~$

[BenoitGEORGELIN]

The problem is only with a swarm cluster using overlay network :

docker: Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded.

Regular docker image run without problem .

root@dock-m1:~# docker -H tcp://$MANAGER_HOST_IP:22222 network create --driver overlay swarm-network

root@dock-m1:~# docker -H tcp://192.168.0.1:22222 network ls
NETWORK ID          NAME                DRIVER
f7c17bea2fce        dock-m1/bridge      bridge              
3013b5577335        dock-m1/host        host                
a16b770083ba        dock-m1/none        null                
6eeef94aeb4d        dock-w1/bridge      bridge              
b2dd89a7f77f        dock-w1/host        host                
3713d6178422        dock-w1/none        null                
ab0e87a92785        swarm-network       overlay     

docker -H tcp://$MANAGER_HOST_IP:22222 run -itd --net swarm-network -e constraint:node==dock-m1 --name u1 ubuntu
afad3f266619e44ce9ae4b6318836e6a28a7c90ef8445c0e79523ad0f371113d
docker: Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded.
root@dock-m1:~# ll /proc/sys/net/bridge/
ls: cannot access '/proc/sys/net/bridge/': No such file or directory

root@dock-m1:~# modprobe br_netfilter modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin' modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/4.4.0-28-generic

After loding the module from the Host i can see it loaded in the container:

root@dock-m1:~# lsmod|grep net br_netfilter 24576 0 bridge 126976 1 br_netfilter vmxnet3 57344 0

But still :

root@dock-m1:~# docker -H tcp://$MANAGER_HOST_IP:22222 start u1 Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded Error: failed to start containers: u1

[stgraber]

Does loading those kernel modules on the host fix things?

Containers are never allowed to load kernel modules, for obvious security reasons. So you need to have the modules loaded on the host (which is what linux.kernel_modules in the LXD config does), then the container "should" be able to use the features offered by the modules (minus kernel or userspace bug).

[BenoitGEORGELIN]

No, it does not fix things.
It's loaded inside the container but looks like docker can't use it.

```root@dock-m1:~# lsmod |grep nf_nat
nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE
nf_nat_ipv4 16384 1 iptable_nat
nf_nat 24576 3 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4
nf_conntrack 106496 6 openvswitch,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4

root@dock-m1:~# lsmod |grep br_net
br_netfilter 24576 0
bridge 126976 1 br_netfilter

``````

I keep having the message :
``....WARN[0001] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin' modprobe: WARNING: Module bridge not found in directory /lib/modules/4.4.0-28-generic modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin' modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.4.0-28-generic , error: exit status 1 WARN[0001] Running modprobe nf_nat failed with message:modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module nf_nat not found in directory /lib/modules/4.4.0-28-generic, error: exit status 1 WARN[0001] Running modprobe xt_conntrack failed with message:modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module xt_conntrack not found in directory /lib/modules/4.4.0-28-generic, error: exit status 1 INFO[0001] Firewalld running: false WARN[0001] Could not load necessary modules for IPSEC rules: Running modprobe xfrm_user failed with message:modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-28-generic/modules.dep.bin'
modprobe: WARNING: Module xfrm_user not found in directory /lib/modules/4.4.0-28-generic`, error: exit status 1
``````

It's a warning about "modprobe" . As it is not suppose to work inside a container, that's fine, but it does not look to be aware that the kernel module is actually loaded and ready to use:

docker -H tcp://$MANAGER_HOST_IP:22222 start u1 Error response from daemon: Error response from daemon: error creating external connectivity network: cannot restrict inter-container communication: please ensure that br_netfilter kernel module is loaded Error: failed to start containers: u1

[stgraber]

Do you actually have anything in /lib/modules? I wonder if that's what's causing all the confusion.

In my container /lib/modules is completely empty, so Docker won't be tempted to go look for modules.

All the failures above seem to be related to module loading, assuming they are all loaded (and it sounds like they are), then it's either modprobe being confused or a Docker bug...

I'm not familiar with swarm (or even Docker really), but none of those errors seem to be tied to an actual failure to use whatever network API they need for their overlay.

If /lib/modules is indeed empty and you're still running into this issue, I'd recommend you file a bug against Docker in Ubuntu at https://launchpad.net/ubuntu/+source/docker.io/+filebug where our Docker maintainer will hopefully be able to help you sort this out (may require an upstream fix).

[BenoitGEORGELIN]

/lib/modules folder does not exist

I will have a look with Docker and open a bug .
Thanks !

[stgraber]

Ok, so sounds like docker just always tries to load the modules even if it's already loaded...

[BenoitGEORGELIN]

Looks like it will manage by docker.io team .
Here is the link : https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1618283

[realdimas]

Docker's libnetwork issue tracking similar symptoms: https://github.com/docker/libnetwork/issues/1439

[magne]

It seems from the docker libnetwork source setup_bridgenetfiltering.go that it is not the modprobe that's causing docker to fail, but rather the absence of several files/values under /proc/sys/net/.

And one way to avoid this error is to create a file /.dockerenv in the LXD/LXC container.

[stgraber]

Good to know for the workaround. I remember another bug report where missing files in /proc/sys/net were discussed in relation to Docker.

For those, the standing recommendation is to file a bug against the Linux kernel at https://launchpad.net/ubuntu/+source/linux/+filebug and someone from our kernel team will hopefully be able to track down the reason why the file isn't showing up.

Some of those files are tied to the network namespace, allowing root in the container to modify them. Some others are global to the whole host, in which case they should show up, be readable but throw errors when attempting to write to them.

Having the file just be plain missing, is a kernel bug.