Moby: Error response from daemon: failed to create endpoint (...) on network bridge: failed to add the host (veth1d85371) <=> sandbox (vethbc264f6) pair interfaces: operation not supported.

Created on 16 Oct 2016  路  22Comments  路  Source: moby/moby

I have a fresly installed docker but when I cannot execute containers

docker run --rm hello-world                                                                                                                          
docker: Error response from daemon: failed to create endpoint jolly_kare on network bridge: failed to add the host (veth1d85371) <=> sandbox (vethbc264f6) pair interfaces: operation not supported.

Docker info are:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.2
Storage Driver: devicemapper
 Pool Name: docker-8:1-799432-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 14.09 MB
 Data Space Total: 107.4 GB
 Data Space Available: 21.34 GB
 Metadata Space Used: 585.7 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.10.23-xxxx-std-ipv6-64-vps
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.854 GiB
Name: vps102867
ID: RTPT:KG2U:SH6J:5KS4:5S27:A57A:E6PV:425A:7KEF:UJKR:5WOG:FQBV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Insecure Registries:
 127.0.0.0/8

I've already tried to delete /var/lib/docker/network/files/local-kv.db as mentioned in similar issues with no success

arenetworking
Source
picture lrkwz

Most helpful comment

I don't see any problem there. I've just left the comment for the ones using Arch Linux so they know a reboot is needed after updating the kernel.

Thanks for your clarification @thaJeztah.

picture whoan on 9 Feb 2017
👍181 ❤67 🎉29 🚀8

All 22 comments

Looks like you're running a non-standard kernel for Ubuntu 14.04. Ubuntu 14.04 ships with Kernel 3.13. Can you try with a standard kernel? It's possible your kernel is missing modules that are required for docker to function properly; you can try running this script to see what's missing; https://github.com/docker/docker/blob/master/contrib/check-config.sh

thaJeztah picture thaJeztah on 17 Oct 2016
❤5

Oops ... script's output is

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: missing
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: missing
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: missing

Optional Features:
- CONFIG_USER_NS: missing
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_MEMCG_KMEM: missing
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_NETPRIO_CGROUP: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: missing
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: enabled
- CONFIG_EXT3_FS_POSIX_ACL: enabled
- CONFIG_EXT3_FS_SECURITY: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: missing
    - CONFIG_DUMMY: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

I wonder how safe can be a kernel change on a non dedicated ovh's vps

lrkwz picture lrkwz on 17 Oct 2016

Looks like there's various things missing, perhaps best to contact ovh support?

Based on the above, I don't think this is a bug in docker, so I'll close this issue, but feel free to continue the discussion here

thaJeztah picture thaJeztah on 17 Oct 2016

In my case, the error appears every time I update my Linux kernel. It disappears when I restart the computer.

I am using Arch Linux.

Update:

An alternative is to downgrade the kernel if restart isn't feasible at the moment.

sudo pacman -U /var/cache/pacman/pkg/linux-$(uname -r | sed 's/-ARCH//')-x86_64.pkg.tar.xz

Credits to @Nowaker . See comment below

whoan picture whoan on 9 Feb 2017
👍190 🎉32 🚀17

Arch linux is a rolling release, and is shipping with "bleeding edge" versions of packages. Because of this, it's known that Arch periodically breaks docker; for that reason we don't ship packages for Arch. It may be best to report this with Arch linux, who maintains their packages

thaJeztah picture thaJeztah on 9 Feb 2017

I don't see any problem there. I've just left the comment for the ones using Arch Linux so they know a reboot is needed after updating the kernel.

Thanks for your clarification @thaJeztah.

whoan picture whoan on 9 Feb 2017
👍181 ❤67 🎉29 🚀8

Thanks @whoan for your answer!

VinyLimaZ picture VinyLimaZ on 15 Sep 2017

@whoan For Arch Linux, after you update the kernel, the old kernel modules will be all deleted (replaced with new version kernel modules) automatically. So if you didn't restart computer, you will be running old kernel without matched kernel modules. Most docker commands will load kernel modules (if they haven't been loaded yet), but your old kernel can't find its modules any more. That is why the error appears every time you update your Linux kernel, and disappears after you restart the computer.

Kernel modules directory is in /lib/modules/. Next time before you update kernel, backup the old kernel modules directory. After update, restore the old kernel modules directory, then you won't see this error even if you don't restart computer.

ckyoog picture ckyoog on 22 Nov 2017
👍33

I have the same error, but did not work after reboot. So I rm all images and containsers with command docker image rm and docker rm to delete. Then docker works fine.

stardiviner picture stardiviner on 2 Jan 2018
👍1

I was having this issue because I was running a custom built kernel without the Virtual ethernet pair device option. So in case you have a custom kernel, make to enable that option before running make. You'll find it under:

Device Drivers -> Network device support -> Network core driver support

Putting this here because this seems to be the first result on Google for this error. Hope it helps!

tomas picture tomas on 1 Feb 2018
👍5

@whoan An alternative is to downgrade the kernel if restart isn't feasible at the moment. I'd be great if you could edit your comment so that all the knowledge is in one place. Thanks :-)

sudo pacman -U /var/cache/pacman/pkg/linux-$(uname -r | sed 's/-ARCH//)'-x86_64.pkg.tar.xz

Nowaker picture Nowaker on 12 Feb 2018
👍8

Thanks @Nowaker for your contribution. I added it to my previous comment as suggested.

whoan picture whoan on 13 Feb 2018
👍5

@Nowaker thanks! Minor typo fix for future searchers:
sudo pacman -U /var/cache/pacman/pkg/linux-$(uname -r | sed 's/-ARCH//')-x86_64.pkg.tar.xz
(note the swapped positions of the final single quote and parenthesis)

morancj picture morancj on 26 Apr 2018
👍5

Thanks @morancj. I fixed it on my previous comment.

whoan picture whoan on 27 Apr 2018

Thanks whoan, that was super helpful!

czeigler picture czeigler on 30 Apr 2018

@tomas , Could you help to see this issue for me? I was running a custom built kernel.
Here is the linkdocker can not run in armv7l

I will try your method tomorrow you remind upon. THANKS A LOT .

jia-zhengwei picture jia-zhengwei on 27 Dec 2018

@tomas , Thanks for your answer above, it solved my problem. Thanks a lot.

jia-zhengwei picture jia-zhengwei on 28 Dec 2018
👍2

The output of the script: https://github.com/docker/docker/blob/master/contrib/check-config.sh

I see some missing packages, what shall we do to fix the error:
failed to create endpoint flamboyant_tereshkova on network bridge: adding interface vethfb90d78 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface

warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.8.0-59-generic ...

Generally Necessary:

  • cgroup hierarchy: properly mounted [/sys/fs/cgroup]
  • apparmor: enabled and tools installed
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_CPUSETS: enabled
  • CONFIG_MEMCG: enabled
  • CONFIG_KEYS: enabled
  • CONFIG_VETH: enabled (as module)
  • CONFIG_BRIDGE: enabled (as module)
  • CONFIG_BRIDGE_NETFILTER: enabled (as module)
  • CONFIG_NF_NAT_IPV4: enabled (as module)
  • CONFIG_IP_NF_FILTER: enabled (as module)
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
  • CONFIG_IP_NF_NAT: enabled (as module)
  • CONFIG_NF_NAT: enabled (as module)
  • CONFIG_NF_NAT_NEEDED: enabled
  • CONFIG_POSIX_MQUEUE: enabled

Optional Features:

  • CONFIG_USER_NS: enabled
  • CONFIG_SECCOMP: enabled
  • CONFIG_CGROUP_PIDS: enabled
  • CONFIG_MEMCG_SWAP: enabled
  • CONFIG_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
  • CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
  • CONFIG_BLK_CGROUP: enabled
  • CONFIG_BLK_DEV_THROTTLING: enabled
  • CONFIG_IOSCHED_CFQ: enabled
  • CONFIG_CFQ_GROUP_IOSCHED: enabled
  • CONFIG_CGROUP_PERF: enabled
  • CONFIG_CGROUP_HUGETLB: enabled
  • CONFIG_NET_CLS_CGROUP: enabled (as module)
  • CONFIG_CGROUP_NET_PRIO: enabled
  • CONFIG_CFS_BANDWIDTH: enabled
  • CONFIG_FAIR_GROUP_SCHED: enabled
  • CONFIG_RT_GROUP_SCHED: missing
  • CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
  • CONFIG_IP_VS: enabled (as module)
  • CONFIG_IP_VS_NFCT: enabled
  • CONFIG_IP_VS_PROTO_TCP: enabled
  • CONFIG_IP_VS_PROTO_UDP: enabled
  • CONFIG_IP_VS_RR: enabled (as module)
  • CONFIG_EXT4_FS: enabled
  • CONFIG_EXT4_FS_POSIX_ACL: enabled
  • CONFIG_EXT4_FS_SECURITY: enabled
  • Network Drivers:

    • "overlay":

    • CONFIG_VXLAN: enabled (as module)

      Optional (for encrypted networks):



      • CONFIG_CRYPTO: enabled


      • CONFIG_CRYPTO_AEAD: enabled


      • CONFIG_CRYPTO_GCM: enabled (as module)


      • CONFIG_CRYPTO_SEQIV: enabled


      • CONFIG_CRYPTO_GHASH: enabled (as module)


      • CONFIG_XFRM: enabled


      • CONFIG_XFRM_USER: enabled (as module)


      • CONFIG_XFRM_ALGO: enabled (as module)


      • CONFIG_INET_ESP: enabled (as module)


      • CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)



    • "ipvlan":

    • CONFIG_IPVLAN: enabled (as module)

    • "macvlan":

    • CONFIG_MACVLAN: enabled (as module)

    • CONFIG_DUMMY: enabled (as module)

    • "ftp,tftp client in container":

    • CONFIG_NF_NAT_FTP: enabled (as module)

    • CONFIG_NF_CONNTRACK_FTP: enabled (as module)

    • CONFIG_NF_NAT_TFTP: enabled (as module)

    • CONFIG_NF_CONNTRACK_TFTP: enabled (as module)

  • Storage Drivers:

    • "aufs":

    • CONFIG_AUFS_FS: enabled (as module)

    • "btrfs":

    • CONFIG_BTRFS_FS: enabled (as module)

    • CONFIG_BTRFS_FS_POSIX_ACL: enabled

    • "devicemapper":

    • CONFIG_BLK_DEV_DM: enabled

    • CONFIG_DM_THIN_PROVISIONING: enabled (as module)

    • "overlay":

    • CONFIG_OVERLAY_FS: enabled (as module)

    • "zfs":

    • /dev/zfs: missing

    • zfs command: missing

    • zpool command: missing

Limits:

  • /proc/sys/kernel/keys/root_maxkeys: 1000000
gagandeepchhabda12305 picture gagandeepchhabda12305 on 17 May 2019

Who knew? The solution was to turn it off and back on again.

adamkdean picture adamkdean on 6 Aug 2019
😄3

Seems that Linux copied that trick from Windows

whoan picture whoan on 6 Aug 2019
😄3

I should add for other Arch users that the reboot in my case was due to a system upgrade the same day. Seems like docker can't establish connection if you upgraded arch without rebooting.

Samuelfaure picture Samuelfaure on 13 Sep 2019
👍6

Same kind of trouble with my Raspberry PI.
Restart solved the problem.

denis54 picture denis54 on 1 Nov 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

karellm picture karellm  路  3Comments
Nurza picture Nurza  路  3Comments
bhuvan picture bhuvan  路  3Comments
lvthillo picture lvthillo  路  3Comments
surlymo picture surlymo  路  3Comments