Rkt: Network Issues: veth pair: operation not supported

[moredhel]

Environment

rkt Version: 1.12.0
appc Version: 0.8.5
Go Version: go1.6.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL
--
Linux 4.6.4-1-ARCH x86_64
--
NAME="Arch Linux"
ID=arch
PRETTY_NAME="Arch Linux"
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
--
systemd 231
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

What did you do?

I was running through the Getting Started Guide. I managed to build an aci using acbuild.

When Running rkt --insecure-options=image run hello-0.0.1-linux-amd64.aci, I got an error.

What did you expect to see?

I expected to see the output of the container successfully starting up.

What did you see instead?

image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.12.0
image: using image from file hello-0.0.1-linux-amd64.aci
networking: loading networks from /etc/rkt/net.d
networking: overriding "default" network with 00-default.conf
networking: loading network default with type ptp
stage1: failed to setup network: failed to make veth pair: operation not supported

Debug Output:

image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.12.0
image: using image from file hello-0.0.1-linux-amd64.aci
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-e01c15814158e68c325713dbf90c846dc3a25a4466dc3b1c33e2a0c9dd8f22a7
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/194bc617-634e-4391-8dba-650d71f073d5
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/194bc617-634e-4391-8dba-650d71f073d5
stage0: Execing /init
networking: loading networks from /etc/rkt/net.d
networking: overriding "default" network with 00-default.conf
networking: loading network default with type ptp
stage1: failed to setup network
  └─error adding network "default"
    └─failed to make veth pair: operation not supported

I have done a little looking into this, and have found the following:

If I set --net=none or --net=host, the pod starts up successfully, and runs as expected.
(gist is here for reference).

I also came across this issue: https://github.com/coreos/rkt/pull/1706 which mentions using NetworkManager. I'm currently using wicd, but am not sure if this is an issue.

[lucab]

I'm not completely sure what's going on here, but I'm not able to reproduce this and it may be something specific to your host. A couple of information which could be helpful in tracking this down:

• what's the content of 00-default.conf?
• it looks like some syscall may be falling here. Can you please strace -f this rkt run and post the output?
• something may be missing on the kernel side. Is this a stock Arch kernel? Can you post its config and what lsmod says?

[moredhel]

Hi, I have since reformatted my computer, and installed another operating system, and it seems to be working fine now.

For reference, I checked the 00-default.conf, and copied one of them (from the etcd aci) into my /etc/net.d/ directory. It seemed to be something affecting systemd_nspawn. I tried running through the following: Arch Nspawn Wiki.

I had the same issue (a slightly reworded error). I never ran an strace, so don't know if that would have given me anything.

Re the kernel, I was running the default. (It was a fresh install, using base base-devel).

I spoke to _sur_ on _freenode/#rkt_, and he mentioned that systemd had recently been updated.

Hardware was a _Macbook Pro 13" retina 12,1 (2015)_ (for reference).

[lucab]

Thanks for the followup. I think we don't have enough elements to track down this one single strange case, which seems anyway related to some specific host state. I'm closing this, but we'll keep an eye open if we spot something similar.

[daurnimator]

I'm also having this issue.

$ sudo rkt --insecure-options=image run docker://centos
run: disabling overlay support: "overlay entry not present in /proc/filesystems"
stage1: failed to setup network: failed to make veth pair: operation not supported

I'm also running arch. rkt version 1.24.0.
End of strace output:

[pid 25448] openat(AT_FDCWD, "/proc/25443/task/25448/ns/net", O_RDONLY|O_CLOEXEC) = 5
[pid 25448] setns(3, CLONE_NEWNET)      = 0
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25448] getrandom("\x0d", 1, GRND_NONBLOCK) = 1
[pid 25448] getrandom("\x68\x5b\xcf\xb0", 4, 0) = 4
[pid 25448] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE <unfinished ...>
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25448] <... socket resumed> )      = 6
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25448] bind(6, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
[pid 25448] sendto(6, {{len=116, type=0x10 /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_ACK|0x600, seq=1, pid=0}, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0"...}, 116, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12 <unfinished ...>
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20}) = 0 (Timeout)
[pid 25448] <... sendto resumed> )      = 116
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25448] getsockname(6, {sa_family=AF_NETLINK, nl_pid=25443, nl_groups=00000000}, [112->12]) = 0
[pid 25448] recvfrom(6, {{len=136, type=NLMSG_ERROR, flags=0, seq=1, pid=25443}, "\241\377\377\377t\0\0\0\20\0\5\6\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...}, 4096, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [112->12]) = 136
[pid 25448] close(6 <unfinished ...>
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25448] <... close resumed> )       = 0
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25448] setns(5, CLONE_NEWNET)      = 0
[pid 25448] close(5)                    = 0
[pid 25448] futex(0xc42002ed10, FUTEX_WAKE, 1) = 1
[pid 25446] <... futex resumed> )       = 0
[pid 25448] futex(0x62c990, FUTEX_WAKE, 1 <unfinished ...>
[pid 25446] futex(0xc42002ed10, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25448] <... futex resumed> )       = 1
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25448] futex(0xc42008e110, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25443] <... futex resumed> )       = 0
[pid 25443] close(4)                    = 0
[pid 25443] close(3)                    = 0
[pid 25443] write(1, "{\n    \"code\": 100,\n    \"msg\": \"f"..., 83 <unfinished ...>
[pid 25444] <... select resumed> )      = 0 (Timeout)
[pid 25444] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25443] <... write resumed> )       = 83
[pid 25440] <... read resumed> "{\n    \"code\": 100,\n    \"msg\": \"f"..., 512) = 83
[pid 25443] exit_group(1 <unfinished ...>
[pid 25440] futex(0xc3c338, FUTEX_WAKE, 1 <unfinished ...>
[pid 25446] <... futex resumed>)        = ?
[pid 25448] <... futex resumed>)        = ?
[pid 25445] <... futex resumed>)        = ?
[pid 25446] +++ exited with 1 +++
[pid 25445] +++ exited with 1 +++
[pid 25448] +++ exited with 1 +++
[pid 25444] <... select resumed> <unfinished ...>) = ?
[pid 25443] <... exit_group resumed>)   = ?
[pid 25444] +++ exited with 1 +++
[pid 25440] <... futex resumed> )       = 1
[pid 25438] <... futex resumed> )       = 0
[pid 25440] read(7,  <unfinished ...>
[pid 25438] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20}) = 0 (Timeout)
[pid 25438] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25440] <... read resumed> "", 1453) = 0
[pid 25443] +++ exited with 1 +++
[pid 25410] <... waitid resumed> {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=25443, si_uid=0, si_status=1, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
[pid 25440] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=25443, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
[pid 25410] wait4(25443,  <unfinished ...>
[pid 25440] rt_sigreturn({mask=[]} <unfinished ...>
[pid 25410] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, {ru_utime={tv_sec=0, tv_usec=0}, ru_stime={tv_sec=0, tv_usec=0}, ...}) = 25443
[pid 25440] <... rt_sigreturn resumed> ) = 0
[pid 25440] close(7 <unfinished ...>
[pid 25410] futex(0xc3cdb0, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25440] <... close resumed> )       = 0
[pid 25438] <... select resumed> )      = 0 (Timeout)
[pid 25440] futex(0xc42002c910, FUTEX_WAKE, 1 <unfinished ...>
[pid 25438] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25440] <... futex resumed> )       = 1
[pid 25439] <... futex resumed> )       = 0
[pid 25440] futex(0xc3cdb0, FUTEX_WAKE, 1 <unfinished ...>
[pid 25439] futex(0xc42002c910, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25410] <... futex resumed> )       = 0
[pid 25440] <... futex resumed> )       = 1
[pid 25440] futex(0xc42002cd10, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25410] mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
[pid 25438] <... select resumed> )      = 0 (Timeout)
[pid 25410] <... mmap resumed> )        = 0x7f1c36e98000
[pid 25438] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25410] write(2, "stage1: failed to setup network:"..., 83stage1: failed to setup network: failed to make veth pair: operation not supported
 <unfinished ...>
[pid 25438] <... select resumed> )      = 0 (Timeout)
[pid 25410] <... write resumed> )       = 83
[pid 25438] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=20} <unfinished ...>
[pid 25410] exit_group(254 <unfinished ...>
[pid 25441] <... futex resumed>)        = ?
[pid 25410] <... exit_group resumed>)   = ?
[pid 25441] +++ exited with 254 +++
[pid 25440] <... futex resumed>)        = ?
[pid 25439] <... futex resumed>)        = ?
[pid 25438] <... select resumed> <unfinished ...>) = ?
[pid 25439] +++ exited with 254 +++
[pid 25438] +++ exited with 254 +++
[pid 25440] +++ exited with 254 +++
+++ exited with 254 +++

[squeed]

This will be a bit tricky to debug from a strace, since the "interesting" operations are all Netlink packets.

This sort of error message usually means your kernel doesn't have veth support enabled. What happens when you do:

sudo ip link add veth0 type veth peer name veth1

[daurnimator]

I think I figured out this issue: I installed a new kernel but hadn't rebooted yet. Therefore kernel modules required by rkt weren't present to be loaded on demand.
Solution: a) reboot into new kernel b) reinstall kernel modules for currently booted kernel