Weave: Using Weave CNI in Mesos: "no such file or directory" error

[philwinder]

I'm attempting to get Weave CNI working with Mesos CNI. I've stolen commands from the K8s tutorial and merged them in with a Mesos tutorial. Everything seems to be setup correctly, but when the call is made to /opt/bin/cni/weave-net and hence the docker run plugin... command, the container fails with the logs:

{
    "code": 100,
    "msg": "no such file or directory"
}

I edited the weave-net script to add some debug messages like so:

#!/bin/sh
set -x
echo "\nEnv Vars\n"
env
echo "------"
ls /etc/cni
ls /opt/cni
ls CNI_PATH
ls /var/run/docker.sock

docker run --rm --privileged --net=host -v /var/run/docker.sock:/var/run/docker.sock --pid=host -i \
 -e CNI_VERSION -e CNI_COMMAND -e CNI_CONTAINERID -e CNI_NETNS \
 -e CNI_IFNAME -e CNI_ARGS -e CNI_PATH -v /etc/cni:/etc/cni -v /opt/cni:/opt/cni \
 weaveworks/plugin:1.6.0  --cni-net -log-level debug

Which resulted in:

ubuntu@ip-10-0-0-249:~$ sudo mesos-execute --command=/bin/bash --docker_image=ubuntu:latest --master=ec2-52-49-23-223.eu-west-1.compute.amazonaws.com:5050 --name=ubuntu --networks=weave --no-shell
I0624 08:28:19.010092  6044 scheduler.cpp:187] Version: 1.0.0
I0624 08:28:19.011386  6050 scheduler.cpp:471] New master detected at [email protected]:5050
Subscribed with ID '85ab8d71-94b0-4d7e-8a43-1938a716c786-0020'
Submitted task 'ubuntu' to agent '85ab8d71-94b0-4d7e-8a43-1938a716c786-S4'
Received status update TASK_FAILED for task 'ubuntu'
  message: 'Failed to launch container: Collect failed: The CNI plugin 'weave-net' failed to attach container 6249ab71-2ba0-4f35-bbe4-385228792284 to CNI network 'weave': 
Env Vars

CNI_PATH=/opt/cni/bin
CNI_CONTAINERID=6249ab71-2ba0-4f35-bbe4-385228792284
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CNI_NETNS=/run/mesos/isolators/network/cni/6249ab71-2ba0-4f35-bbe4-385228792284/ns
CNI_IFNAME=eth0
CNI_COMMAND=ADD
PWD=/
------
net.d
bin
/var/run/docker.sock
{
    "code": 100,
    "msg": "no such file or directory"
}; Abnormal executor termination'
  source: SOURCE_AGENT
  reason: REASON_CONTAINER_LAUNCH_FAILED

As you can see, all of the directories appear to be present and correct.

Also, I tracked this down to an error being thrown by the containernetworking/cni code here: https://github.com/containernetworking/cni/blob/master/vendor/golang.org/x/sys/unix/zerrors_linux_amd64.go

But I don't know what code calls that error message.

Setup below:

ubuntu@ip-10-0-0-69:~$ cat /etc/mesos-slave/network_cni_config_dir 
/etc/cni/net.d
ubuntu@ip-10-0-0-69:~$ cat /etc/mesos-slave/network_cni_plugins_dir 
/opt/cni/bin
ubuntu@ip-10-0-0-69:~$ cat /etc/cni/net.d/10-weave.conf
{
    "name": "weave",
    "type": "weave-net"
}
ubuntu@ip-10-0-0-69:~$ cat /opt/cni/bin/weave-net 
#!/bin/sh
docker run --rm --privileged --net=host -v /var/run/docker.sock:/var/run/docker.sock --pid=host -i \
 -e CNI_VERSION -e CNI_COMMAND -e CNI_CONTAINERID -e CNI_NETNS \
 -e CNI_IFNAME -e CNI_ARGS -e CNI_PATH -v /etc/cni:/etc/cni -v /opt/cni:/opt/cni \
 weaveworks/plugin:1.6.0  --cni-net
ubuntu@ip-10-0-0-69:~$ sudo mesos-execute --command=/bin/bash --docker_image=ubuntu:latest --master=ec2-52-49-23-223.eu-west-1.compute.amazonaws.com:5050 --name=ubuntu --networks=weave --no-shell
I0624 08:54:44.619662  5762 scheduler.cpp:187] Version: 1.0.0
I0624 08:54:44.621544  5766 scheduler.cpp:471] New master detected at [email protected]:5050
Subscribed with ID '85ab8d71-94b0-4d7e-8a43-1938a716c786-0025'
Submitted task 'ubuntu' to agent '85ab8d71-94b0-4d7e-8a43-1938a716c786-S3'
Received status update TASK_FAILED for task 'ubuntu'
  message: 'Failed to launch container: Collect failed: The CNI plugin 'weave-net' failed to attach container 0de98310-57df-4f5e-9eb3-39d9d4499e34 to CNI network 'weave': {
    "code": 100,
    "msg": "no such file or directory"
}; Abnormal executor termination'
  source: SOURCE_AGENT
  reason: REASON_CONTAINER_LAUNCH_FAILED

[squaremo]

Does Mesos try to use IPAM as well? And are the other plugins (that might be used for IPAM) in /opt/cni/bin/?

[rade]

@bboreham has just pointed out that the directory referenced in

CNI_NETNS=/run/mesos/isolators/network/cni/6249ab71-2ba0-4f35-bbe4-385228792284/ns

is not volume-mounted by the docker run invocation, as a result of which the plugin won't be able to find the netns, which is probably what it is complaining about.

Adding a -v /run/mesos/isolators/network/cni:/run/mesos/isolators/network/cni should fix that.

PS: we should really improve the error reporting here: "no such file or directory" without naming the file or directory is a cruel tease.

[squaremo]

If the net namespace not being mounted is the problem, how does this work for any other use of Weave's CNI plugin?

[bboreham]

For other uses, the file is /proc/pid/ns/net, and /proc is the same inside the container due to the --pid=host option.

[bboreham]

Rather than attempting to modify the container-calling script, you may find it easier (and faster) to run outside of a container, as illustrated here: https://github.com/weaveworks/weave/issues/2224#issuecomment-215092833

[squaremo]

For other uses, the file is /proc/pid/ns/net, and /proc is the same inside the container due to the --pid=host option.

Got it, ta. I had presumed it would involve /var/run/netns.

[philwinder]

@rade That worked! The working script looks like:

ubuntu@ip-10-0-0-196:~$ cat /opt/cni/bin/weave-net 
#!/bin/sh
docker run --rm --privileged --net=host -v /var/run/docker.sock:/var/run/docker.sock --pid=host -i \
 -e CNI_VERSION -e CNI_COMMAND -e CNI_CONTAINERID -e CNI_NETNS \
 -e CNI_IFNAME -e CNI_ARGS -e CNI_PATH -v /etc/cni:/etc/cni -v /opt/cni:/opt/cni \
 -v /run/mesos/isolators/network/cni:/run/mesos/isolators/network/cni \
 weaveworks/plugin:1.6.0  --cni-net

So it starts now, I just need to test whether it has actually connected to the network...

[philwinder]

Ok, Mesos 1.0 is using a new unifed containerizer. When you run a docker container it doesn't look like it's using the docker daemon to do it. It's using it's own. So I can't easily see anything about the status of the container, other than it's running.

However according to weave ps, it doesn't look like weave knows about the container (probably because it's not going through docker)

ubuntu@ip-10-0-0-196:~$ weave ps
weave:expose 52:7a:97:e1:c3:8d 10.44.0.0/12

Is that right? Should the container appear here if it was connected?

[rade]

Should the container appear here if it was connected?

I am pretty sure it should, provided this is a recent weave version.

[rade]

@philwinder is your last query covered by #2404?

[philwinder]

@rade Yes, thanks. But obviously the above issue is still a bug and needs to be fixed.

[rade]

the above issue

The path/volume issue? Right. I guess the weave-net script needs to add a volume-mount for the CNI_NETNS dir.

[philwinder]

Correct. See my working example above:

 -v /run/mesos/isolators/network/cni:/run/mesos/isolators/network/cni \

[rade]

I wonder whether we should mount $CNI_PATH too, rather than the hard-coded /opt/cni. @bboreham thoughts?

[bboreham]

I wonder whether we should mount $CNI_PATH too

Initially I thought you were asking this about weave setup, which has a complex answer.

If you were asking about the script output by create_cni_script, that sounds like a good idea, but note that $CNI_PATH may contain a colon-separated list of directories.

[bboreham]

What are the arguments for sticking with the exec-via-Docker model rather than extracting the binary, which wouldn't need any -v options?

[rade]

note that $CNI_PATH may contain a colon-separated list of directories.

urgh. I will punt on that.

What are the arguments for sticking with the exec-via-Docker model rather than extracting the binary,

First I've heard of that idea. I think it would be too risky for 1.6

[bboreham]

First I've heard of that idea.

As noted here: https://github.com/weaveworks/weave/issues/2394#issuecomment-228344935