Runtime: network configuration hotplug support

[egernst]

[egernst]

cc/ @WeiZhang555

[sboeuf]

Ok I'll try to summarize the status of the previous discussion here.

We want to be able to support docker network connect. This command is not part of the OCI lifecycle, meaning it does not call into our kata-runtime to perform this operation.
Instead, it simply adds the new network to the network namespace of the container/pod. For this reason, we need a process running inside the network namespace to detect any change about interfaces and routes inside the namespace. That's the first part of what we need.

Now the second part is the API itself which does not currently allow us for adding a new interface or route. This is already part of the scope as you can see here (also referenced here). This means someone could already start working on this if this is important for you guys (@guangxuli @miaoyq).

Now, to get back to the detection part, I think we should make sure this can be disabled in case we don't need it (k8s case) since this extra binary (even if this is a reexec from the library itself) will increase the memory footprint for each pod. Since this is only a problem that we'll face with Docker (one container per pod), the kata-shim representing the container process could handle this directly without pulling the whole virtcontainers package if we were exposing a special command at the CLI level (something like kata-runtime add-network ...).

@bergwolf @laijs @WeiZhang555 I'd like your input about this !

[WeiZhang555]

@sboeuf This sounds feasible.

Now, to get back to the detection part, I think we should make sure this can be disabled in case we don't need it (k8s case) ...

I like it :)

the kata-shim representing the container process could handle this directly without pulling the whole virtcontainers package if we were exposing a special command at the CLI level (something like kata-runtime add-network ...).

kata-shim works as a "detecter" (optionally), and when new interface is added into its namespace, it will invoke kata-runtime add-network to add a new interface into qemu, am I right? If so this sound quite good.

And a CNI plugin can also make use of kata-runtime add-network. Everything makes sense, I like your whole idea 👍

[WeiZhang555]

@sboeuf

the kata-shim representing the container process could handle this directly without pulling the whole virtcontainers package if we were exposing a special command at the CLI level (something like kata-runtime add-network ...).

I just realized one problem runv once met one minute ago. Kata-shim is representing one container process(init or exec), we need to make sure the detector part will still work after one kata-shim exits.
There are some ways we can do is:

1. first kata-shim works as detector and send kata add-network once a nic detected: this is not workable, because once the container exits with first kata-shim, the POD can has other containers living, this will make docker network connect losing functions. Unless we have some kind of election mechanism to make sure one new detector is elected once old network detector died.
2. every kata-shim works as detector to make sure one kata-shim exits won't disable CNM network hotplug: this is resource inefficient, and need to sync up among all kata-shim processes.
3. do this inside kata-proxy, as kata-proxy is always alongside VM, so it's rational to put network detector inside kata-proxy: problem is with vsock we don't have a kata-proxy.

[amshinde]

If we need to support scenarios other than docker, we really need to have a separate lightweight process monitoring the network namespace for network changes. This would then invoke the kata-runtime library directly with a call such as kata-runtime add-interface or kata-runtime update-route.

[sboeuf]

@WeiZhang555

1. first kata-shim works as detector and send kata add-network once a nic detected: this is not workable, because once the container exits with first kata-shim, the POD can has other containers living, this will make docker network connect losing functions. Unless we have some kind of election mechanism to make sure one new detector is elected once old network detector died.

I agree this would not work for other cases than Docker (only one container per pod), and this could be solved by introducing a different binary for this purpose only. But my question is, do we need to support other cases than Docker here ? Are we expecting CRI cases to be able to hotplug some more network that our shim should also detect ?

2.every kata-shim works as detector to make sure one kata-shim exits won't disable CNM network hotplug: this is resource inefficient, and need to sync up among all kata-shim processes.

If we're handling only the Docker case, the only shim who needs to monitor is the one representing the container process. We are sure this shim won't terminate before, meaning it won't miss anything if a docker network connect happens during the container's lifetime.

1. do this inside kata-proxy, as kata-proxy is always alongside VM, so it's rational to put network detector inside kata-proxy: problem is with vsock we don't have a kata-proxy.

You've answered yourself, this is not a suitable design as we always have to take into account that we might not have a proxy process.

@kata-containers/runtime

To summarize a bit, here are two questions I have and each of them has two options:

• How do we want the "monitoring" process (shim or separate process) to call into add network capability of Kata ?
  
  
  
  • Either we reexec from virtcontainers itself, meaning the process will be able to call directly into the Kata API sandbox.AddNetwork(). This involves a bigger memory footprint since we will have the monitoring process being as large as the runtime process, but in this case the process stays alive for the whole pod lifetime.
  
  
  • Or the monitoring process would be calling into the runtime path .../.../kata-runtime add-network name=... ip=..., which would allow for a much simpler and lightweight process (written in C probably), that would only need to know about the runtime path. The drawback of this approach is that we would be spawning a new process from the monitoring process itself.
  
  

• Do we expect other cases than Docker to have a monitoring process running in the network namespace ?
  
  
  
  • If we don't, let's go with the specific implementation, and directly handle this through the kata-shim. No overhead, and minimal changes from virtcontainers design perspective.
  
  
  • If we do, then the only option IMO is a dedicated monitoring process (I don't want to sync kata-shims together so that they know who is monitoring). This process should be as tiny as possible and that's why I was thinking about C code for this. And this would involve a little bit more changes in virtcontainers to properly spawn/handle this monitoring process.
  
  

@kata-containers/runtime We really need some input here before we take a decision and we go ahead with the implementation.

[bergwolf]

Speaking of network monitoring, is the monitor process going to do a busy-polling or is there some notification mechanism that would let docker network connect notify runtime that there is a network change?

[WeiZhang555]

@bergwolf

github.com/vishvananda/netlink has a good subscribe and notification mechanism for this, and you already has this though it's obsolete :) Check this: https://github.com/hyperhq/runv/blob/master/cli/network.go#L501-L536

@sboeuf

Do we expect other cases than Docker to have a monitoring process running in the network namespace ?

Hold on, we might get diverged on "what is Docker case". If we are planning to support libnetwork(CNM) model, don't we need to support "docker run --net container:xxxx" to share two containers network namespace? In other words, don't we want to support composing a POD from docker's client? This was the missing part of CC initially in my opinion, and my hope is we can enhance and support this later.

[egernst]

@WeiZhang555 -- I think we should discuss this in architecture committee on Monday.

Hold on, we might get diverged on "what is Docker case". If we are planning to support libnetwork(CNM) model, don't we need to support "docker run --net container:xxxx" to share two containers network namespace? In other words, don't we want to support composing a POD from docker's client? This was the missing part of CC initially in my opinion, and my hope is we can enhance and support this later.

Regarding docker network connect: this could be achieved in CC (this is how I did multi-interface early setup for a VNF like chain in docker), but the original veths would be unconfigured in the namespace, so connectivity of original container would be lost if you were connecting to a runc container's network with a Clear Container. In other words, I didn't use this for shared-netns between containers, but as a workaround to not being able to do hotplug!) AFAIU, if we moved to something like a TC based solution, this should work. @mcastelino -- agreed? I think this is seperate from the hot-plug discussion (such as network connect).

[sboeuf]

@egernst @WeiZhang555 yes I think this is a separate discussion (not related to detecting a network being hotplugged without notice) since this can be tied to an OCI lifecycle event (docker run translating into kata-runtime create + start in that case).

[WeiZhang555]

@egernst @sboeuf Yes, we can discuss this in Monday's arch meeting.

[egernst]

@guangxuli @miaoyq - can you confirm you are planning on adding this feature?

[sboeuf]

@egernst @bergwolf @WeiZhang555 hey guys, as discussed on #287 with @caoruidong, we were wondering what approach we should choose for network hotplug, and more specifically regarding CLI/API parameters. API parameters are easy, we can provide anything (interface name or full interface definition, this is not a big deal), but for the CLI, this is a different story. If we follow the declarative model we've been talking all the time, this means we'll have to describe the interface(s) through the CLI, which can be heavy but feasible. This approach would follow the initial discussions we had, and it would avoid virtcontainers to re-scan the network namespace to retrieve details about a specific interface.
On the other hand, we could rely on interface name only, and this would simplify both the API/CLI, but this would trigger some extra scanning to be performed by virtcontainers.

This comment is here to make sure we're on the same page because @caoruidong is currently working on the implementation and he cannot move forward until we agree on how this should be handled.

[WeiZhang555]

The command line interface provided by @caoruidong in #287 is basically what we used in runv command line, it worked very well in our CNI use case. Following the declarative model, I would say we can make the command like behave like this:

 // "-" indicates input from stdio, "[filename]" means input from some file
kata-runtime update-network <container> - | [filename]
kata-runtime list-network <container>

The input can contain enough information for setting up the network. a rough example:

{
"interfaces": [
  {
    "name": "eth0",
     "device": "tap0",
    "ipv4":["192.168.0.2/24"， “192.168.1.2/24”],
     "ipv6":["xxxx"],
    "mac": "aa:bb:cc:dd:ee:aa",
     "mtu": 1500,
  },
  {
    "name": "eth1",
     "device": "tap1",
    "ipv4":["xxxx”],
     "ipv6":["xxxx"],
    "mac": "aa:bb:cc:dd:ee:bb",
     "mtu": 1500,
  }
],
"route": [
  {
    "dest": "192.168.0.0/24",
    "source": "192.168.0.2",
    "gateway": "192.168.0.1",
    "device": "eth0"
  }
]
}

so kata-runtime update-network <container> - with input above can give container exactly two network interfaces and one route rule.

And if echo "{}" | kata-runtime update-network <container> - can delete all the interfaces and route rule from container/pod.

kata-runtime list-network <container> will show same content in json(or another) format.

Though this is different from what we were using (more close to imperative way), I think this new interface is not a bad choice.

The command line showed above is just for illustrating my idea, we need to discuss more about the detailed interface format

[bergwolf]

@WeiZhang555 @sboeuf I agree we should still follow the declarative way for both API and cli interfaces, and push the network config setup/scanning to upper layers.

[caoruidong]

I'll try with the declarative way and take runv as a guidance

[sboeuf]

@WeiZhang555 @caoruidong Yes I think the format is nice, and actually appropriate for such a huge amount of data.

[WeiZhang555]

@sboeuf Nice, then I think @caoruidong can move forward now and make an initial implementation based on this design for reviewing.

[caoruidong]

In agent proto, UpdateRoutes is declarative already. But for interface, we just have AddInterface, RemoveInterface and UpdateInterface.
As discussed here, we should follow the declarative way for API and CLI. Do we just work in the runtime side and keep those agent grpcs unchanged?
Or we need to add UpdateInterfaces to agent grpc that can set all interfaces in one shot by format @WeiZhang555 suggested. It's much closer to the first edition of https://github.com/kata-containers/agent/pull/126.
@bergwolf @sboeuf WDYT ?

[bergwolf]

@caoruidong I would prefer to keep the agent implementation simple and let either kata CLI or virtcontainers handle the sliced update operation (and possible rollback).

That said, do you guys agree we need a list network API and CLI? It's documented in the API design ListNetwork

[caoruidong]

Yes. I think ListNetwork is necessary.

[caoruidong]

I have updated my PR. It implements the CLI interface. Network config file is in following format:

{
"interfaces":[
        {
        "device":"tap1",
        "name":"eth0",
        "IPAddresses":[{"address":"192.168.0.2","mask":"24"}],
        "mtu":1500,
        "hwAddr":"00:e0:4c:97:ad:66"
        },
        {
        "device":"tap3",
        "name":"eth1",
        "IPAddresses":[{"address":"192.168.0.3","mask":"16"}],
        "mtu":1400,
        "hwAddr":"00:e0:4c:97:ad:67"
        }
],
"routes":[
        {
        "dest": "192.168.0.0/24",
        "source": "192.168.0.2",
        "device": "eth0"
        }
]
}

But network is not working well. Is kata's QEMU in the net namespace of shim not host? If send QMP "netdev_add ifname=tap0" to runv's QEMU, it will use the tap0 in the host. But kata's QEMU will use the tap0 in the shim net namespace. So the network is not connected correctly.
What is the proper process to hotplug a nic to kata? Need to add veth pair first?

[WeiZhang555]

As @caoruidong talked about , I realized there's an issue here, the qemu is in standalone net namespace, which means even we can plug a tap device to the qemu, we still need (maybe) a bridge and an veth pair to expose to host.

You see in previous runv's implementation, the qemu is in root net ns, so we can easily plug the tap to qemu and then add the tap to a (let's say) OVS bridge which can gives us a better performance. But I think this can not apply to kata-runtime use case due to netns isolation, And the veth and additional bridge could bring network overhead and is not good.

I am still trying to find a solution, does anyone have good ideas? @egernst @sboeuf @bergwolf

[liwei]

@WeiZhang555 agreed. IMHO, a pair of veth is required due to netns isolation, maybe macvtap on veth or XDP redirect between tap and veth can give better performance than a bridge.

[gnawux]

or, could we have a configuration to put qemu outside the netns? which make things simpler.

[WeiZhang555]

Moving the qemu outside of the netns can definitely make things simpler, but https://github.com/kata-containers/runtime/issues/345 is trying to put qemu in more kinds of namespaces, so I don't know if everyone likes the new proposal

[gnawux]

Actually, either moving qemu out of the netns, or puting qemu in some other namespaces could be treat as a same way -- decouple the qemu (sandbox) namespace from the control plane (such as k8s, docker). i.e. sandbox is runtime stuff, and the runtime should capable to put them in any sandbox by its design and configuration.

[grahamwhaley]

I was thinking it might want to be a configurable option, due to the implicit balance between security and performance. It will add some complexity to the implementation (to support multiple modes).
And we'd have to pick a default mode for the packages. I suspect we should pick 'security' over 'performance' by default.

[gnawux]

Is this (in a netns) means more secure? I think that's more about different connection methods.

[grahamwhaley]

@gnawux I was thinking/meaning that having qemu in a netns was more secure than placing it back out into none or the rootns iyswim. I'm no expert here, just thinking out loud.

[gnawux]

@grahamwhaley I am not a networking guy either. Personally I don't think netns may bring more traffic security for ovs connections.

[bergwolf]

I guess we need to determine how many levels of security isolation we want to provide. Right now we have two and a half:

1. namespaces inside the guest
2. hypervisor
3. netns on the host side

And we are now looking into adding more isolations by introducing more host namespaces in #345. It reminds me of the two-level isolation principle (of Google?) mentioned in the slack channel when we were discussing gvisors. If we follow the same principle, host side netns (and other namespaces) can be optional IMHO. I am not calling out host side namespaces but just want to say that it may be considered optional since we already have at least two level of isolations.

[WeiZhang555]

I'll vote to optional "netns on the host side", as there're no obvious security reasons we have to enable net ns for qemu. And we can easily enable this when we finally find it's necessary.

After disable the host netns, I think the new "update-network" interface can be much more simpler

[amshinde]

But network is not working well. Is kata's QEMU in the net namespace of shim not host? If send QMP "netdev_add ifname=tap0" to runv's QEMU, it will use the tap0 in the host. But kata's QEMU will use the tap0 in the shim net namespace. So the network is not connected correctly.
What is the proper process to hotplug a nic to kata? Need to add veth pair first?

@caoruidong You will need to add the tap interface in the network namespace that has been created by the runtime. In case of docker (or CNI plugin in k8s that uses virtual net interfaces) , a veth pair is created in the network namespace created by the runtime (or provided by the CRI runtime). We then create a macvtap based on one end of the veth pair in the net namespace or a tap+bridge combination with the bridge connected to the other end of the veth pair.
So you will need to create macvtap/tap interfaces after entering into the network namespace. See this:
https://github.com/kata-containers/runtime/blob/b2bec3362bc6af9c2412d483acfbf5b8d6975df6/virtcontainers/network.go#L609-L618

[caoruidong]

@amshinde Yes. Runtime create a veth pair already and the default eth0 uses it. But if we hot plug a second nic like eth1 to the Sandbox, do you mean that we should create another veth pair and put one in the net namespace mannully ?

[WeiZhang555]

I think the "kata-runtime update-network" should only create the "tap/macvtap" interface, it shouldn't care about veth and bridge creation.

So in CNM(libnetwork) scenario, docker will create the veth pair, and then kata-runtime's embedded "net ns monitor" could call "kata-runtime update-network" to create a macvtap based on the existing veth pair in "macvtap mode" or create a tap and a bridge to connect veth and the newly created tap. In both network modes, "kata-runtime update-network" is only responsible for creating the tap/macvtap but isn't responsible for the veth and bridge creation.

In CNI use case, the CNI plugin should be responsible for veth and bridge, and also use "kata-runtime update-network" to create the tap or macvtap, it's basically the same workflow as CNM model.

However, we could have a new configuration option to NOT put qemu inside a standalone net ns, then the CNI plugin can choose to simply put the tap on any bridge and avoid the veth and additional bridge. That's still under discussion and is another topic though.

[sboeuf]

@WeiZhang555 you summarized the situation pretty well.
As a first step, we should implement network hotplug following the current architecture, that is, by creating new interfaces to the VM whenever new interfaces are created inside the network namespace. This is needed to follow the generic runc use case which completely relies on network namespaces.

Now as a second step, and in parallel through a specific document, we could investigate what it would cost (security and performance wise) to put the VM process out of the network namespace. And the outcome could be a specific configuration mode to enable this specific design if it brings better performances.

As a side note, and as mentioned by @amshinde during the Arch meeting, have you looked into CNI plugins using OVS and how they interact with network namespaces usually ? Might worth investigating to get your custom plugin working the same way.

[amshinde]

@WeiZhang555 Thats a great summary. Just one nit:

In both network modes, "kata-runtime update-network" is only responsible for creating the tap/macvtap but isn't responsible for the veth and bridge creation.
I think you just meant veth here, since Kata needs to create the bridge+tap or macvtap.

If I understand correctly from the discussion in the Arch meeting, your custom network plugin expects the runtime to create a tap interface and connect that to an OVS bridge in the host namespace. What is the type of interface passed by the plugin itself? Can you describe your usage scenario a little more?

In the past, we were able to connect a vhost-user interface to a OVS bridge, that was possible of course because vhost-user itself is just a socket file(not network namespaced).
On the same lines, is it possible for you to use a macvtap with your network plugin? (macvtap attached to the network interface your plugin provides). The reason I am asking is because macvtap creates an actual file (/dev/tapXX) which can be accessed from the host namespace. Perhaps that may work for you without having to change the current flow.

[gnawux]

There are two change requests described in this issue right now:

• insert network interfaces after the pod is created.
• make it possible to start the vm in the host netns.

The first is addressed in #287 by @caoruidong right now. And the latter one is disclosed during the #287 was developing.

As @sboeuf said, reaching the target in two separated steps would be good. My question is -- is the two steps conflict? Or say, if we merged #287 now, shall we revert many codes of it for "make it possible to start the vm in the host netns" later, @WeiZhang555 ?

[gnawux]

add a figure for the cases by @WeiZhang555 , correct me if I am wrong

[WeiZhang555]

Sorry for late response.
@gnawux Your picture illustrates our case correctly, thank you for drawing the picture for me! :smiley:

To answer some questions:
@amshinde

your custom network plugin expects the runtime to create a tap interface and connect that to an OVS bridge in the host namespace. What is the type of interface passed by the plugin itself?

the plugin doesn't provide the interface, the runtime will create a normal tap device, and CNI plugin will put the tap on some ovs bridge.

We're simply using ovs+tap now, macvtap isn't supported. In future, we'll try more powerful network solutions such as ovs+vhost-user or ovs + dpdk. But these are under development and final decision wasn't made yet.

@gnawux

Or say, if we merged #287 now, shall we revert many codes of it for "make it possible to start the vm in the host netns" later, @WeiZhang555 ?

I don't see any conflicts, and I think actually we can do the 2 steps in parallel PRs is we think it's reasonable change.

[WeiZhang555]

As a side note, and as mentioned by @amshinde during the Arch meeting, have you looked into CNI plugins using OVS and how they interact with network namespaces usually ? Might worth investigating to get your custom plugin working the same way.

That's a great idea. But sorry that I didn't look into many details in other network solutions such as Calico/Flannel/Canal, I know that the CNI official(/example) plugin is also using the veth, but veth doesn't provide good performance, and additional linux bridge could make performance worse.

And I remember that @egernst mentioned that cri-o can works well with most common CNI plugins, but in the kata-runtime codes, I see that the network interfaces are configured in OCI "prestart"hook and is appended in qemu boot parameters, which should happen in "kata-runtime create" phase, and most CNI plugins should do their work after "kata-runtime create" is done or there won't be any netns to operate. So I'm a little confused, question is, how does CRIO +kata works with common CNI plugins? Do I miss something?

Edit: I just read kata-runtime codes again, CNI supporting code is there(virtcontainers/cni.go) but I cant find any function invoking this, only find a hardcoded CNM network support:

505     sandboxConfig := vc.SandboxConfig{                                           
...                                   
525                                                                                  
526         NetworkModel:  vc.CNMNetworkModel,                                       
527         NetworkConfig: networkConfig,             

[caoruidong]

@WeiZhang555 I think CRIO +kata is CNM too. See https://github.com/kata-containers/runtime/issues/170#issuecomment-380842601

[WeiZhang555]

@caoruidong Oh, I get it now. Thank you!

[amshinde]

@WeiZhang555 @gnawux Thanks for the clarification on the use case.
Just thinking out loud here, have we tried creating the tap first in the host net namespace, connecting it to the OVS bridge and then moving the tap to the container network namespace followed by launching qemu with the tap in the container network namespace. I have been able to do this with a linux bridge using ip cmd. Havent given this a shot with OVS bridge yet.

[sboeuf]

/cc @egernst @caoruidong @amshinde @WeiZhang555 @bergwolf

Following the discussion on Slack, it'd be great to keep things tracked and ack'ed properly here:

First, that's really cool that you all agreed on the different API levels:

• CLI: add-iface, del-iface, list-ifaces, update-routes, list-routes
• Sandbox API: AddInterface, RemoveInterface, ListInterfaces, UpdateRoutes, ListRoutes
• Agent API: UpdateInterface, UpdateRoutes, ListInterfaces, ListRoutes

Also, a few key points where everybody agreed:

• Routes need to be updated after any AddInterface() or RemoveInterface(). This will be the caller responsibility to take care of this.
• We need ListInterfaces() and ListRoutes() to both go all the way to the agent, and return what is really seen by the agent inside the VM
• The agent API will be simply updated by removing not needed API AddInterface() and RemoveInterface()
• We need the agent to handle an interface being hotplugged, basically using uevents
• The Sandbox API will simply error whenever something wrong happens, without trying to rollback anything

Now, there's one thing that is not clear from the conversation is the part about the parameters we will provide to and receive from AddInterface() and RemoveInterface() from the Sandbox API:

• Those functions will receive one single interface since it will avoid virtcontainers from figuring out from a list of interfaces which one has to be added or removed. Is that correct?
• Those functions will return the interface details, no matter the action ended up with error or success. Is that correct?

[bergwolf]

[WeiZhang555]

[sboeuf]

[WeiZhang555]

[grahamwhaley]

[jodh-intel]

[WeiZhang555]

[grahamwhaley]

[bergwolf]