Weave: ability to enable cni portmap plugin

[ekozan]

Hi,

Can you add the possibility of enable portmap plugin on all host on kubernetes

This will solve the hostport problems

Regards :)

[bboreham]

Something like this

[ekozan]

Yeah but it's dont work :/

[bboreham]

Any more details?

[ekozan]

Yes sorry :)
I have compiled it and add the binary on the /opt/cni/bin
/etc/cni/10-mynet.conflist

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

And restart kubelet and cni pod
After that the cni pod restart with no problems but ... all others pod stay stuck on container creating :/

[bboreham]

You can't run with _just_ a portmap plugin; you need a network too.

By analogy to the example I pointed to, for Weave Net it would look something like:

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

caution: I haven't tried this

[ekozan]

this conflist ovewrite other conf file ? it's not realy clear for me :)

I'll give a try this night

[bboreham]

Yes, Kubernetes will only read one CNI config file, the first one it comes to (probably in alphanumeric order)

[nicolai86]

@bboreham I tried the following, but this sadly did not work:

1. place https://github.com/projectcalico/cni-plugin/releases/download/v1.9.0/portmap in /opt/cni/bin/portmap
2. configure /etc/cni/net.d/10-weave.conflist like this (no other file in the directory):

{
    "cniVersion": "0.5.1",
    "name": "weave",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

1. restart the VM

I'm using https://gist.github.com/nicolai86/abfda3b82ec2ba6b4eac788a68c5ff86 as an example to test if hostPort is working. but it doesn't.

[ekozan]

I have made somme mistake on my post "cniVersion" is "0.3.1" ( protocol version )
And i correct the double "type" the correct is "weave-net"
But Nop :)

it's not work

{
    "cniVersion": "0.3.1",
    "name": "mynet",
    "plugins": [
    {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
    },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

[ekozan]

o/ working :)
"cniVersion": "0.3.0",

{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

[nicolai86]

@ekozan can you -vvv your steps a little? download portmap and save your content to /etc/cni/net.d/10-mynet.conflist?

[luxas]

@bboreham Will writing this new CNI spec happen automatically in the future?

[bboreham]

@luxas that would be best, yes. Need to think about how to detect if the user has the portmap plugin installed (or provide a way for Weave Net to install it).

PRs welcome :slightly_smiling_face:

[luxas]

Maybe just do something like hit /version of the API Server?

[ekozan]

full step

$ git clone https://github.com/containernetworking/plugins 
$ cd plugins
$ ./build.sh
$ sudo cp bin/* /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf
$ kubectl delete po --all --namespace=kube-system  

Enjoy

[nicolai86]

@ekozan thank you for the steps. ~how did you verify the hostPort is actually working?~

[bboreham]

Maybe just do something like hit /version of the API Server?

@luxas as I see it two things are key:

1. does the machine have the portmap CNI plugin installed on disk?
2. does the local kubelet have the code to pass details of hostPort mappings?

The API server version doesn't indicate either of these things. I don't think there is a published API to get the kubelet version.

I think at present we would need a manual switch for the operator to turn on if they are sure the prerequisites are met.

[lukemarsden]

FTR, I got this working without requiring git or go (but adding a dependency on wget 😏) on the target system with a small tweak to the instructions in https://github.com/weaveworks/weave/issues/3016#issuecomment-313174867 to use the released cni-plugins v0.6.0-rc1 package. On each of your nodes, run:

$ cd /tmp && mkdir cni-plugins && cd cni-plugins && \
    wget https://github.com/containernetworking/plugins/releases/download/v0.6.0-rc1/cni-plugins-amd64-v0.6.0-rc1.tgz && \
    tar zxfv cni-plugins-amd64-v0.6.0-rc1.tgz
$ sudo cp /tmp/cni-plugins/portmap /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf

On a node with working kubectl (e.g. the master), run:

$ kubectl delete po --all --namespace=kube-system  

Note: if the /etc/cni/net.d/ file doesn't end .conflist, this won't work, and will fail with mysterious no plugin name provided errors.

[Bregor]

@lukemarsden why you need to restart all pods in kube-system?
Maybe just kubectl delete po -n kube-system -l name=weave-net will do the trick?

[lukemarsden]

@Bregor I was just copying https://github.com/weaveworks/weave/issues/3016#issuecomment-313174867 😄 I'm not sure what you need to delete exactly to "restart CNI", would be good if you could run some tests to find out?

[bboreham]

Restarting pods has no effect on CNI.

I might have expected you need to restart kubelet, but if not, that's cool.

[Bregor]

@lukemarsden, @ekozan which kubernetes version did you use?

I use kubernetes-1.6.8 with no success at all.

I tried following CNI + Plugins combinations:

• cni-0.5.1 + plugins-0.6.0-rc1
• cni-0.5.1 + plugins-0.6.0
• cni-0.5.2 + plugins-0.6.0-rc1
• cni-0.5.2 + plugins-0.6.0
• cni-0.6.0 + plugins-0.6.0-rc1
• cni-0.6.0 + plugins-0.6.0

Also tried all combinations above with weave-1.9.8 and 2.0.1.

No success at all.

Events:

nginx     2017-08-21 15:10:15 +0300 MSK   2017-08-21 15:00:48 +0300 MSK   45        nginx-3341093162-f3kkq   Pod                 Warning   FailedSync   kubelet, 10.130.20.89   Error syncing pod, skipping failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Kubelet logs:

Aug 21 15:11:47 kube01 kubelet[24531]: W0821 15:11:47.377124   24531 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "nginx-3341093162-f3kkq_nginx": Cannot find the network namespace, skipping pod network status for container {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.379565   24531 cni.go:278] Error deleting network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.382972   24531 remote_runtime.go:109] StopPodSandbox "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "nginx-3341093162-f3kkq_nginx" network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383066   24531 kuberuntime_manager.go:784] Failed to stop sandbox {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383285   24531 kuberuntime_manager.go:573] killPodWithSyncResult failed: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383340   24531 pod_workers.go:182] Error syncing pod 3b3bc4a9-8664-11e7-a06b-264916ab5742 ("nginx-3341093162-f3kkq_nginx(3b3bc4a9-8664-11e7-a06b-264916ab5742)"), skipping: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Contents of /etc/cni/net.d/10-weave.conf:

{
    "cniVersion": "0.3.0",
    "name": "weave",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

What am I doing wrong?..

[Bregor]

Ah, nevermind.
Just realized, that ability to use NetworkConfigList became available only from 1.7.0-alpha3 :(

[deitch]

@bboreham when I run the kube add-on, it automatically generates /etc/cni/net.d/10-weave.conf. If I have 10-weave.conflist, CNI will pick up the first and miss my generated conflist.

Short of giving it a lower number (higher priority), is there any way to get it to not generate 10-weave.conf?

Conversely, if there were a way to tell the add-on to add the portmap - generate the conflistwith portmap instead of the single conf - perhaps via env var, that would be valuable.

You mentioned earlier that it has an issue detecting; this would give at least a partial resolution.

[ArchiFleKs]

Hi,
@deitch your right and kubelet pick-up the first file and only one.

In fact, Kubelet CNI implementation supports Json, Conf and ConfList (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go#L99). You cannot specifiy multiple plugins in a conf file but you can in a conflist, I think you can do both in Json.

@deitch did the lower priority for the config file work ?

[deitch]

@ArchiFleKs yes. I used the following config manually.

        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I saved it as /etc/cni/net.d/00-weave.conflist. This way, even though weave automatically adds 10-weave.conf, it takes precedence.

I would have preferred to have it auto-generate like weave normally does, but for the future, I guess.

[fabiand]

Nice @deitch so I wonder - what is blocking weave from adding this to their default configuration?

[deitch]

what is blocking weave from adding this to their default configuration?

I do not know. @bboreham ?

[bboreham]

See https://github.com/weaveworks/weave/issues/3016#issuecomment-313644623

[fabiand]

Thanks.

To summarize: As weave, we don't know if we can enable portmap, thus it should be left to the admin to enable it.
best is to make this enablement easy.

[fabiand]

This is actually nasty, I wonder if this can be solved in the realm of Kube. After all portmap is a common thing whenever NodePort is used.

[ArchiFleKs]

@fabiand actually it is needed only if you are using HostPort directly on a container, without Services nor NodePort

[fabiand]

Hm. That's a good hint, and it then sounds as if this is not the issue I am seeing.

I'm using kubelet with CNI and weave, and using NodePort - or the definition used here kubevirt/kubevirt#542 - then the user defined ports will not be opened (only the randomly assigned ones).
The very same definition works with docker networking.

[ArchiFleKs]

@fabiand I replied to you here also https://github.com/kubevirt/kubevirt/issues/542

[ArchiFleKs]

@bboreham

does the machine have the portmap CNI plugin installed on disk?

maybe we could provide the whole CNI package and plugins as packaged in https://github.com/containernetworking/cni/releases at least the will take care of this.

does the local kubelet have the code to pass details of hostPort mappings?

For this one I don't know which kubernetes version introduced HostPort mapping but it is not a very new feature i guess.

[bboreham]

maybe we could provide

who is 'we' ?

which kubernetes version introduced HostPort mapping

...via CNI was 1.7; 30th June 2017

[ArchiFleKs]

who is 'we' ?

I meant weave, the CNI plugin are symlink when running weave setup right ? maybe the portmap plugin could also be shipped / symlinked

...via CNI was 1.7; 30th June 2017

Oh OK. Is it really an issue ? because even before portmap plugin support, I believe it failed silently if you tried to specified an hostPort into a manifest, it just had no effect.

[bboreham]

Yes, we could ship a copy of the portmap plugin. Not completely sold on the idea, though, as it complicates our release process.

The support in kubelet I am referring to is for new CNI config file syntax; all CNI calls will fail if you supply a config that lists plugins and kubelet doesn't understand that.

[luxas]

The portmap plugin will be available in most places in v1.9, as k8s v1.9 ships with CNI v0.6.0

[ryan-dyer-sp]

Am running Kube 1.9.1 with weave 2.0.5. Is there a way to use kubectl apply to do this? From seeing others instructions, is appears this is done from a node. Does it need to just be a single node? all nodes?

[ArchiFleKs]

@ryan-dyer-sp on every node

[bboreham]

@ryan-dyer-sp and there's no real way with kubectl apply - it's config at a level below kubelet so something needs to go onto the node and install it.

[yanhongwang]

Hello all,

In my Kubernetes cluster, SlaveIP:80 can be shown by netstat and accessed by browser.
but MasterIP:80 can't be opened.

I think portmap plugin have already enabled by slave side.
But, why doesn't on master side?

My environment condition:

Ubuntu: 16.04 LTS
Kubernetes: 1.9.3
Docker-ce: 17.03.2~ce-0~ubuntu-xenial
Weave Net: 2.2.1

I follow the official Kubernetes ingress page.
https://github.com/kubernetes/ingress-nginx/tree/master/deploy

In default "with-rbac.yaml", I appended "hostnetwork=true" and "hostport=80" within
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml

/////////////////////////////////////////////////////////////////////////////////////
...
...
spec:
serviceAccountName: nginx-ingress-serviceaccount
hostNetwork: true
containers:
...
...
ports:
- name: http
hostPort: 80
containerPort: 80
...
...
/////////////////////////////////////////////////////////////////////////////////////

Is there something parameter or step that need to enable master portmap in Kubernetes cluster?

Thanks very much.

Hong

[bboreham]

@yanhongwang please open a new issue rather than muddling an existing conversation.

However hostNetwork: true means that the pod will use the host network, rather than Weave Net (or any other CNI plugin), so this is not a Weave Net issue at all.

[yanhongwang]

@bboreham

Thanks.
Sorry for muddling.

Hong

[dtshepherd]

@bboreham I see you tagged "help wanted" on here. Do you have any thoughts how you would solve this? I'm using kops to spin up our cluster, but we need portmap. It seems the way Calico solved this with kops was to package the portmap binary with their docker image and then copy it onto the host through the /opt/cni/bin mount.

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries), then update weave to install the proper /etc/cni/net.d/10-weave.conflist. The current conf doesn't include portmapper obviously.

[murali-reddy]

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries)

With latest releases of kops portmap plugin is part of the bundle of CNI plugins downloaded on to the node. So nothing extra needed to download the plugin. However it is not by default placed in /opt/cni/bin.

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@dtshepherd let me know if you would like to give it a try, otherwise I am happy make the change in Kops.

[bboreham]

Do you have any thoughts how you would solve this?

Not really. Probably the whole thing needs taking apart and doing again.

The current implementation is a bit minamalist and simplest-thing-that-works. If I remember correctly, at the time nobody was installing add-ons purely by running a daemonset; the state of the art was to modify the Kubernetes in-tree saltstack.

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

[deitch]

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

The problem with places outside is that they are, well, outside. There is something great about it all just being in a single yml you can kubectl apply, and keeping it clean.

FWIW, the last installation I did was a mix of the cloud-init downloading the standard CNI distro and installing it, and a modified yml for the DaemonSet with an init container that installs an overriding config file:

          initContainers:
            # set up our correct conflist in /etc/cni/net.d
            - name: init-conflist
              image: busybox
              command: ['sh','-c','mkdir -p /host/etc/cni/net.d/ && cp /conf/00-weave.conflist /host/etc/cni/net.d/']
              volumeMounts:
              - name: cni-conf
                mountPath: /host/etc
              - name: config-volume
                mountPath: /conf

Where the 00-weave.conflist looks like:

        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I rely on the ordering of numbers 00 to get it pulled in. It is somewhat brittle, but at least everything "weave-ish" is in that one place, and everything "standard" is in my cloud-init.

[dtshepherd]

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@murali-reddy Yes, it is a one-liner change in that file but we also need to update the weave install template for kops (https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.weave/k8s-1.6.yaml.template). However, it would be nice for weave to support generating the conflist rather than the conf in /etc/cni/net.d/. I guess for now, we could use the hack suggested by @deitch.

@bboreham As an initial rework to weave, could we do something similar to flannel and use a configmap to generate the /etc/cni/net.d/ file? Currently, the CNI conf is hardcoded in the weave script.

[bboreham]

configmap fine as long as it's optional.

It was so long ago when I did it, optional configmaps didn't exist in Kubernetes...

[dtshepherd]

Yeah the optional configmaps are nice. So maybe implement the logic "if cni configmap exists, use it... otherwise, default to the hardcoded 10-weave.conf.

[murali-reddy]

So portmap CNI will be installed by default in case of kops with upcoming 1.10 release.

https://github.com/kubernetes/kops/pull/5474
https://github.com/kubernetes/kops/pull/5494

this will only cover kops side of things. Still weave need to create .conflist

[fjudith]

Hello,

Do you know this version will be publish to DockerHub.

Lastest (git-ad4e4238a5d5) still provides non portmap 10-weave.conf

Can't wait to test it !

[murali-reddy]

Do you know this version will be publish to DockerHub.

@fjudith support for portmap will be part of 2.5 release

I just need to add a integration test to finish https://github.com/weaveworks/weave/pull/3356

Let me know if you would like to test out the changes in https://github.com/weaveworks/weave/pull/3356, i will share the docker image.

[fjudith]

I have a lab for this puposal running Containerd (1.2.0-Beta.2) and Docker (17.03.2) along with Kubernetes 1.11.2.
I'm open for further testings.

I'm not using the stable containerd v1.1.3 as it does not detects cni configs pushed by k8s like weave-kube. It works find with 1.2.0

[murali-reddy]

@fjudith Great! Please use the image muralireddy/weave-kube:portmap. Note that once you use this image its going to remove /etc/cni/net.d/10-weave.conf on all nodes and replace with /etc/cni/net.d/10-weave.conflist. So if you have revert back please delete .conflist file.

[annismckenzie]

That is exactly why I'd go with an at least in part backwards compatible version if you're going to remove the 10-weave.conf file. Use 11-weave.conflist and then on downgrade the old Weave image will still work because it'll create the 10-weave.conf again and as it takes precedence, it'll not crash. Upgrading to your beta image will drop it again and use the conflist again. Might be worth the effort.

[murali-reddy]

Use 11-weave.conflist and then on downgrade the old Weave image will still work

Ok. that make sense. That might work. As you say its worth the effort to try out. Let me test out and see if can make both forward and backward compatible.

[murali-reddy]

I tested out my PR #3356 again from the perspective of backward compatibility. There is nothing in particular need to be done. When upgrade is done to new release 10-weave.conf is deleted and 10-weave.conflist will be used further. If Weave need to be downgraded to old release, 10-weave.conf will be created. In case there are multiple conf file first one in lexicographical order is chosen. So 10-weave.conf will be used.