Weave: Network overlaps with existing route on host

[awh]

_From @dj-hedgehog on January 15, 2017 17:9_

I am trying to setup a single node Kubernetes instance using kubeadm and kubectl on a Ubuntu 16.04 host.
When choosing the network & policy I applied kubectl apply -f https://git.io/weave-kube but weave-net keeps erroring.

The result of kubectl logs -c weave weave-net-ft9gj --namespace=kube-system is:

Network 10.32.0.0/12 overlaps with existing route 10.42.0.0/16 on host.

How can I fix this?

_Copied from original issue: weaveworks/weave-kube#48_

[awh]

@dj-hedgehog you can change the range Weave uses for IP address allocation - see the IPALLOC_RANGE bullet point here:

https://www.weave.works/docs/net/latest/kube-addon/#changing-configuration-options

We tried quite hard to avoid out-of-the-box conflicts with other networking software when we chose our default allocation range (10.32.0.0/12) - what do you have that's using the 10.42.0.0/16 subnet out of interest?

[deitch]

I am confused. Is IPALLOC_RANGE the weave equivalent of --cluster-cidr (which is completely undocumented, of course) in kube-controller-manager?

And is there any way to use the standard kubectl apply -f https://git.io/weave-kube-1.6 but still override just the individual IPALLOC_RANGE? Or do I need to make a copy of the file and change it?

[bboreham]

@deitch you have it about right. We cannot read the value of --cluster-cidr because it's not in an API object. At #2754 we describe work under way to allow things like IPALLOC_RANGE to be set as part of the URL, but right now you need to copy the file and modify it.

[deitch]

We cannot read the value of --cluster-cidr because it's not in an API object

So you have to set it as part of kube-controller-manager but it doesn't give an API way to retrieve it? Ouch. I see why it is constrained.

2754 looks pretty smart. In the meantime, is it possible to set those 7 options at https://www.weave.works/docs/net/latest/kube-addon/#changing-configuration-options explicitly in the static https://git.io/weave-kube-1.6 and https://git.io/weave-kube ? This way:

1. They are explicitly obvious as documentation right in the yml files
2. One is less likely to make errors changing them
3. One can get poor man's automation by doing curl https://git.io/weave-kube-1.6 | sed .... > /tmp/weave-kube-1.6-edited.yml and then kubectl apply -f /tmp/weave-kube-1.6-edited.yml

It isn't as good as putting it in the URL, but reduces error probability and allows cheap automation without causing issues for #2754 . Should I put in a PR for it? Is it the files here https://github.com/weaveworks/weave/tree/master/prog/weave-kube ?

[deitch]

BTW, @bboreham I tried to look at https://github.com/weaveworks/launch-generator and the tracking issues and got 404. Error? Or just a private repo?

[deitch]

Yeah, I had to do some mighty weird awk-fu to get it to do the right thing without manual editing:

awk -v vartoset="IPALLOC_RANGE" -v valtoset="10.10.0.0/24" '
    BEGIN {hasenv = 0; weaveindent = -1; envindent = -1; inweave = 0; inenv = 0; addedenv = 0; intargetenv = 0; }
    function printpad(str,pad)
    {
        padstr = ("%" pad "s%s")
         printf padstr, " ", str
    }
    function printlnpad(str,pad)
    {
        printpad(str"\n", pad)
    }
    {

        # ignore lines that start with #
        if ($0 ~ /^[[:space:]]*\#/) {
            print $0
            next
        }
        # if we already added the env var, just go on
        if (addedenv) {
            print $0
            next
        }
        indent = match($0,/[^[:space:]]/)-1
        if ($0 ~ / name: weave\s*$/) {
          inweave = 1
          weaveindent = indent
          print $0
          next
        }
        # did we start processing weave?
        if (inweave) {
          # are we inside env?
          if (inenv) {
              if (intargetenv) {
                  intargetenv = 0
                  printlnpad(("value: " valtoset), envindent+4)
                  # do NOT print $0 - we are overriding it
                  addedenv = 1
                  next
              }
              # if we left env, see if we addedenv
              if (indent <= envindent ) {
                  inenv = 0
                  if (!addedenv) {
                      printlnpad(("- name: " vartoset),envindent+2)
                      printlnpad(("value: " valtoset), envindent+4)
                      addedenv = 1
                      print $0
                      next
                  }
              }
              # if the line actually is our var, then we will change the next line
              if (match($0, ("name: " vartoset))) {
                  intargetenv = 1
                  print $0
                  next
              }
              print $0
              next
          }
          # if we left weave, see if we added the IP
          # if we had env, we already added it, so no worry. We only will have addedenv == 0 if we did not have env
          if (indent <= weaveindent) {
              inweave = 0
              if (!addedenv) {
                  printlnpad("env:", weaveindent+2)
                  printlnpad(("- name: " vartoset), weaveindent+4)
                  printlnpad(("value: " valtoset), weaveindent+6)
                  addedenv = 1
              }
              print $0
              next
          }
          # see if we are processing env
          if ($0 ~ /^\s+env:\s*$/) {
            hasenv = 1
            inenv = 1
            envindent = indent
          }
      }
      print $0
    }
'

[rade]

That's awkful.

[deitch]

No kidding!!

[mlromramse]

The answer is a bit late but I would have liked to find my answer here.

I simply followed the documentation and did this (the magic is at the end of the row):

  kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&env.IPALLOC_RANGE=10.32.0.0/16"

Documentation link: https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#-changing-configuration-options

[tongchao199]

@mlromramse cool, fix mine problem

[godwhoa]

@mlromramse Thank you so much!!

[lgg42]

Thanks @mlromramse !

[rashad-farajullayev]

The answer is a bit late but I would have liked to find my answer here.

I simply followed the documentation and did this (the magic is at the end of the row):

  kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&env.IPALLOC_RANGE=10.32.0.0/16"

Documentation link: https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#-changing-configuration-options

Thank you.
It fixed my problem

[pg942665]

Worked for me too after i changed the IP address allocation.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d 'n')&env.IPALLOC_RANGE=10.33.0.0/16"

[sreekarachanta]

You guys should be updating the kubernetes.io to include the IPALLOC_RANGE along with the default command that is given there. This is to help people when they write the exam. Exam rules doesn't allow you to access any page outside of kubernetes.io so it will be a problem. Apart from the exam thing, I feel it needs to be updated to have people from not wasting time searching for answers when the default behaviour for sure conflicts during installation. Thank you.

[bboreham]

@sreekarachanta please add some context, ideally in a new issue which is not closed.

• Which exam?
• Which default command?
• What conflict?

[ricsonchua]

I have a problem with this as well. Does it mean then that I have to reserve the IP in my physical network? 10.32.0.0/12 is being used by my organization's network team and I will be unable to SSH to the machine when the DHCP decides to assign my IP within 10.32.0.0/12 for that specific day.

[bboreham]

@ricsonchua the range used by Weave Net is configurable.
Again you have given no context how you are using Weave Net so I cannot point you at the most relevant documentation but the top level is here: https://www.weave.works/docs/net/latest/overview/

[ricsonchua]

@bboreham may I know what other information I should share? I am testing weave as it was something my organization is recommending. Unlike Flannel, Weave seems to create a record in ip addr.

My ip addr result looks something like this below.
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff inet 10.32.0.1/12 brd 10.47.255.255 scope global weave valid_lft forever preferred_lft forever
Problem here is that 10.32.0.1/12 is a range that my organization is using for something else and this is honestly a huge range. I know this can be configured but does that mean that I need to let my network admin know?

My main question here is to understand if I need to let my network admin know. If yes, then I will let my admin know.

Another bigger question is that why is this IP going out of my host and causing conflicts. Shouldn't the IP address used by the CNI be restricted to its own network namespace?

[bboreham]

why is this IP going out of my host and causing conflicts

That should not happen. Weave Net works by encapsulation, as explained in the documentation, so container addresses are not visible on the underlying host network.

Please click on 'new issue' and provide the requested information, then it may be possible to understand what you are seeing.

[ricsonchua]

@bboreham Thanks will do that. but I also need to time my DHCP to replicate the problem. I might not be able to provide any more information as I have pressure to move on with the project as well.

Right now my IP address in DHCP is outside of 10.32.0.1/12 so I am able to work in the machine. I will try to raise a new issue if we will be given time to explore this again.

[bboreham]

You need to pick a range for Weave Net which does not overlap with any addresses you expect to use for anything outside of Weave Net. That is the whole subject of this issue which you have commented to: if Weave Net detects a conflict it will refuse to start.

[GabeChurch]

@mlromramse you are the man