Kops: working with pre-defined private subnets

[bdshibly]

Hi,

What will be the command to create kubernetes cluster with pre defined private subnets? I am trying with the following command but getting error.

Note: our DNS zone is also private.

kops create cluster --cloud=aws --name=qa-k8s.mydomain.com --state=s3://qa-kubernetes --zones=us-east-1c --node-count=2 --node-size=t2.micro --master-size=t2.micro --dns-zone=qa-k8s.mydomain.com --vpc=vpc-7c9b311a --network-cidr=10.14.86.0/23

I1229 16:27:42.167333 91221 executor.go:68] Tasks: 0 done / 49 total; 25 can run
W1229 16:27:43.086495 91221 executor.go:84] error running task "vpc/qa-k8s.mydomain.com": Field cannot be changed: CIDR

Thanks
Ferdous Shibly

[gopinatht]

+1 for this requirement

I personally get this error:
W0111 21:43:19.871211 49988 executor.go:109] error running task "Subnet/utility-us-east-1e.<URL>" (7m12s remaining to succeed): error creating subnet: InvalidSubnet.Range: The CIDR '10.44.76.0/29' is invalid. status code: 400, request id: 4f9bfa42-abfd-4ef1-8d98-e889f94ba54d W0111 21:43:19.871238 49988 executor.go:109] error running task "Subnet/us-east-1e.<URL>" (7m12s remaining to succeed): error creating subnet: InvalidSubnet.Conflict: The CIDR '10.44.76.64/26' conflicts with another subnet status code: 400, request id: 31b3cee5-46f8-4501-8277-c9d804fa63b2

I tried to follow the documentation provided in this document by removing the subnets section and adding a zones section but that did not help either.

[Miyurz]

We have been hit by the same issue. Every time we specify the network cidr, it comes back with the error saying

Field cannot be changed: CIDR

Looking at this open issue, it looks like its a genuine one. @justinsb @kris-nova @chrislovecnm can you guys please take a look. I am blocked because of this and can't setup my cluster.

[kris-nova]

So we are putting some work into https://github.com/kubernetes/kops/pull/1366 around specifying subnets, elb, and nat gateways for clusters..

It's a newer improvement that we are hoping to get into 1.5 before the next stable release (we are only in Alpha today)

I will make it a point to test these use cases in the next day or so, and will release a patch if necessary to unblock anyone trying to use existing subnets..

Thanks for your patience! And thanks to @gopinatht @bdshibly and @Miyurz for bringing this to our attention.

[clarkdave]

Just a heads up: I found that if the CIDR you pass to create cluster doesn't actually match the VPC's CIDR, you'll get this somewhat cryptic error on a loop:

Field cannot be changed: CIDR error

So if your VPC CIDR is 10.0.0.0/16 and you are a bit sleepy and pass in --network-cidr=10.100.0.0/16, you'll get the error and things will go badly :)

[jdwilly2001]

I am also receiving this when I am trying to add a new instance group. I am sharing a VPC, but let kops create/manage the subnets for the cluster.
kops version...

jwilliams@awsdevdocker01:~$ kops version
Version 1.5.3 (git-46364f6)

Cluster config:

kops get cluster k8s.domain.com -o yaml
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2017-03-21T20:33:20Z"
  name: k8s.domain.com
spec:
  api:
    loadBalancer:
      type: Public
  channel: stable
  cloudProvider: aws
  configBase: s3://kops.state.bucket/k8s.domain.com
  docker:
    bridgeIP: 172.24.0.1/16
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1b
      name: b
    - instanceGroup: master-us-east-1d
      name: d
    - instanceGroup: master-us-east-1e
      name: e
    name: main
  - etcdMembers:
    - instanceGroup: master-us-east-1b
      name: b
    - instanceGroup: master-us-east-1d
      name: d
    - instanceGroup: master-us-east-1e
      name: e
    name: events
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.5.2
  masterInternalName: api.internal.k8s.domain.com
  masterPublicName: api.k8s.domain.com
  networkCIDR: 172.16.0.0/16
  networkID: vpc-c90e7da7
  networking:
    weave: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.16.132.0/19
    name: us-east-1b
    type: Private
    zone: us-east-1b
  - cidr: 172.16.164.0/19
    name: us-east-1d
    type: Private
    zone: us-east-1d
  - cidr: 172.16.196.0/19
    name: us-east-1e
    type: Private
    zone: us-east-1e
  - cidr: 172.16.100.0/22
    name: utility-us-east-1b
    type: Utility
    zone: us-east-1b
  - cidr: 172.16.104.0/22
    name: utility-us-east-1d
    type: Utility
    zone: us-east-1d
  - cidr: 172.16.108.0/22
    name: utility-us-east-1e
    type: Utility
    zone: us-east-1e
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

and the new instance group I am trying to create:

jwilliams@awsdevdocker01:~$ kops get ig edge -o yaml
Using cluster from kubectl context: k8s.domain.com

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2017-03-22T17:37:07Z"
  labels:
    kops.k8s.io/cluster: k8s.domain.com
  name: edge
spec:
  image: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09
  machineType: t2.micro
  maxSize: 2
  minSize: 2
  nodeLabels:
    lt.cluster.zone: edge
  role: Node
  subnets:
  - us-east-1b
  - us-east-1d
  - us-east-1e

and the output of running kops update cluster:

jwilliams@awsdevdocker01:~$ kops update cluster
Using cluster from kubectl context: k8s.domain.com

I0323 12:06:32.885230   20023 executor.go:91] Tasks: 0 done / 96 total; 32 can run
I0323 12:06:33.255645   20023 executor.go:91] Tasks: 32 done / 96 total; 20 can run
W0323 12:06:33.511638   20023 executor.go:109] error running task "Subnet/us-east-1b.k8s.domain.com" (9m59s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:33.511678   20023 executor.go:109] error running task "Subnet/us-east-1d.k8s.domain.com" (9m59s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:33.511692   20023 executor.go:109] error running task "Subnet/us-east-1e.k8s.domain.com" (9m59s remaining to succeed): Field cannot be changed: CIDR
I0323 12:06:33.511722   20023 executor.go:91] Tasks: 49 done / 96 total; 29 can run
W0323 12:06:35.198901   20023 executor.go:109] error running task "Subnet/us-east-1b.k8s.domain.com" (9m58s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.198925   20023 executor.go:109] error running task "Subnet/us-east-1d.k8s.domain.com" (9m58s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.198939   20023 executor.go:109] error running task "Subnet/us-east-1e.k8s.domain.com" (9m58s remaining to succeed): Field cannot be changed: CIDR
I0323 12:06:35.198960   20023 executor.go:91] Tasks: 75 done / 96 total; 7 can run
I0323 12:06:35.261325   20023 dnsname.go:108] AliasTarget for "api.k8s.domain.com." is "api-k8s-domain-com-dgjth3-132123123123.us-east-1.elb.amazonaws.com."
W0323 12:06:35.569463   20023 executor.go:109] error running task "Subnet/us-east-1b.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.569491   20023 executor.go:109] error running task "Subnet/us-east-1d.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.569503   20023 executor.go:109] error running task "Subnet/us-east-1e.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
I0323 12:06:35.569525   20023 executor.go:91] Tasks: 79 done / 96 total; 6 can run
W0323 12:06:35.637778   20023 executor.go:109] error running task "Subnet/us-east-1e.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.637809   20023 executor.go:109] error running task "Subnet/us-east-1b.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.637823   20023 executor.go:109] error running task "Subnet/us-east-1d.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
I0323 12:06:35.637851   20023 executor.go:91] Tasks: 82 done / 96 total; 3 can run
W0323 12:06:35.682436   20023 executor.go:109] error running task "Subnet/us-east-1b.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.682461   20023 executor.go:109] error running task "Subnet/us-east-1d.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
W0323 12:06:35.682473   20023 executor.go:109] error running task "Subnet/us-east-1e.k8s.domain.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
I0323 12:06:35.682484   20023 executor.go:124] No progress made, sleeping before retrying 3 failed task(s)
I0323 12:06:45.682689   20023 executor.go:91] Tasks: 82 done / 96 total; 3 can run

[sudermanjr]

I am also seeing this error in exactly the same situation as @jdwilly2001 . Not sure if this was determined to be a valid bug, or an issue with our configuration.

[kenden]

This comment described a different issue. Edited and created bug https://github.com/kubernetes/kops/issues/2294 for it.

[sudermanjr]

@kenden Thanks for the detailed workaround writeup, but that is not the issue that I am seeing. I already encountered what you explained. I exported the yaml file, replaced the configuration with my new subnets, and successfully created a cluster.

The issue is that once that cluster is created, any time I perform a kops update cluster, I see the error again. Here are my subnets (which are in a cluster that is already running)

  subnets:
  - cidr: 172.16.6.1/27
    name: kube-blue-us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: 172.16.6.32/27
    name: kube-blue-us-east-1b
    type: Private
    zone: us-east-1b
  - cidr: 172.16.6.64/27
    name: kube-blue-us-east-1c
    type: Private
    zone: us-east-1c
  - cidr: 172.16.6.96/27
    name: kube-blue-utility-us-east-1a
    type: Utility
    zone: us-east-1a
  - cidr: 172.16.6.128/27
    name: kube-blue-utility-us-east-1b
    type: Utility
    zone: us-east-1b
  - cidr: 172.16.6.160/27
    name: kube-blue-utility-us-east-1c
    type: Utility
    zone: us-east-1c

And here a sample of the output when I perform a kops update cluster

I0405 17:45:04.629366   34856 dns.go:90] Private DNS: skipping DNS validation
I0405 17:45:04.951643   34856 executor.go:91] Tasks: 0 done / 114 total; 33 can run
I0405 17:45:05.951948   34856 executor.go:91] Tasks: 33 done / 114 total; 26 can run
W0405 17:45:06.431535   34856 executor.go:109] error running task "Subnet/kube-blue-us-east-1a.kube-blue.ops.ecovate.com" (9m59s remaining to succeed): Field cannot be changed: CIDR
I0405 17:45:06.431605   34856 executor.go:91] Tasks: 58 done / 114 total; 36 can run
W0405 17:45:08.562647   34856 executor.go:109] error running task "Subnet/kube-blue-us-east-1a.kube-blue.ops.ecovate.com" (9m57s remaining to succeed): Field cannot be changed: CIDR
I0405 17:45:08.562681   34856 executor.go:91] Tasks: 93 done / 114 total; 9 can run
I0405 17:45:08.652427   34856 dnsname.go:108] AliasTarget for "bastion.kube-blue.ops.ecovate.com." is "bastion-kube-blue-ops-eco-p0ea42-873805172.us-east-1.elb.amazonaws.com."
I0405 17:45:08.743831   34856 dnsname.go:108] AliasTarget for "api.kube-blue.ops.ecovate.com." is "api-kube-blue-ops-ecovate-co8nmc-923137784.us-east-1.elb.amazonaws.com."
W0405 17:45:08.964354   34856 executor.go:109] error running task "Subnet/kube-blue-us-east-1a.kube-blue.ops.ecovate.com" (9m56s remaining to succeed): Field cannot be changed: CIDR
I0405 17:45:08.964437   34856 executor.go:91] Tasks: 101 done / 114 total; 7 can run
W0405 17:45:09.176460   34856 executor.go:109] error running task "Subnet/kube-blue-us-east-1a.kube-blue.ops.ecovate.com" (9m56s remaining to succeed): Field cannot be changed: CIDR
I0405 17:45:09.176511   34856 executor.go:91] Tasks: 107 done / 114 total; 1 can run
W0405 17:45:09.259950   34856 executor.go:109] error running task "Subnet/kube-blue-us-east-1a.kube-blue.ops.ecovate.com" (9m56s remaining to succeed): Field cannot be changed: CIDR
I0405 17:45:09.259983   34856 executor.go:124] No progress made, sleeping before retrying 1 failed task(s)

Kops Version: 1.5.3

[kenden]

@sudermanjr That's a different error indeed, sorry for highjacking the bug report. I'll edit my comment and replace by a link to a new bug report: https://github.com/kubernetes/kops/issues/2294

[sudermanjr]

I found the issue with my setup. Apparently amazon defaults the first CIDR to x.x.x.0/27. When editing the cluster, your first subnet must match that, otherwise kops tries to change it.

[geojaz]

closing. fixed in https://github.com/kubernetes/kops/pull/1366

[VolodymyrCL]

Hello,

I have a existing infrastructure in AWS - VPC and subnets. The VPC includes two CIDR
10.0.68.0/24
10.0.33.0/24

The VPC has a complex connections with the infrastructure of our partners (VPC peereng, VPN etc). Therefore, I want to use the existing CIDR, which is already integrated into the overall infrastructure.

10.0.68.0/24 is default CIDR for VPC, but this network have a low number of unused IPs.

I am trying to create a cluster on 10.0.33.0/24:

kops create cluster \
--name kube.dev.domain.net \
--dns private \
--dns-zone kube.dev.domain.net \
--zones us-east-1a \
--master-count 3 \
--master-size t2.medium \
--node-count 3 \
--node-size t2.medium \
--state s3://kube-kops \
--vpc vpc-id \
--network-cidr 10.0.33.0/24 \
--subnets subnet-id \
--admin-access 10.0.33.0/24 \
--ssh-access 10.0.33.0/24 \
--ssh-public-key /path/to/id_rsa.pub \
--yes

but I get the error
W0516 12:02:40.640118 5575 executor.go:118] error running task "VPC/kube.dev.domain.net" (9m50s remaining to succeed): CIDR: Invalid value: "10.0.33.0/24": field is immutable: old="10.0.68.0/24" new="10.0.33.0/24"

If I do not specify --network-cidr 10.0.33.0/24, I get error
Spec.Subnets[0].CIDR: Invalid "10.0.33.0/24": Subnet "us-east-1a" had a CIDR "10.0.33.0/24" that was not a subnet of the NetworkCIDR "10.0.68.0/24"

Is this configuration possible? If yes, where I make a mistake?

Thank you!

[mpenate]

Similar error on my cluster creation job, it looks that you can't change your CIDR once you have created the cluster... even if it collides with an existent net. Any ideas on how to bypass this?