Kops: Lack of Internet Gateway is Fatal Error on Existing Private VPC

[druidsbane]

Hi,

It would be useful to not require an internet gateway when setting up on a pre-existing VPC, especially if it is intended to be completely internal and not have public IP's. Currently it fails with:

internetgateway.go:197] ID must be set, if InternetGateway is shared:
 *awstasks.InternetGateway 
{
  "Name":"cluster.mydomain.com",
  "ID":null,
  "VPC":
  {
    "Name":"cluster.mydomain.com",
    "ID":"vpc-xxxxxxxx",
    "CIDR":"10.2.0.0/16",
    "EnableDNSHostnames":true,
    "EnableDNSSupport":true,
    "Shared":true
   },
  "Shared":true
}

Is there a flag that we can use today to not require it?

Also might be useful option to ensure works when implementing: #428 #694

[kris-nova]

@druidsbane - I will look into how much it will take to get this into #694 - if it's a minor change I can shoehorn it in for you.

Otherwise this might have to be a feature added after the initial private networking PR goes through.

Either way I think it's a valid point, and worthwhile to get into the code base.

[chrislovecnm]

We have PR inbound thay may help with this ... I am looking through issues to test against #1183 - no promises but we need to test.

@kris-nova thoughts?

[chrislovecnm]

See https://github.com/kubernetes/kops/issues/1206 ~ The PR is in. Who can test?

[justinsb]

We do have --topology private now, but we do (in general) still require internet access to download e.g. docker images. It would be great to have an option to have that entirely driven from an S3 bucket, and to let you mirror that S3 bucket if you want to. I _think_ that would be most of what is needed to drop the internet gateway.

@druidsbane can you comment on whether --topology private fits your use case, and whether you have found a way around requiring internet access (i.e. I think things would break if you didn't have an internet gateway)

[fraenkel]

If an internet gateway is required for downloading, can we support egress only internet gateways?

[justinsb]

@fraenkel egress-only internet gateways were new to me; it sounds like they are IPv6 only (reading http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html). But we do support NAT gateways (use --topology private) and it sounds like this is what you are asking for?

[fraenkel]

@justinsb Not exactly but there doesn't seem to be a way to install without an inbound internet connection which is my requirement.

[chrislovecnm]

@fraenkel have a PR in with a fix, client needs it as well

[justinsb]

@fraenkel cool, yes @chrislovecnm is doing some work here. Can you expand on your requirements though? Where should we source docker images from? The three primary options I know of so far:

• An S3 bucket which we then docker load
• A private registry running insider your VPC, and we rewrite the image names so that they point to that private registry
• A private registry in conjunction with docker registry mirroring

Do you have a preferred approach @fraenkel ?

[fraenkel]

@justinsb Yes. I see bootstrapping the system needs s3 since I would not have a private registry or proxy running yet. Once I have a minimal system running, I would want to run either a registry or proxy. I have use cases for both, one for a dev environment (registry) and the other for all other environments which pull from a CI registry.

[chrislovecnm]

@fraenkel I know users have a use case where:

1. provide kops with vpc id
2. provide kops with subnet id(s)
3. kops does not create or modify any networking components: vpc, subnet, route tables, igw, etc

@justinsb private container registry, file repository, and HTTP proxies are often used in conjunction with these requirements, but I think @fraenkel is just concerned about the network. I think this requirement could be fulfilled by allowing our new network phase not touch any networking components.

@fraenkel if kops created only IAM and EC2 components, would that fulfill your needs? Also, I assume that we need to not update or delete any networking components as well.

[fraenkel]

@chrislovecnm Not really. There are many pieces that have hard coded where images and other bits are downloaded from. Providing a locked down VPC would fail immediately because there is no way to easily configure where the bits and pieces required to stand up the various components would come from.

[chrislovecnm]

@fraenkel so that is a different set of PRs that are being work on as well

1. Custom container registry location
2. Custom file repository location
3. Capability to upload to those locations

This includes basically everything. All bits/containers for kops and k8s.

Also working on using an http proxy.

But, back to the topic at hand with networks. Do you want kops to create or update any network components.

[fraenkel]

What is there is fine with me. I always have a pre-created VPC and subnet(s), just no IGW by default.

[adrianmkng]

I can give my 2 cents on the issue, even though I don't have much of an idea how kops works, but I can explain my use case.
I'm using direct connect to connect to instances in my VPC and routing all other traffic from the VPC to the virtual private gateway so that it's only accessible from within the corporate network. Hence internet access from the EC2 instances will be routed through the corporate network and behind the corporate proxy at all times. Like the above examples I'm creating the VPCs and subnets myself and just using kops to setup the Kube cluster.

[chrislovecnm]

My phase work will fix this

/assign

[wadeholler]

+1 very interested in this

[cb-crucks]

+1 Also very interested.

[tewing]

+1 very interested.

[dreampuf]

+1 very interested.

[towens]

+1 Also very interested.

[gozer2222]

+1 Also very very interested.

[zuyong]

+1 Also very very very interested.

[chrislovecnm]

If anyone has some time to test https://github.com/kubernetes/kops/pull/3031, it would be much appreciated. I have tested the PR, but I do not have a VPN or other solution setup to test with.

[dreampuf]

@chrislovecnm I tested it with an InternetGateWay but didn't work with an internal VPN. We have a simular environment.

[chrislovecnm]

@dreampuf can I get more details please?

[dreampuf]

@chrislovecnm Sorry for the misleading. I thought the wrong issue. It's worked both having an IGW to let the kops finish the setting up progress or utilizing this patch to skip forcing create new IGW.

We have a VPN and route the out bandwidth to that VGW. Kops with this patch can work with that. I can send you the history echo if you need it.

[chrislovecnm]

@dreampuf yes please set the API value and test for me. Let me know if the IGW is created and if it isn't if we try to delete one.

  networkRequireGateway: false

Needs to be set either via an edit or use a YAML manifest. You will need to build kops off of my branch or apply the patch to master. https://patch-diff.githubusercontent.com/raw/kubernetes/kops/pull/3031.patch

The help is very much appreciated!!

[dreampuf]

@chrislovecnm

I just created a cluster in a non-IGW VPC.

$ git clone https://github.com/kubernetes/kops.git
Cloning into 'kops'...
remote: Counting objects: 167284, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 167284 (delta 6), reused 9 (delta 3), pack-reused 167247
Receiving objects: 100% (167284/167284), 168.43 MiB | 33.77 MiB/s, done.
Resolving deltas: 100% (97678/97678), done.
$ cd kops
$ patch -p1 < <(curl -s https://patch-diff.githubusercontent.com/raw/kubernetes/kops/pull/3031.patch)
patching file pkg/apis/kops/cluster.go
patching file pkg/apis/kops/v1alpha1/cluster.go
patching file pkg/apis/kops/v1alpha1/zz_generated.conversion.go
patching file pkg/apis/kops/v1alpha1/zz_generated.deepcopy.go
patching file pkg/apis/kops/v1alpha2/cluster.go
patching file pkg/apis/kops/v1alpha2/zz_generated.conversion.go
patching file pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go
patching file pkg/apis/kops/zz_generated.deepcopy.go
patching file pkg/model/network.go
$ docker run --rm -it -v "$PWD":/go/src/k8s.io/kops -w /go/src/k8s.io/kops golang make kops
mkdir -p /go/src/k8s.io/kops/.build/local
go build  -ldflags "" -o /go/src/k8s.io/kops/.build/local/go-bindata k8s.io/kops/vendor/github.com/jteeuwen/go-bindata/go-bindata
cd /go/src/k8s.io/kops; /go/src/k8s.io/kops/.build/local/go-bindata -o upup/models/bindata.go -pkg models -ignore="\\.DS_Store" -ignore="bindata\\.go" -ignore="vfs\\.go" -prefix upup/models/ upup/models/...
go build  -ldflags "-X k8s.io/kops.Version=1.8.1 -X k8s.io/kops.GitVersion=ff5c625a8 " -o /go/src/k8s.io/kops/.build/local/kops k8s.io/kops/cmd/kops/
$ .build/local/kops version
Version 1.8.1 (git-ff5c625a8)

$ kops version
Version 1.8.1 (git-ff5c625a8)
$ aws s3 cp s3://${BUCKETNAME}/${CLUSTERNAME}/config - | grep networkRequire
  networkRequireGateway: false
$ kops update cluster --yes --name ${CLUSTERNAME}
W0213 22:14:31.816094   10090 network.go:116] kops is skipping the creation of an Internet Gateway, as networkRequireGateway is set to false.
I0213 22:14:31.979327   10090 executor.go:91] Tasks: 0 done / 81 total; 33 can run
...
kops has set your kubectl context to kube-canal.adsdev.aws.ec2.internal

Cluster is starting.  It should be ready in a few minutes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[michaelajr]

@chrislovecnm Did this ever make it into the main branch?
@austinmoore-